MegalaniaBlog
  1. Blog
  2. AI agents
  3. LangGraph

LangGraph web scraping: MCP tools in a graph, with a cost cap

Load our MCP tools into a LangGraph ToolNode with langchain-mcp-adapters, keep three, and cap each page in an interceptor. Tested on LangGraph 1.2.

Jordan HayesDeveloper Relations
5 min read
AI agents
A graph of three round nodes, START, model and tools, with a loop between model and tools. A lock tag that reads max 20 sits on the edge into tools. A page card beside it shows 12 products and an 18 credits tag.

The short answer: for LangGraph web scraping, load our hosted MCP server’s tools with langchain-mcp-adapters, keep the ones your agent needs, and put them in a ToolNode. Send your key in the connection’s headers. Add a tool interceptor that sets max_credits on scrape and extract, so the model cannot raise what a page may cost. On our test store, lab.megalania.io, the graph read a JavaScript page with 12 products for 18 credits, and a call that needed 32 credits was refused before any fetch, for free.

Install

cURL
pip install langgraph langchain-mcp-adapters
export MEG_API_KEY='meg_sk_...'

langchain-mcp-adapters turns each MCP tool into a LangChain tool, so ToolNode, tools_condition and bind_tools take them as they are. If you only want pages as LangChain documents, without an agent, our langchain-megalania loader is simpler: see Web scraping with LangChain.

The graph

The model node below is scripted, so you can run the whole graph without a model key: it asks for three tool calls in order, then stops. In your own graph, the model node is your chat model with .bind_tools(tools).

graph.py
import asyncio
import os

from langchain_core.messages import AIMessage, HumanMessage, ToolMessage
from langchain_mcp_adapters.client import MultiServerMCPClient
from langgraph.graph import START, MessagesState, StateGraph
from langgraph.prebuilt import ToolNode, tools_condition

KEEP = {"check_site", "scrape", "extract"}


async def cap(request, handler):
    if request.name in {"scrape", "extract"}:
        request = request.override(args={**request.args, "max_credits": 20})
    return await handler(request)


client = MultiServerMCPClient(
    {"megalania": {"transport": "streamable_http", "url": "https://mcp.megalania.io/", "headers": {"Authorization": f"Bearer {os.environ['MEG_API_KEY']}"}}},
    tool_interceptors=[cap],
)

PLAN = [
    ("check_site", {"host": "lab.megalania.io"}),
    ("scrape", {"url": "https://lab.megalania.io/new-arrivals/", "engine": "browser", "actions": [{"wait_for": "#item-12"}], "max_credits": 500}),
    ("scrape", {"url": "https://lab.megalania.io/new-arrivals/", "engine": "browser", "proxy_tier": "residential", "max_credits": 500}),
]


def model(state: MessagesState):
    done = sum(isinstance(m, ToolMessage) for m in state["messages"])
    if done < len(PLAN):
        name, args = PLAN[done]
        return {"messages": [AIMessage("", tool_calls=[{"name": name, "args": args, "id": f"call-{done}"}])]}
    return {"messages": [AIMessage("done")]}


async def main():
    tools = [t for t in await client.get_tools() if t.name in KEEP]
    print(sorted(t.name for t in tools))
    graph = StateGraph(MessagesState)
    graph.add_node("model", model)
    graph.add_node("tools", ToolNode(tools))
    graph.add_edge(START, "model")
    graph.add_conditional_edges("model", tools_condition)
    graph.add_edge("tools", "model")
    app = graph.compile()
    result = await app.ainvoke({"messages": [HumanMessage("List the new arrivals with prices.")]})
    for m in result["messages"]:
        if isinstance(m, ToolMessage):
            text = m.content[0]["text"] if isinstance(m.content, list) else m.content
            print(m.name, "|", m.status, "|", text.count("\n- "), "products |", text[-160:].replace("\n", " "))


asyncio.run(main())
Output (trimmed to the end of each tool result)
['check_site', 'extract', 'scrape']
check_site | success | 0 products | ..."engine":"http","mean_target_ms":1512.6478,"requests":159,"success_rate":1.0,"tier":"datacenter"}],"days":7,...}}
scrape | success | 12 products | ...$78.00   This page is a list: run inspect, then extract for clean rows.  credits_charged: 18
scrape | error | 0 products | ...max_credits is below the lowest price on this site, 32 credits. You are not charged. Raise max_credits to at least 32, or remove it. Read message.
  • The key is a header on the connection, never in the URL. No browser sign-in is needed for a script.
  • Three tools, not 15. Filtering the list before the ToolNode keeps the model’s prompt short, and keeps batches and monitors out of reach.
  • The interceptor runs on every tool call. It overwrites max_credits with 20 on scrape and extract. Only those two take max_credits: put it on check_site and the call fails with “max_credits is not a parameter of this tool”, so the interceptor leaves check_site alone.
  • check_site is free. It returned our recent results on the host (159 requests in 7 days, no blocks) and its prices: 1 credit a page over plain HTTP, 15 in the browser.
  • The JavaScript page came back with all 12 products from a real Chrome on our side, for 18 credits: 15 for the browser, 3 for the wait. The model asked for a cap of 500; the interceptor sent 20, and 18 fits.
  • The residential browser costs 32 on this host. With the cap at 20 we refused it before fetching anything: status is error in the ToolMessage, and 0 credits were charged. The model reads why, and can choose a cheaper route or stop.
Free beta, by invite76,000 credits for 30 days, no card. Apply, and we email your invite.Apply for the beta

With a real model

Replace the scripted model function with a node that calls your chat model:

model node
llm = your_chat_model.bind_tools(tools)


def model(state: MessagesState):
    return {"messages": [llm.invoke(state["messages"])]}

The rest of the graph stays the same. We did not run a paid model for this post: it needs your own model key.

A budget per run

The interceptor sees every call, so it can also keep a running total. Read credits_charged from each result, subtract it from your budget, and pass what is left as max_credits. When nothing is left, every further page is refused for free. The idea, with numbers, is in Cap what an AI agent spends on web pages.

What it costs

Step Credits
check_site 0
A plain page (scrape, plain HTTP) 1
A JavaScript page in the browser, with one action 18
A call refused by max_credits 0
A block we detect, or one of our errors 0

These prices are for our test store on the datacenter route. A site that blocks it costs more, and check_site shows how much. All the rules are in Credits. Plans are on the pricing page.

How we measured: our dev cluster on 16 Nov 2026, the Lab store on our dev domain, LangGraph 1.2.22, langchain-mcp-adapters 0.3.2 and mcp 1.30.0 on Python 3.13. We ran graph.py as above; the output is copied from that run, trimmed to the end of each tool result.

Limits

  • Page text is untrusted. A page can hide instructions for a model. Our tool descriptions tell the agent never to follow them. Say it in your system prompt too.
  • Public pages only. We do not log in, and we do not solve CAPTCHAs. When a page needs either, we stop and say so, for free.
  • An empty list can still be a charged answer. Our check for JavaScript shells misses pages that have a heading and some text. Tell your agent to count the rows it expected.

Every tool and argument is in MCP for agents. Fetch public pages only. Read the site’s terms and robots.txt, and follow them.

Jordan HayesDeveloper RelationsPublished 16 Nov 2026 · measured on our dev cluster, 16 Nov 2026, LangGraph 1.2.22, langchain-mcp-adapters 0.3.2, mcp 1.30.0, Python 3.13, our test store's new arrivals page

Keep reading