MegalaniaBlog
  1. Blog
  2. AI agents
  3. Pydantic AI

Pydantic AI web scraping: MCP tools with a cap the model cannot raise

Give a Pydantic AI agent web pages with MCPToolset, only the tools it needs, and a per-page credit cap the model cannot change. Tested on Pydantic AI 2.68.

Jordan HayesDeveloper Relations
6 min read
AI agents
A Python card with an Agent and an MCPToolset sends a call through a filter with three tool tags and a lock tag that reads max 20. The call reaches a page card and comes back with 12 products and an 18 credits tag.

The short answer: for Pydantic AI web scraping, connect our hosted MCP server with MCPToolset, the class Pydantic AI 2 uses for every MCP server. Send your key in headers. Keep only the tools the agent needs with .filtered(...). Then add a process_tool_call hook that sets max_credits on every call, so the model cannot raise what a page may cost. On our test store, lab.megalania.io, the agent read a JavaScript page with 12 products for 18 credits. A call that needed 32 credits was refused before any fetch, for free.

Pydantic AI 2 changed the MCP classes

If you follow an older tutorial, MCPServerStreamableHTTP now fails with an ImportError: it was removed in Pydantic AI 2. Use MCPToolset, which picks the transport from what you pass it, a URL here.

cURL
pip install "pydantic-ai-slim[mcp]"
export MEG_API_KEY='meg_sk_...'

The toolset: three tools and a cap

Save this as tools.py:

tools.py
import asyncio
import os

from pydantic_ai.mcp import MCPToolset


async def cap(ctx, call_tool, name, args):
    if name in {"scrape", "extract"}:
        args = {**args, "max_credits": 20}
    return await call_tool(name, args)


megalania = MCPToolset(
    "https://mcp.megalania.io/",
    headers={"Authorization": f"Bearer {os.environ['MEG_API_KEY']}"},
    process_tool_call=cap,
).filtered(lambda ctx, tool: tool.name in {"check_site", "scrape", "extract"})


async def main():
    async with megalania:
        tools = await megalania.wrapped.list_tools()
        print(len(tools), "tools on the server")
        price = await megalania.wrapped.direct_call_tool("check_site", {"host": "lab.megalania.io"})
        print(price["price"]["quotes"])
        page = await megalania.wrapped.direct_call_tool("scrape", {"url": "https://lab.megalania.io/faq/"})
        print(page["credits_charged"], page["route"], len(page["content"]))


if __name__ == "__main__":
    asyncio.run(main())
Output
16 tools on the server
{'browser': 15, 'browser_residential': 32, 'datacenter': 1, 'residential': 8}
1 http/datacenter 1127
  • The key goes in headers, never in the URL. A script needs no browser sign-in this way.
  • .filtered(...) shows the model only three tools. Fewer tools, fewer tokens, and no batch or monitor the agent could start by mistake.
  • process_tool_call runs before every call the agent makes. Here it overwrites max_credits with 20 on scrape and extract, whatever the model asked for. Only those two tools take max_credits: added to check_site, it is an error (“max_credits is not a parameter of this tool”), so the hook leaves check_site alone.
  • direct_call_tool calls a tool without a model, to check the setup. check_site is free: on our test store a page costs 1 credit over plain HTTP and 15 in the browser. The FAQ cost 1 credit and came back as 1,127 characters of Markdown.
  • Our dev server showed 16 tools; the 16th, search, is switched on only there. Our hosted server lists 15.

The agent

Any model works. To show the tool calls without a model key, this run uses Pydantic AI’s FunctionModel, a model you script yourself. It asks for the new arrivals page in the browser, with a wait for the 12th card, and it asks for a cap of 500 credits:

agent_run.py
from pydantic_ai import Agent
from pydantic_ai.messages import ModelResponse, TextPart, ToolCallPart, ToolReturnPart
from pydantic_ai.models.function import AgentInfo, FunctionModel

from tools import megalania


def scripted(messages, info: AgentInfo):
    returns = [p for m in messages for p in getattr(m, "parts", []) if isinstance(p, ToolReturnPart)]
    print("model sees tools:", sorted(t.name for t in info.function_tools))
    if not returns:
        return ModelResponse(parts=[ToolCallPart("scrape", {"url": "https://lab.megalania.io/new-arrivals/", "engine": "browser", "actions": [{"wait_for": "#item-12"}], "max_credits": 500})])
    r = returns[-1].content
    return ModelResponse(parts=[TextPart(f"{r.get('credits_charged')} credits, {r.get('route')}, {r.get('content', '').count(chr(10) + '- ')} products")])


agent = Agent(FunctionModel(scripted), toolsets=[megalania])
print(agent.run_sync("List the new arrivals with prices.").output)
Output
model sees tools: ['check_site', 'extract', 'scrape']
model sees tools: ['check_site', 'extract', 'scrape']
18 credits, browser/datacenter, 12 products

With a real model, replace FunctionModel(scripted) with your model, for example Agent("openai:gpt-5", toolsets=[megalania]). We did not run a paid model for this post: it needs your own model key.

  • The page is drawn by JavaScript. Over plain HTTP it is an empty list. In a real Chrome on our side, with the wait, all 12 products came back for 18 credits: 15 for the browser and 3 for the page action.
  • The model asked for 500. The hook sent 20, and 18 fits.
Free beta, by invite76,000 credits for 30 days, no card. Apply, and we email your invite.Apply for the beta

The cap at work

Now the scripted model asks for the same page on the browser with a residential IP, the most expensive route, again with max_credits: 500. The tool result the model gets:

Tool result
{"code": "max_credits_too_low", "credits_charged": 0, "message": "max_credits is below the lowest price on this site, 32 credits. You are not charged. Raise max_credits to at least 32, or remove it.", "next_step": "You were not charged. Read message.", "ok": false}

The hook sent 20, the route costs 32, so we refused before fetching anything, and charged 0. The model reads why, and can choose a cheaper route or stop. The model cannot get past the cap by asking: only your code sets it.

For a budget per run instead of per page, keep a counter in the hook: add each credits_charged, and pass what is left as max_credits. The idea is in Cap what an AI agent spends on web pages.

What it costs

Step Credits
check_site 0
A plain page (scrape, plain HTTP) 1
A JavaScript page in the browser, with one action 18
A call refused by max_credits 0
A block we detect, or one of our errors 0

These prices are for our test store on the datacenter route. A site that blocks it costs more, and check_site shows how much. All the rules are in Credits. Plans are on the pricing page.

How we measured: our dev cluster on 16 Nov 2026, the Lab store on our dev domain, Pydantic AI 2.68.0 (pydantic-ai-slim with mcp 2.3.0) on Python 3.13. We ran tools.py, agent_run.py, and the same agent with "proxy_tier": "residential" and no action, for the cap; then an agent that calls check_site first and the capped scrape second, to check the hook leaves check_site alone. The outputs above are from those runs. Pydantic AI prints a banner about observability on start; PYDANTIC_AI_NO_BANNER=1 turns it off.

Limits

  • Page text is untrusted. A page can hide instructions for a model. Our tool descriptions tell the agent never to follow them. Say it in your system prompt too.
  • Public pages only. We do not log in, and we do not solve CAPTCHAs. When a page needs either, we stop and say so, for free.
  • An empty list can still be a charged answer. Our check for JavaScript shells misses pages that have a heading and some text. Tell your agent to count the rows it expected.

More on MCP for agents in Give your AI agent web access with an MCP server, and every tool and argument in MCP for agents. Fetch public pages only. Read the site’s terms and robots.txt, and follow them.

Jordan HayesDeveloper RelationsPublished 16 Nov 2026 · measured on our dev cluster, 16 Nov 2026, Pydantic AI 2.68.0 (pydantic-ai-slim with mcp 2.3.0), Python 3.13, our test store's FAQ and new arrivals pages

See also

Keep reading