The short answer: for Pydantic AI web scraping, connect our hosted MCP server with MCPToolset, the class Pydantic AI 2 uses for every MCP server. Send your key in headers. Keep only the tools the agent needs with .filtered(...). Then add a process_tool_call hook that sets max_credits on every call, so the model cannot raise what a page may cost. On our test store, lab.megalania.io, the agent read a JavaScript page with 12 products for 18 credits. A call that needed 32 credits was refused before any fetch, for free.
Pydantic AI 2 changed the MCP classes
If you follow an older tutorial, MCPServerStreamableHTTP now fails with an ImportError: it was removed in Pydantic AI 2. Use MCPToolset, which picks the transport from what you pass it, a URL here.
pip install "pydantic-ai-slim[mcp]" export MEG_API_KEY='meg_sk_...'
The toolset: three tools and a cap
Save this as tools.py:
import asyncio
import os
from pydantic_ai.mcp import MCPToolset
async def cap(ctx, call_tool, name, args):
if name in {"scrape", "extract"}:
args = {**args, "max_credits": 20}
return await call_tool(name, args)
megalania = MCPToolset(
"https://mcp.megalania.io/",
headers={"Authorization": f"Bearer {os.environ['MEG_API_KEY']}"},
process_tool_call=cap,
).filtered(lambda ctx, tool: tool.name in {"check_site", "scrape", "extract"})
async def main():
async with megalania:
tools = await megalania.wrapped.list_tools()
print(len(tools), "tools on the server")
price = await megalania.wrapped.direct_call_tool("check_site", {"host": "lab.megalania.io"})
print(price["price"]["quotes"])
page = await megalania.wrapped.direct_call_tool("scrape", {"url": "https://lab.megalania.io/faq/"})
print(page["credits_charged"], page["route"], len(page["content"]))
if __name__ == "__main__":
asyncio.run(main())16 tools on the server
{'browser': 15, 'browser_residential': 32, 'datacenter': 1, 'residential': 8}
1 http/datacenter 1127- The key goes in
headers, never in the URL. A script needs no browser sign-in this way. .filtered(...)shows the model only three tools. Fewer tools, fewer tokens, and no batch or monitor the agent could start by mistake.process_tool_callruns before every call the agent makes. Here it overwritesmax_creditswith 20 onscrapeandextract, whatever the model asked for. Only those two tools takemax_credits: added tocheck_site, it is an error (“max_creditsis not a parameter of this tool”), so the hook leavescheck_sitealone.direct_call_toolcalls a tool without a model, to check the setup.check_siteis free: on our test store a page costs 1 credit over plain HTTP and 15 in the browser. The FAQ cost 1 credit and came back as 1,127 characters of Markdown.- Our dev server showed 16 tools; the 16th,
search, is switched on only there. Our hosted server lists 15.
The agent
Any model works. To show the tool calls without a model key, this run uses Pydantic AI’s FunctionModel, a model you script yourself. It asks for the new arrivals page in the browser, with a wait for the 12th card, and it asks for a cap of 500 credits:
from pydantic_ai import Agent
from pydantic_ai.messages import ModelResponse, TextPart, ToolCallPart, ToolReturnPart
from pydantic_ai.models.function import AgentInfo, FunctionModel
from tools import megalania
def scripted(messages, info: AgentInfo):
returns = [p for m in messages for p in getattr(m, "parts", []) if isinstance(p, ToolReturnPart)]
print("model sees tools:", sorted(t.name for t in info.function_tools))
if not returns:
return ModelResponse(parts=[ToolCallPart("scrape", {"url": "https://lab.megalania.io/new-arrivals/", "engine": "browser", "actions": [{"wait_for": "#item-12"}], "max_credits": 500})])
r = returns[-1].content
return ModelResponse(parts=[TextPart(f"{r.get('credits_charged')} credits, {r.get('route')}, {r.get('content', '').count(chr(10) + '- ')} products")])
agent = Agent(FunctionModel(scripted), toolsets=[megalania])
print(agent.run_sync("List the new arrivals with prices.").output)model sees tools: ['check_site', 'extract', 'scrape'] model sees tools: ['check_site', 'extract', 'scrape'] 18 credits, browser/datacenter, 12 products
With a real model, replace FunctionModel(scripted) with your model, for example Agent("openai:gpt-5", toolsets=[megalania]). We did not run a paid model for this post: it needs your own model key.
- The page is drawn by JavaScript. Over plain HTTP it is an empty list. In a real Chrome on our side, with the wait, all 12 products came back for 18 credits: 15 for the browser and 3 for the page action.
- The model asked for 500. The hook sent 20, and 18 fits.
The cap at work
Now the scripted model asks for the same page on the browser with a residential IP, the most expensive route, again with max_credits: 500. The tool result the model gets:
{"code": "max_credits_too_low", "credits_charged": 0, "message": "max_credits is below the lowest price on this site, 32 credits. You are not charged. Raise max_credits to at least 32, or remove it.", "next_step": "You were not charged. Read message.", "ok": false}The hook sent 20, the route costs 32, so we refused before fetching anything, and charged 0. The model reads why, and can choose a cheaper route or stop. The model cannot get past the cap by asking: only your code sets it.
For a budget per run instead of per page, keep a counter in the hook: add each credits_charged, and pass what is left as max_credits. The idea is in Cap what an AI agent spends on web pages.
What it costs
| Step | Credits |
|---|---|
check_site |
0 |
A plain page (scrape, plain HTTP) |
1 |
| A JavaScript page in the browser, with one action | 18 |
A call refused by max_credits |
0 |
| A block we detect, or one of our errors | 0 |
These prices are for our test store on the datacenter route. A site that blocks it costs more, and check_site shows how much. All the rules are in Credits. Plans are on the pricing page.
How we measured: our dev cluster on 16 Nov 2026, the Lab store on our dev domain, Pydantic AI 2.68.0 (pydantic-ai-slim with mcp 2.3.0) on Python 3.13. We ran tools.py, agent_run.py, and the same agent with "proxy_tier": "residential" and no action, for the cap; then an agent that calls check_site first and the capped scrape second, to check the hook leaves check_site alone. The outputs above are from those runs. Pydantic AI prints a banner about observability on start; PYDANTIC_AI_NO_BANNER=1 turns it off.
Limits
- Page text is untrusted. A page can hide instructions for a model. Our tool descriptions tell the agent never to follow them. Say it in your system prompt too.
- Public pages only. We do not log in, and we do not solve CAPTCHAs. When a page needs either, we stop and say so, for free.
- An empty list can still be a charged answer. Our check for JavaScript shells misses pages that have a heading and some text. Tell your agent to count the rows it expected.
More on MCP for agents in Give your AI agent web access with an MCP server, and every tool and argument in MCP for agents. Fetch public pages only. Read the site’s terms and robots.txt, and follow them.