The short answer: for Vercel AI SDK web scraping, npm install @megalania/ai-sdk megalania ai and pass megalaniaTools() as tools to generateText or streamText. The model gets five tools that read pages: scrape, extract, map_site, create_batch and get_batch. On our test store, lab.megalania.io, the FAQ page came back as 1,127 characters of Markdown for 1 credit. Plain fetch gave the model 56,038 characters of HTML for the same page. You set the most one page may cost, and the model cannot raise it.
The free way first
The AI SDK lets you write any tool with tool(). The simplest web tool is plain fetch:
import { jsonSchema, tool } from "ai";
const fetchPage = tool({
description: "Fetch a web page and return its HTML.",
inputSchema: jsonSchema({ type: "object", properties: { url: { type: "string" } }, required: ["url"] }),
execute: async ({ url }) => (await fetch(url)).text(),
});
const html = await fetchPage.execute({ url: "https://lab.megalania.io/faq/" }, { toolCallId: "1", messages: [] });
console.log(html.length, html.indexOf("<h1"));56038 44142
It works on the Lab. But the model reads the whole page: 56,038 characters, and the page’s first heading comes after 44,142 characters of head, menu and scripts. That is context you pay your model provider for, on every call. And fetch gets nothing from a site that blocks plain requests or needs a browser.
Use our tools when you want the main content only, fields as JSON, or pages that block fetch.
Install the tools
npm install @megalania/ai-sdk megalania ai export MEG_API_KEY='meg_sk_...'
Node 22.18 or newer, AI SDK 5 or newer. The key comes from MEG_API_KEY.
Give the tools to a model
import { generateText, stepCountIs } from "ai";
import { megalaniaTools } from "@megalania/ai-sdk";
const { text } = await generateText({
model: "openai/gpt-5",
tools: megalaniaTools({ maxCredits: 8 }),
stopWhen: stepCountIs(5),
prompt: "What does express shipping cost at https://lab.megalania.io/shipping/?",
});
console.log(text);megalaniaTools()returns an object of AI SDK tools. Pass it astools, or pick some:const { scrape, extract } = megalaniaTools().maxCreditsis the most one page may cost. More on it below.stopWhen: stepCountIs(5)is the AI SDK’s own limit on tool steps. Keep one: it stops a model that calls tools in a loop.
The model call needs your own model key, and we did not run it for this post. The tool calls below are what the model gets back.
Run the tools yourself
Each tool has execute, so you can call it with no model, to see what the model will see:
import { megalaniaTools } from "@megalania/ai-sdk";
const tools = megalaniaTools({ maxCredits: 8 });
const call = { toolCallId: "1", messages: [] };
const faq = await tools.scrape.execute({ url: "https://lab.megalania.io/faq/" }, call);
console.log(faq.credits_charged, faq.text.length);
console.log(faq.text.slice(0, 160));
const rates = await tools.extract.execute({
url: "https://lab.megalania.io/shipping/",
extract: "rates:table tbody tr{service:td:nth-child(1)|time:td:nth-child(2)|price:td:nth-child(3)}*",
}, call);
console.log(rates.credits_charged, rates.text);
try {
await tools.scrape.execute({ url: "https://lab.megalania.io/faq/", engine: "browser", max_credits: 100 }, call);
} catch (e) {
console.log(e.message.slice(0, 150));
}1 1127
# FAQ
Is this a real shop?
No. Gibber & Co. is a demo store by Megalania for testing web scraping. Nothing here is for sale. Checkout is switched off and no p
2 {"rates":[{"service":"Standard","time":"3 to 5 working days","price":"$6.00, free over $75"},{"service":"Express","time":"1 to 2 working days","price":"$14.00"},{"service":"Collect in store","time":"Ready the next working day","price":"Free"}]}
max_credits_too_low: max_credits=8 is below the cheapest route for this page. Create the tools with a higher maxCredits (up to 32767). max_credits isscrapereturns the main content as Markdown. The FAQ page was 1,127 characters, not 56,038, for 1 credit. It returns{ text, credits_charged, request_id }.extractreturns fields, not a page. CSS rules read the shipping table into three rows of JSON for 2 credits: 1 for the page and 1 for the JSON. More on rules in How to scrape an HTML table into CSV.- The cap held. The browser costs 15 credits on this site, above the cap of 8. The call stopped before any try, for free, and the error tells the model why.
The cap the model cannot raise
Look at the last call again. It asked for max_credits: 100, and the cap of 8 still won. max_credits is not in the schema the model sees, and the tools send your maxCredits on every call, after the model’s own fields. A model cannot talk its way past it.
| Option | Default | What it limits |
|---|---|---|
maxCredits |
128 | The most one page may cost, up to 32767. 128 covers every route. |
maxPages |
100 | The most pages in one create_batch call, crawls included. More URLs are refused. |
maxChars |
100,000 | The most characters a tool returns, so one page cannot fill the context. |
maxCredits is per page, so one create_batch call can cost up to maxCredits times maxPages. With the defaults that is a lot. If your agent does not need batches, leave create_batch and get_batch out. For a cap on the whole run, see Cap what an AI agent spends on web pages.
Tools from our MCP server
@megalania/ai-sdk calls our API directly. It has no check_site, the free tool that tells an agent the price of a site before it spends anything. That one is on our MCP server, with 15 tools in all. If you build in Python instead, the same tools come in LangChain.
What it costs
| Step | Pages | Credits |
|---|---|---|
Plain fetch on the FAQ page |
1 | 0 |
scrape on the FAQ page |
1 | 1 |
extract on the shipping table |
1 | 2 |
scrape with the browser, over the cap |
— | 0 |
| A blocked page or one of our errors | — | 0 |
These prices are for this site on the datacenter route. A site that blocks it costs more. Plans are on the pricing page.
How we measured: our dev cluster on 16 Nov 2026, the Lab store on our dev domain, @megalania/ai-sdk 0.2.0, megalania 0.3.0 and ai 7.0.173 from npm, Node 24. We ran free.mjs and web.mjs once each. We did not run agent.mjs.
Limits
- Node 22.18 or newer, and AI SDK 5 or newer.
- The cap is per page, not per run. Keep
stopWhenon, and leave out the batch tools you do not need. - Page content is untrusted. The tool descriptions tell the model so. Do not give the same agent tools that send mail or spend money.
- Blocked pages are not content. The tool throws, free, and the model sees the reason.
The setup is also in Vercel AI SDK. Fetch public pages only. Read the site’s terms and robots.txt, and follow them.