What your agent gets
- Markdown, not HTML. The main text of the page, without the menu. Fewer tokens for the model to read.
- Fields as JSON.
extractreads fields with CSS rules, one object per card.inspectsuggests the rules. - Whole sites.
map_sitelists a site's URLs.create_batchfetches many pages in one call. - The price first, for free.
check_sitetells the agent what a site costs, and how our routes did there lately, before it spends anything. - Blocks we detect are free. So are our own errors. When a page needs a browser, the answer says so, and the agent can decide.
Connect your agent
Claude Code, Cursor, VS Code, ChatGPT: add our MCP server. In Claude Code it is one command:
claude mcp add --transport http megalania https://mcp.megalania.io
You sign in with Google or GitHub in the browser. You copy no key. Not a tester yet? Apply for the beta first: sign-in works once your invite arrives. The other clients take one short block of config each, in MCP for agents.
- LangChain:
langchain-megalania, a page loader and five tools. See LangChain. - LlamaIndex:
llama-index-readers-megalaniafor documents, and our MCP server for agent tools. See LlamaIndex. - Vercel AI SDK:
@megalania/ai-sdk, tools forgenerateTextandstreamText. See Vercel AI SDK. - CrewAI and other MCP clients: connect to the same MCP server, with a key in the
Authorizationheader, never in the URL. See CrewAI web scraping.
Load pages in five lines
This is LangChain. Each page comes back as one Document, in Markdown, with its cost and status in the metadata. The pages are on our test store, the Lab.
pip install langchain-megalania export MEG_API_KEY='meg_sk_...'
from langchain_megalania import MegalaniaLoader
docs = MegalaniaLoader(["https://lab.megalania.io/about/", "https://lab.megalania.io/faq/"]).load()
for d in docs:
print(d.metadata["source"], d.metadata["credits_charged"], d.metadata["target_status"], len(d.page_content))
print(docs[1].page_content[:400])What it printed on our dev cluster on 16 Nov 2026:
https://lab.megalania.io/about/ 1 200 1265 https://lab.megalania.io/faq/ 1 200 1127 # FAQ Is this a real shop? No. Gibber & Co. is a demo store by Megalania for testing web scraping. Nothing here is for sale. Checkout is switched off and no payment can be made. How long does delivery take? Standard delivery takes 3 to 5 working days. See [Shipping](https://lab.megalania.io/shipping/) for all options. Can I return something? Yes, within 60 days if it is unused. See [Returns]
Each page cost 1 credit. More in Web scraping with LangChain.
Caps the model cannot raise
- A cap per page.
max_creditsis the most one page may cost. We skip every route above it. If no route is left, the answer is an error before any try, and it costs nothing. - Set by you, not the model. In
langchain-megalaniaand@megalania/ai-sdkyou set the cap when you create the tools. The model cannot change it. - Over MCP, the model fills
max_credits. So the account owner can set a monthly spend cap. A request past it is refused, for free. - A budget per run. Keep one in your own tool, and pass what is left as
max_credits. Cap what an AI agent spends on web pages shows the code.
Limits
- Page text is untrusted. A page can hide instructions for a model. Our tool descriptions tell the agent never to follow them. Keep that rule in your own prompts too.
- Public pages only. We do not log in, and we do not solve CAPTCHAs. When a page needs either, we stop and say so, for free.
- Browser pages cost more. A page that needs JavaScript runs in a real Chrome.
check_siteshows the price; plans are on the pricing page.