The short answer: for LlamaIndex web scraping, pip install llama-index-readers-megalania and call MegalaniaReader().load_data(urls). Each page comes back as one LlamaIndex Document, as clean Markdown, with its cost and status in the metadata. On our test store, lab.megalania.io, the Shipping and Returns pages cost 1 credit each and split into 7 nodes, one per heading. For an agent, connect our MCP server with McpToolSpec, keep only the tools it needs, and fix the most one page may cost.
The free way first
LlamaIndex has a reader for plain pages: SimpleWebPageReader in llama-index-readers-web. It downloads the HTML and turns it into text.
from llama_index.readers.web import SimpleWebPageReader
docs = SimpleWebPageReader(html_to_text=True).load_data(["https://lab.megalania.io/shipping/"])
print(len(docs[0].text), docs[0].text.find("# Shipping"))It works on the Lab. But it keeps the whole page: the Shipping page came back as 4,061 characters, and its heading came after 2,154 characters of menu and links. The same page is 633 characters with our reader. You clean the rest up yourself before it goes into an index or a prompt.
Use our reader when you want the main content as Markdown, when a site blocks plain requests, or when a page needs a browser.
Load pages as LlamaIndex documents
pip install llama-index-readers-megalania export MEG_API_KEY='meg_sk_...'
from llama_index.core.node_parser import MarkdownNodeParser
from llama_index.readers.megalania import MegalaniaReader
docs = MegalaniaReader().load_data(["https://lab.megalania.io/shipping/", "https://lab.megalania.io/returns/"])
for d in docs:
print(d.metadata["source"], d.metadata["credits_charged"], d.metadata["target_status"], len(d.text))
nodes = MarkdownNodeParser().get_nodes_from_documents(docs)
print(len(nodes), "nodes")
for n in nodes:
print(" ", n.metadata["header_path"], "|", n.text.splitlines()[0])What it printed:
https://lab.megalania.io/shipping/ 1 200 633 https://lab.megalania.io/returns/ 1 200 801 7 nodes / | # Shipping /Shipping/ | ## When will my order leave? /Shipping/ | ## International / | # Returns /Returns/ | ## How to return /Returns/ | ## Faults and repairs /Returns/ | ## What we can’t take back
textis Markdown: headings, text, tables and links, without the menu. SoMarkdownNodeParsercan split at real headings, and each node knows its place inheader_path.- The metadata has
source,request_id,credits_chargedandtarget_status. - The key comes from
MEG_API_KEY.
Any key of the request body works as a keyword, for example country_code="DE" or response_types="text". A blocked page raises MegalaniaError with code blocked, and it is free. With MegalaniaReader(continue_on_failure=True) the reader skips that page and goes on.
To index the nodes, use any embedding model you already have: VectorStoreIndex(nodes). That step calls your model provider, not us, so we did not run it for this post.
A whole site
Pass one start page and crawl. The reader runs it as a batch, waits, and returns one document per page, with depth in the metadata:
docs = MegalaniaReader().load_data("https://lab.megalania.io/journal/", crawl={"max_pages": 20, "max_depth": 1})On the Lab that gave 19 pages: the journal page at depth 0 and 18 pages it links to. A crawl follows links in page order, so menu pages come first. To pick pages, list the site first: How to scrape every URL from a sitemap. For chunking a docs site, see Crawl a docs site into Markdown for RAG.
Web tools for a LlamaIndex agent
LlamaIndex agents reach our MCP server through llama-index-tools-mcp. No extra package from us. The server has 15 tools; an agent rarely needs all of them. allowed_tools keeps three, and partial_params_by_tool fixes max_credits, the most one page may cost:
pip install llama-index-tools-mcp
import asyncio
import json
import os
from llama_index.tools.mcp import BasicMCPClient, McpToolSpec
client = BasicMCPClient("https://mcp.megalania.io/", headers={"Authorization": f"Bearer {os.environ['MEG_API_KEY']}"})
cap = {"max_credits": 8}
spec = McpToolSpec(client=client, allowed_tools=["check_site", "scrape", "extract"],
partial_params_by_tool={"scrape": cap, "extract": cap})
async def main():
tools = {t.metadata.name: t for t in await spec.to_tool_list_async()}
print(sorted(tools))
site = await tools["check_site"].acall(host="lab.megalania.io")
print(json.loads(site.raw_output.content[0].text)["price"]["quotes"])
rates = await tools["extract"].acall(url="https://lab.megalania.io/shipping/",
extract="rates:table tbody tr{service:td:nth-child(1)|time:td:nth-child(2)|price:td:nth-child(3)}*")
print(rates.raw_output.content[0].text)
browser = await tools["scrape"].acall(url="https://lab.megalania.io/faq/", engine="browser")
print(browser.raw_output.content[0].text[:170])
asyncio.run(main())We called the tools ourselves, to see what the agent will see:
['check_site', 'extract', 'scrape']
{'browser': 15, 'browser_residential': 32, 'datacenter': 1, 'residential': 8}
{"rates":[{"service":"Standard","time":"3 to 5 working days","price":"$6.00, free over $75"},{"service":"Express","time":"1 to 2 working days","price":"$14.00"},{"service":"Collect in store","time":"Ready the next working day","price":"Free"}]}
credits_charged: 2
Failed: max credits too low.
max_credits is below the lowest price on this site, 15 credits. You are not charged. Raise max_credits to at least 15, or remove it. Read mescheck_siteis free. It gives the price of each route on a host, and our recent results there. An agent can ask before it spends anything.extractreturns fields, not a page. Three rows of JSON for 2 credits: 1 for the page and 1 for the JSON. More on rules in How to scrape an HTML table into CSV.- The cap held. The browser costs 15 credits on this site, above the cap of 8, so the call stopped before any try, for free. The message tells the agent what to do next.
Then give the same spec to any LlamaIndex agent. Add this to agent_tools.py and run asyncio.run(ask()) instead of main():
pip install llama-index-llms-openai
from llama_index.core.agent.workflow import FunctionAgent
from llama_index.llms.openai import OpenAI
async def ask():
agent = FunctionAgent(tools=await spec.to_tool_list_async(), llm=OpenAI(model="gpt-4.1"))
print(await agent.run("What does express shipping cost at https://lab.megalania.io/shipping/?"))The agent’s model call needs your own model key, and we did not run it for this post.
How far the cap goes
partial_params_by_tool takes max_credits out of the schema the model sees, and sends your value on every call. That is the right default. It is not a lock: in our test, a call that passed max_credits=100 by hand won over the partial value. A model only sends the fields it sees, but if you need a hard limit, check credits_charged in each answer and stop the run yourself. All the rules are in Credits.
What it costs
| Step | Pages | Credits |
|---|---|---|
MegalaniaReader on Shipping and Returns |
2 | 2 |
A crawl from the journal page, max_pages 20 |
19 | 19 |
check_site on the Lab |
— | 0 |
extract on the shipping table |
1 | 2 |
scrape with the browser, over the cap |
— | 0 |
| A blocked page or one of our errors | — | 0 |
These prices are for this site on the datacenter route. A site that blocks it costs more; check_site shows how much. Plans are on the pricing page.
How we measured: our dev cluster on 16 Nov 2026, the Lab store on our dev domain, llama-index-readers-megalania 0.1.1 and llama-index-tools-mcp 0.7.0 from PyPI, llama-index-core 0.14.26, one run of each script above. The free reader is llama-index-readers-web 0.8.0.
Limits
- Python 3.10 or newer for the reader.
- The cap is per page. One
create_batchcall can cost up tomax_creditstimes its pages, so this example leaves batch tools out. - Blocked pages are not content. A page we could not get raises
MegalaniaError, free. Usecontinue_on_failure=Truein a long list.
The setup is also in LlamaIndex. Fetch public pages only. Read the site’s terms and robots.txt, and follow them.