MegalaniaBlog
  1. Blog
  2. AI agents
  3. Haystack

Haystack web scraping: pages as Markdown, MCP tools with a cap

Feed web pages into a Haystack pipeline as Markdown split at headings, and give a Haystack Agent MCP web tools with a cost cap. Tested on Haystack 3.5.

Jordan HayesDeveloper Relations
6 min read
AI agents
A Haystack pipeline of three boxes, fetch, split and write, with Markdown chunks falling into a store. Beside it, an Agent box with a hook tag that reads max 20 and a page card with 12 products.

The short answer: for Haystack web scraping, use two things. For a pipeline, a small custom component calls our Python SDK and returns each page as a Haystack Document in Markdown, so MarkdownHeaderSplitter can split it at real headings. For an agent, MCPToolset from mcp-haystack loads our hosted MCP tools, and a before_tool hook sets max_credits, so the model cannot raise what a page may cost. On our test store, lab.megalania.io, two pages became 7 chunks for 2 credits, and the agent read a JavaScript page with 12 products for 18 credits.

The free way first

Haystack can read a page on its own: LinkContentFetcher downloads it, and HTMLToDocument keeps the main text.

free.py
from haystack.components.converters import HTMLToDocument
from haystack.components.fetchers import LinkContentFetcher
from haystack.components.preprocessors import MarkdownHeaderSplitter

streams = LinkContentFetcher().run(urls=["https://lab.megalania.io/shipping/"])["streams"]
doc = HTMLToDocument().run(sources=streams)["documents"][0]
chunks = MarkdownHeaderSplitter().run(documents=[doc])["documents"]
print(len(streams[0].data), "bytes of HTML ->", len(doc.content), "characters,", doc.content.count("#"), "heading marks,", len(chunks), "chunk(s)")
Output
55411 bytes of HTML -> 604 characters, 0 heading marks, 1 chunk(s)

That is good. HTMLToDocument uses trafilatura (pip install trafilatura), and it dropped the menus and kept the shipping table. It also dropped the headings: the page’s title was gone and its section titles became plain lines, so the splitter made one chunk of the whole page. Use it when that is enough, it is free.

Use our API when you want the headings for chunking, when a site blocks plain requests, or when a page needs a browser.

A pipeline: fetch, split at headings, write

cURL
pip install haystack-ai megalania
export MEG_API_KEY='meg_sk_...'
pipeline.py
from haystack import Document, Pipeline, component
from haystack.components.preprocessors import MarkdownHeaderSplitter
from haystack.components.retrievers.in_memory import InMemoryBM25Retriever
from haystack.components.writers import DocumentWriter
from haystack.document_stores.in_memory import InMemoryDocumentStore
from megalania import Megalania


@component
class MegalaniaFetcher:
    @component.output_types(documents=list[Document])
    def run(self, urls: list[str]):
        with Megalania() as meg:
            pages = meg.read_pages(urls)
            return {"documents": [Document(content=text, meta=meta) for text, meta in pages]}


store = InMemoryDocumentStore()
index = Pipeline()
index.add_component("fetch", MegalaniaFetcher())
index.add_component("split", MarkdownHeaderSplitter())
index.add_component("write", DocumentWriter(document_store=store))
index.connect("fetch", "split")
index.connect("split", "write")

result = index.run({"fetch": {"urls": ["https://lab.megalania.io/shipping/", "https://lab.megalania.io/returns/"]}})
print(result["write"]["documents_written"], "chunks written")
for d in store.filter_documents():
    print(" ", d.meta["source"], d.meta["credits_charged"], "|", d.content.splitlines()[0])

hits = InMemoryBM25Retriever(document_store=store, top_k=1).run(query="Do you ship to Canada?")
print(hits["documents"][0].content)
Output
7 chunks written
  https://lab.megalania.io/shipping/ 1 | # Shipping
  https://lab.megalania.io/shipping/ 1 | ## When will my order leave?
  https://lab.megalania.io/shipping/ 1 | ## International
  https://lab.megalania.io/returns/ 1 | # Returns
  https://lab.megalania.io/returns/ 1 | ## How to return
  https://lab.megalania.io/returns/ 1 | ## Faults and repairs
  https://lab.megalania.io/returns/ 1 | ## What we can’t take back
## International

We ship to Canada and Mexico. Duties and taxes are paid by the receiver.

*This is a demo store. No order is ever shipped.*
  • read_pages asks for Markdown and yields each page’s text with source, request_id, credits_charged and target_status. The key comes from MEG_API_KEY.
  • Each chunk knows its place. MarkdownHeaderSplitter puts the section in meta["header"] and the headings above it in meta["parent_headers"].
  • BM25 matches words, not meaning. We first asked “How long does international shipping take?”, and the top chunk was “Faults and repairs”: its text says “as long as you own it”. With the page’s own words, “ship to Canada”, the right chunk came first. For questions in your users’ words, use an embedding retriever.
Free beta, by invite76,000 credits for 30 days, no card. Apply, and we email your invite.Apply for the beta

An Agent with MCP tools and a cap

cURL
pip install haystack-ai mcp-haystack

The model below is scripted, so you can run the whole agent without a model key: it asks for three tool calls, then stops. In your own agent, chat_generator is your chat model.

agent.py
import json

from haystack import component
from haystack.components.agents import Agent, State
from haystack.components.agents.state.state_utils import replace_values
from haystack.dataclasses import ChatMessage, ToolCall
from haystack.hooks import hook
from haystack.utils import Secret
from haystack_integrations.tools.mcp import MCPToolset, StreamableHttpServerInfo

tools = MCPToolset(
    server_info=StreamableHttpServerInfo(url="https://mcp.megalania.io/", token=Secret.from_env_var("MEG_API_KEY")),
    tool_names=["check_site", "scrape", "extract"],
)


@hook
def cap(state: State) -> None:
    last = state.data["messages"][-1]
    calls = [ToolCall(c.tool_name, {**c.arguments, "max_credits": 20} if c.tool_name in {"scrape", "extract"} else c.arguments, c.id)
             for c in last.tool_calls]
    state.set("messages", state.data["messages"][:-1] + [ChatMessage.from_assistant(tool_calls=calls)], handler_override=replace_values)


PLAN = [
    ("check_site", {"host": "lab.megalania.io"}),
    ("scrape", {"url": "https://lab.megalania.io/new-arrivals/", "engine": "browser", "actions": [{"wait_for": "#item-12"}], "max_credits": 500}),
    ("scrape", {"url": "https://lab.megalania.io/new-arrivals/", "engine": "browser", "proxy_tier": "residential", "max_credits": 500}),
]


@component
class ScriptedModel:
    @component.output_types(replies=list[ChatMessage])
    def run(self, messages: list[ChatMessage], tools=None, **kwargs):
        done = sum(m.tool_call_result is not None for m in messages)
        if done < len(PLAN):
            name, args = PLAN[done]
            return {"replies": [ChatMessage.from_assistant(tool_calls=[ToolCall(name, args, f"call-{done}")])]}
        return {"replies": [ChatMessage.from_assistant("done")]}


agent = Agent(chat_generator=ScriptedModel(), tools=tools, hooks={"before_tool": [cap]})
result = agent.run(messages=[ChatMessage.from_user("List the new arrivals with prices.")])
print(sorted(t.name for t in tools))
for m in result["messages"]:
    r = m.tool_call_result
    if r is None:
        continue
    if r.error:
        print(r.origin.tool_name, "| error |", r.result.split("Failed: ")[1][:150].replace("\\n", " "))
        continue
    out = json.loads(r.result)["structuredContent"]
    if r.origin.tool_name == "check_site":
        print("check_site | ok | quotes", out["price"]["quotes"])
    else:
        print(r.origin.tool_name, "| ok |", out["content"].count("\n- "), "products |", out["route"], "| credits", out["credits_charged"])
Output
['check_site', 'extract', 'scrape']
check_site | ok | quotes {'browser': 15, 'browser_residential': 32, 'datacenter': 1, 'residential': 8}
scrape | ok | 12 products | browser/datacenter | credits 18
scrape | error | max credits too low. max_credits is below the lowest price on this site, 32 credits. You are not charged. Raise max_credits to at least 32, or remove
  • The key is a Secret read from MEG_API_KEY and sent as a bearer token on the connection, never in the URL.
  • Three tools, not 15. tool_names keeps the model’s prompt short, and keeps batches and monitors out of its reach.
  • The hook runs before every tool call. The model asked for max_credits: 500 twice; the hook sent 20. Only scrape and extract take max_credits, so it leaves check_site alone.
  • check_site is free, and it gave this host’s prices: 1 credit a page over plain HTTP, 15 in the browser, 32 in the browser on a residential exit.
  • The JavaScript page came back with all 12 products for 18 credits: 15 for the browser and 3 for the wait. 18 fits under 20.
  • The residential browser costs 32 on this host, so we refused it before fetching anything, for 0 credits. The Agent got the reason as a tool error, and a real model can choose a cheaper route or stop.

What it costs

Step Credits
check_site 0
A plain page as Markdown 1
A JavaScript page in the browser, with one action 18
A call refused by max_credits 0
A block we detect, or one of our errors 0

These prices are for our test store on the datacenter route. A site that blocks it costs more, and check_site shows how much. All the rules are in Credits. Plans are on the pricing page.

How we measured: our dev cluster on 16 Nov 2026, the Lab store on our dev domain, haystack-ai 3.5.0, mcp-haystack 2.1.2, megalania 0.3.0 and trafilatura 2.3.1 on Python 3.13. We ran the three scripts as above; the outputs are copied from those runs.

Limits

  • Page text is untrusted. A page can hide instructions for a model. Our tool descriptions tell the agent never to follow them. Say it in your system prompt too.
  • Public pages only. We do not log in, and we do not solve CAPTCHAs. When a page needs either, we stop and say so, for free.
  • An empty list can still be a charged answer. Our check for JavaScript shells misses pages that have a heading and some text. Count the rows you expected.

Every tool and argument is in MCP for agents. The same pattern in LlamaIndex is in LlamaIndex web scraping. Fetch public pages only. Read the site’s terms and robots.txt, and follow them.

Jordan HayesDeveloper RelationsPublished 16 Nov 2026 · measured on our dev cluster, 16 Nov 2026, haystack-ai 3.5.0, mcp-haystack 2.1.2, megalania 0.3.0, Python 3.13, our test store

See also

Keep reading