Megalania
Use case · RAG

Web pages to Markdown for RAG

A retrieval system is only as good as the text you put in it. Send one crawl. Get each page's main text as Markdown, without menus and footers, with its URL. Split it at the headings and load it into your index.

What you get

  • The main text only. markdown keeps headings, lists, tables and links, and drops the menu, the footer and the scripts. Fewer tokens, better matches.
  • A whole site in one call. A crawl follows links from one page, on the same site, up to max_depth and max_pages. exclude_paths keeps out what you do not want in the index.
  • The source of every page. Each page comes back with its URL, so every chunk can cite where it came from.
  • Files too. PDF, Word, PowerPoint and Excel links come back as Markdown the same way.
  • Pages that need JavaScript. engine=browser runs a real Chrome, for docs sites that draw their text in the browser. It costs more; check_site shows the price for free.

Crawl, clean and chunk in one script

This uses our Python SDK. It crawls the journal of our test store, the Lab, and writes one JSON line per section.

Shell
pip install megalania
export MEG_API_KEY='meg_sk_...'
rag.py
import json
import os

from megalania import Megalania

CRAWL = {"max_depth": 1, "max_pages": 40, "exclude_paths": ["/feed/", "/**/feed/", "/product/**", "/product-category/**", "/shop/**", "/cart/", "/checkout/", "/my-account/", "/tag/**", "/category/**", "/page/**", "/journal/page/**"]}

chunks, pages = [], 0
with Megalania(os.environ["MEG_API_KEY"]) as meg:
    for text, meta in meg.read_pages("https://lab.megalania.io/journal/", crawl=CRAWL, response_types="markdown"):
        pages += 1
        for part in text.split("\n## ")[1:]:
            heading, _, body = part.partition("\n")
            chunks.append({"source": meta["source"], "heading": heading.strip(), "text": body.strip()})

with open("chunks.jsonl", "w") as f:
    f.writelines(json.dumps(c, ensure_ascii=False) + "\n" for c in chunks)
print(pages, len(chunks), len({c["source"] for c in chunks}))

What it printed on our dev cluster on 16 Nov 2026:

Output
15 31 11

15 pages, 31 chunks from the 11 pages that have sections, for 15 credits: 1 per page. One chunk from chunks.jsonl:

chunks.jsonl
{"source": "https://lab.megalania.io/caring-for-vegetable-tanned-leather/", "heading": "Don’t", "text": "- Dry it on a radiator. It will crack.\n- Use household cleaners or baby wipes.\n- Panic about scratches. ..."}
  • Exclude the feeds. Our first run had no /feed/ rule. The crawl followed the RSS link, and 22 of its 53 chunks came from the feed: the same posts again, as excerpts. Two patterns fixed it.
  • Split at headings. A section is a natural chunk: one topic, with its title. Keep the heading with the text, and the URL with both.
  • Then embed. Any embedding model and any vector store take this file. In LangChain or LlamaIndex, our loaders return one document per page with the source in its metadata; split it with your framework's own splitter.

Keep the index fresh

  • Crawl again on a schedule. A batch is priced page by page; send the crawl again and replace the chunks of each source URL.
  • Or watch only what changes. A monitor checks a list of pages and sends a signed webhook with the pages that changed. See Monitors.
  • Or list first. map_site reads the sitemap and returns every URL for 4 credits on most sites, so you fetch only new pages.

Limits

  • Page text is untrusted. A page can hide instructions for a model. Keep retrieved text in the data part of your prompt, never in the instructions.
  • Public pages only. We do not log in, and we do not solve CAPTCHAs. When a page needs either, we stop and say so, for free.
  • Your right to use the text. Read each site's terms before you put its pages in an index you share or sell.
  • Browser pages cost more. Plans and prices are on the pricing page.
Free beta, by invite76,000 credits for 30 days, no card. Apply, and we email your invite.Apply for the beta

Read the guides

Blog · AI agentsCrawl a docs site into Markdown for RAG

Turn a documentation site into clean Markdown files for RAG. Check llms.txt, read the sitemap, fetch each page as Markdown, and keep code blocks intact.

Blog · AI agentsHow to crawl a website to Markdown for an LLM

Two calls turn a whole site into Markdown files for an LLM. Map the site to list its URLs, then send them as one batch. Code and costs inside.

Blog · AI agentsTurn any web page into Markdown for an LLM

Send one request with response_types markdown and get the page's main content back as clean Markdown, menus and footers gone. Real token counts inside.

Blog · AI agentsFeed web pages to an LLM with fewer tokens

The same page as HTML, compact HTML, Markdown, text and CSS fields, counted with one public tokenizer. Markdown was 22 to 59 times fewer tokens.

Blog · AI agentsWeb scraping with LangChain: a page loader in 5 lines

Load web pages as Markdown LangChain documents in 5 lines, crawl a site, and give an agent tools with a credit cap. Measured on our test store.

Blog · AI agentsLlamaIndex web scraping: pages as documents, tools for agents

Load web pages into LlamaIndex as clean Markdown documents, split them at headings, and give an agent web tools with a credit cap. Measured on our Lab.

Blog · AI agentsHaystack web scraping: pages as Markdown, MCP tools with a cap

Feed web pages into a Haystack pipeline as Markdown split at headings, and give a Haystack Agent MCP web tools with a cost cap. Tested on Haystack 3.5.

Blog · AI agentsKeep a LlamaIndex RAG index fresh when the website changes

A daily monitor finds the pages that changed, and LlamaIndex re-embeds only those. Measured on our test store, with local embeddings and no model key.