What you get
- The main text only.
markdownkeeps headings, lists, tables and links, and drops the menu, the footer and the scripts. Fewer tokens, better matches. - A whole site in one call. A crawl follows links from one page, on the same site, up to
max_depthandmax_pages.exclude_pathskeeps out what you do not want in the index. - The source of every page. Each page comes back with its URL, so every chunk can cite where it came from.
- Files too. PDF, Word, PowerPoint and Excel links come back as Markdown the same way.
- Pages that need JavaScript.
engine=browserruns a real Chrome, for docs sites that draw their text in the browser. It costs more;check_siteshows the price for free.
Crawl, clean and chunk in one script
This uses our Python SDK. It crawls the journal of our test store, the Lab, and writes one JSON line per section.
pip install megalania export MEG_API_KEY='meg_sk_...'
import json
import os
from megalania import Megalania
CRAWL = {"max_depth": 1, "max_pages": 40, "exclude_paths": ["/feed/", "/**/feed/", "/product/**", "/product-category/**", "/shop/**", "/cart/", "/checkout/", "/my-account/", "/tag/**", "/category/**", "/page/**", "/journal/page/**"]}
chunks, pages = [], 0
with Megalania(os.environ["MEG_API_KEY"]) as meg:
for text, meta in meg.read_pages("https://lab.megalania.io/journal/", crawl=CRAWL, response_types="markdown"):
pages += 1
for part in text.split("\n## ")[1:]:
heading, _, body = part.partition("\n")
chunks.append({"source": meta["source"], "heading": heading.strip(), "text": body.strip()})
with open("chunks.jsonl", "w") as f:
f.writelines(json.dumps(c, ensure_ascii=False) + "\n" for c in chunks)
print(pages, len(chunks), len({c["source"] for c in chunks}))What it printed on our dev cluster on 16 Nov 2026:
15 31 11
15 pages, 31 chunks from the 11 pages that have sections, for 15 credits: 1 per page. One chunk from chunks.jsonl:
{"source": "https://lab.megalania.io/caring-for-vegetable-tanned-leather/", "heading": "Don’t", "text": "- Dry it on a radiator. It will crack.\n- Use household cleaners or baby wipes.\n- Panic about scratches. ..."}- Exclude the feeds. Our first run had no
/feed/rule. The crawl followed the RSS link, and 22 of its 53 chunks came from the feed: the same posts again, as excerpts. Two patterns fixed it. - Split at headings. A section is a natural chunk: one topic, with its title. Keep the heading with the text, and the URL with both.
- Then embed. Any embedding model and any vector store take this file. In LangChain or LlamaIndex, our loaders return one document per page with the source in its metadata; split it with your framework's own splitter.
Keep the index fresh
- Crawl again on a schedule. A batch is priced page by page; send the crawl again and replace the chunks of each source URL.
- Or watch only what changes. A monitor checks a list of pages and sends a signed webhook with the pages that changed. See Monitors.
- Or list first.
map_sitereads the sitemap and returns every URL for 4 credits on most sites, so you fetch only new pages.
Limits
- Page text is untrusted. A page can hide instructions for a model. Keep retrieved text in the data part of your prompt, never in the instructions.
- Public pages only. We do not log in, and we do not solve CAPTCHAs. When a page needs either, we stop and say so, for free.
- Your right to use the text. Read each site's terms before you put its pages in an index you share or sell.
- Browser pages cost more. Plans and prices are on the pricing page.