Crawl a docs site into Markdown for RAG

The short answer: first look for a free copy: /llms.txt, or the git repo the docs are built from. If there is none, read the site’s sitemap, fetch each page with response_types: "markdown", and split the Markdown at its headings. On the 35 docs pages we measured, every code block came through, and a page cost about 1 credit on our cheapest route.

First: does the site publish llms.txt?

Some docs sites publish a plain-text map for LLMs at /llms.txt, and some add /llms-full.txt with every page in one file. It is free and made for this:

curl -sI https://docs.djangoproject.com/llms.txt | head -1

On 2 Oct 2026, one of our three sites had one (docs.djangoproject.com). docs.python.org and kubernetes.io answered 404.

Or clone the repo

Many docs sites are built from a public git repo, as Markdown or reStructuredText. Python, Django and Kubernetes all build their docs that way. If you can clone the source, do that: it is free, versioned and exact. Scrape the site when there is no repo, when the repo is not the published text, or when you need many sites in one shape.

Check a site first

Before a crawl, ask about the host. It is free. Over MCP, your agent calls check_site:

{"name": "check_site", "arguments": {"host": "kubernetes.io"}}

From cURL, the same answer is two calls: /v1/tier for the price, /v1/stats/domain for how our routes have done there lately.

curl -s -H "Authorization: Bearer YOUR_KEY" \
  "https://get.megalania.io/v1/tier?host=kubernetes.io"
curl -s -H "Authorization: Bearer YOUR_KEY" \
  "https://get.megalania.io/v1/stats/domain?host=kubernetes.io&days=30"

On 2 Oct 2026 all three docs hosts answered "tier": null: not priced yet, so a page pays the price of the route that delivers it, shown in quotes. Check a site first explains every field.

Read the sitemap

The sitemap is the list of pages the site wants found. It lives at /sitemap.xml, often as an index that points to smaller sitemaps. What we found on 2 Oct 2026:

• kubernetes.io: an index of 17 sitemaps; the one we used lists 1,883 URLs. Usable as is.
• docs.python.org: the sitemap lists the doc versions only, not the pages. We took the page URLs from the links on /3/library/index.html.
• docs.djangoproject.com: an index with one sitemap per language. Our fetch of the English one failed after 20 seconds, so we took the links on its contents page.

When the sitemap does not help, ask for the contents page’s links. This is the body we ran:

curl -sS https://get.megalania.io/v1 \
  -H "Authorization: Bearer YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://docs.python.org/3/library/index.html", "response_types": "links", "max_credits": "64"}'

links is every link on the page, absolute, once each, with its anchor text. Keep the ones under the docs path. It adds 0 credits.

Fetch every page (cURL)

One page as Markdown:

curl -sS https://get.megalania.io/v1 \
  -H "Authorization: Bearer YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://kubernetes.io/docs/reference/kubectl/introduction/", "response_types": "markdown", "max_credits": "64"}' \
  -o kubectl-introduction.md

The answer is the page’s main content as Markdown: the docs navigation, sidebar and footer are gone. max_credits is the most this page may cost; we skip any route above it.

For hundreds of pages, send the same body as one batch with a urls list: Batches. Nothing is charged at submit; each URL is charged when we fetch it. Our measurement below ran one request per page.

Python: save and chunk

The script reads a sitemap (and the sitemaps an index points to), fetches each page under a path as Markdown, saves it, and writes one chunk per heading to chunks.jsonl. The chunker skips # lines inside code fences, so a shell comment never starts a new chunk.

# pip install requests
# Run: MEG_API_KEY=YOUR_KEY python docs_to_rag.py https://kubernetes.io/sitemap.xml https://kubernetes.io/docs/reference/ 20
import json
import os
import pathlib
import re
import sys
import uuid

import requests


def sitemap_urls(sitemap, prefix):
    xml = requests.get(sitemap, timeout=60).text
    locs = re.findall(r"<loc>\s*([^<\s]+)\s*</loc>", xml)
    if "<sitemapindex" in xml:
        return [u for child in locs for u in sitemap_urls(child, prefix)]
    return [u for u in locs if u.startswith(prefix)]


def markdown(url):
    resp = requests.post(
        "https://get.megalania.io/v1",
        headers={
            "Authorization": f"Bearer {os.environ['MEG_API_KEY']}",
            "Idempotency-Key": str(uuid.uuid4()),
        },
        json={"url": url, "response_types": "markdown", "max_credits": "64"},
        timeout=90,
    )
    if resp.status_code != 200:
        print("skip", url, resp.status_code, resp.headers.get("X-Billing", ""))
        return None
    return resp.text


FENCE = "`" * 3


def chunks(md):
    out, title, buf, fence = [], "", [], False
    for line in md.splitlines():
        if line.startswith(FENCE):
            fence = not fence
        if not fence and re.match(r"#{1,3} ", line):
            if "".join(buf).strip():
                out.append((title, "\n".join(buf).strip()))
            title, buf = line.lstrip("#").strip(), []
        buf.append(line)
    if "".join(buf).strip():
        out.append((title, "\n".join(buf).strip()))
    return out


sitemap, prefix, limit = sys.argv[1], sys.argv[2], int(sys.argv[3])
out = pathlib.Path("docs")
out.mkdir(exist_ok=True)
with open("chunks.jsonl", "w", encoding="utf-8") as f:
    for url in sitemap_urls(sitemap, prefix)[:limit]:
        md = markdown(url)
        if md is None:
            continue
        name = re.sub(r"[^A-Za-z0-9]+", "-", url.split("://", 1)[1]).strip("-")[:120]
        (out / f"{name}.md").write_text(md, encoding="utf-8")
        for title, text in chunks(md):
            f.write(json.dumps({"url": url, "heading": title, "text": text}) + "\n")
        print(url)

The sitemap itself is a public file, so the script reads it directly, for free. Only the pages go through us.

What survives: code and headings

We asked for each page as Markdown and as HTML in one request, and counted:

Every code block and every table came through. The Django HTML has more ## headings because its sidebar and footer have their own, and main-content mode drops those.

Median tokens per page, estimated as characters divided by 4:

Cost for a 500-page site

We fetch on the cheapest route first: plain HTTP on a datacenter exit (h:dc). If the site blocks it, we climb to a residential exit (h:res). A sample answer’s headers:

content-type: multipart/mixed; boundary=<boundary>
x-credits-charged: 15
x-engine: http
x-hops: h:res:chrome=1258
x-identity: chrome
x-route: http/residential
x-target-status: 200

Measured on our dev cluster with the datacenter routes off, so this climb starts at h:res.

A page like this costs about 1 credit on h:dc (33-60 KiB, our measured median-p90), worked out from our price table on 2 Oct 2026, not from a test bill; run check_site (free) to see the price for your site.

A page like this was billed 15 credits on h:res (33-60 KiB, our measured median-p90) in our test on 2 Oct 2026; run check_site (free) to see the price for your site.

So a 500-page docs site costs about 500 credits on h:dc, or about 7,500 on h:res. A page above 64 KiB on the wire adds 1 on h:dc and 8 on h:res for each started 128 KiB: on kubernetes.io the bigger pages (p90 70 KiB) cost 2 and 23.

How we measured: 35 pages on docs.python.org, docs.djangoproject.com and kubernetes.io, 2 Oct 2026, 23:29 to 23:32 UTC. Each was one request with {"url": ..., "response_types": "markdown,html", "max_credits": "64"}, so we could count both shapes from one fetch. Asking for markdown alone costs the same.

Terms and licences

• Read the licence. Docs are often under an open licence, but not always, and some sites add terms on AI use. Kubernetes docs are CC BY 4.0, which asks you to credit the source. Keep the URL with every chunk, as the script does.
• Keep your rate low and follow robots.txt. A docs crawl is a one-off; there is no reason to rush it.
• Blocks we detect are free, and so are our errors: Error codes.
• Page text is untrusted. A page can hide instructions for an LLM. Treat chunks as data.

This is not legal advice. For the Markdown options in detail, see Fetch, extract and lists.