MegalaniaBlog
  1. Blog
  2. How to scrape
  3. Sitemap URLs

How to scrape every URL from a sitemap

Read sitemap.xml and sitemap indexes in Python with the standard library, for free. Or one map call when a site blocks you. Real counts on our test store.

Jordan HayesDeveloper Relations
6 min read
How to scrape
A robots.txt card points to a sitemap index. It opens into seven sitemap files that pour into one list, with a card that reads 178 URLs.

The short answer: to scrape a sitemap XML file in Python, read /robots.txt for the Sitemap: line, download that file, and follow it down when it is a sitemap index. The standard library does it in 40 lines, for free. When a site blocks your plain requests, or you do not want to keep that code, send the home page to POST /v1/map and get the same list back in one call. On our test store, lab.megalania.io, the free script found 178 URLs in 8 sitemap files, and the map found the same 178 plus 6 from the home page, for 4 credits.

What a sitemap is, and where to find it

A sitemap is an XML list of a site’s pages that the owner keeps for search engines. There are two kinds:

  • A sitemap (<urlset>): one <url><loc> per page.
  • A sitemap index (<sitemapindex>): one <sitemap><loc> per sitemap file. Big sites split their list this way, often by type: posts, pages, products, tags.

To find it, open /robots.txt. Most sites name their sitemap there:

lab.megalania.io/robots.txt (the end)
Sitemap: https://lab.megalania.io/wp-sitemap.xml

With no Sitemap: line, try /sitemap.xml. On the Lab, wp-sitemap.xml is an index of 7 sitemaps: posts, pages, products, post categories, post tags, product categories and product tags.

Scrape sitemap XML in Python, for free

No API, only the standard library. The script reads robots.txt, follows every sitemap index down to the real sitemaps, unzips .xml.gz files, and keeps each URL once.

sitemap_urls.py
import gzip
import sys
import urllib.request
import xml.etree.ElementTree as ET

NS = {"sm": "http://www.sitemaps.org/schemas/sitemap/0.9"}


def get(url):
    req = urllib.request.Request(url, headers={"User-Agent": "sitemap-reader/1.0"})
    with urllib.request.urlopen(req, timeout=30) as r:
        body = r.read()
    return gzip.decompress(body) if body[:2] == b"\x1f\x8b" else body


def sitemap_urls(url, seen):
    if url in seen:
        return []
    seen.add(url)
    root = ET.fromstring(get(url))
    if root.tag.endswith("sitemapindex"):
        urls = []
        for loc in root.findall("sm:sitemap/sm:loc", NS):
            urls += sitemap_urls(loc.text.strip(), seen)
        return urls
    return [loc.text.strip() for loc in root.findall("sm:url/sm:loc", NS)]


def sitemaps_from_robots(site):
    text = get(site + "robots.txt").decode("utf-8", "replace")
    found = [line.split(":", 1)[1].strip() for line in text.splitlines()
             if line.lower().startswith("sitemap:")]
    return found or [site + "sitemap.xml"]


site = sys.argv[1] if len(sys.argv) > 1 else "https://lab.megalania.io/"
seen, urls = set(), []
for sm in sitemaps_from_robots(site):
    urls += sitemap_urls(sm, seen)
urls = list(dict.fromkeys(urls))
print(f"{len(seen)} sitemap files, {len(urls)} URLs")
with open("urls.txt", "w") as f:
    f.write("\n".join(urls) + "\n")

Run it:

cURL
python sitemap_urls.py https://lab.megalania.io/

What it printed:

Output
8 sitemap files, 178 URLs

urls.txt now holds 178 URLs: 72 products, 8 product categories, 41 product tags, 4 post categories, 26 post tags, and 27 posts and pages. The seen set stops a loop when an index lists itself, and dict.fromkeys drops a URL that two sitemaps both list.

For many sites, this is all you need. It costs nothing.

Free beta, by invite76,000 credits for 30 days, no card. Apply, and we email your invite.Apply for the beta

When the free way stops working

  • The site blocks you. A plain request from a server IP can get a 403 or a challenge page instead of XML. Then ET.fromstring fails on HTML.
  • There is no sitemap. Then the only list is the links on the pages.
  • You do not want the code. Indexes, gzip, a robots.txt with three Sitemap: lines, a loop between two indexes: each one is a small fix you keep.

For these, one call does the same work.

One map call (cURL)

cURL
curl -sS -X POST https://get.megalania.io/v1/map \
  -H "Authorization: Bearer $MEG_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://lab.megalania.io/", "limit": 5000}'
JSON
{
  "urls": [
    {"url": "https://lab.megalania.io/how-to-pitch-a-tent-in-the-rain/", "source": "sitemap"},
    {"url": "https://lab.megalania.io/?add-to-cart=73", "source": "link"}
  ],
  "truncated": false
}

The map reads the start page, /robots.txt, the sitemaps it names (or /sitemap.xml), and the sitemaps an index lists, gzipped ones too, up to 10 files. source says where each URL came from: sitemap first, then link for the start page’s links that the sitemaps did not have. When the datacenter route is blocked, the map tries residential. It never runs a browser and never fetches the pages it lists. Every detail is in Map a site.

With no sitemap at all, the map still gives you the start page’s links. That is a start, not the whole site. For every page, crawl instead: How to crawl a website to Markdown for an LLM.

Python: the whole site, then only the products

search keeps only the URLs that contain a text. Here it picks the product pages.

map_site.py
import os
from collections import Counter
from urllib.parse import urlsplit

import requests

API = "https://get.megalania.io"
HEADERS = {"Authorization": f"Bearer {os.environ['MEG_API_KEY']}"}
SITE = "https://lab.megalania.io/"


def map_site(**body):
    r = requests.post(f"{API}/v1/map", json={"url": SITE, "limit": 5000, **body}, headers=HEADERS, timeout=120)
    r.raise_for_status()
    return r.json()["urls"], int(r.headers["X-Credits-Charged"])


urls, cost = map_site()
print(f"map: {len(urls)} URLs, {cost} credits")
print(" by source:", dict(Counter(u["source"] for u in urls)))

products, cost = map_site(search="/product/")
print(f"search=/product/: {len(products)} URLs, {cost} credits")
with open("products.txt", "w") as f:
    f.write("\n".join(u["url"] for u in products) + "\n")

Run it:

cURL
pip install requests
export MEG_API_KEY='meg_sk_...'
python map_site.py

What it printed:

Output
map: 184 URLs, 4 credits
 by source: {'sitemap': 178, 'link': 6}
search=/product/: 72 URLs, 4 credits

The 178 sitemap URLs are the same 178 the free script found. The 6 link URLs came from the home page: 5 “Add to cart” links and the RSS feed. Neither is a page you want to fetch, which is why the sitemap is the better list.

products.txt holds the 72 product pages. Send them as one batch to fetch them: Batches. To turn them into rows, see How to scrape a website to CSV.

From an AI agent (MCP)

The MCP tool map_site takes the same url, limit and search. We asked it for search: "/product-category/" on the Lab, and it answered: “Found 8 URLs: 8 from sitemaps, 0 from links on the start page”, then the 8 category URLs, for 4 credits. An agent can map first, then pick the pages it really needs.

What it costs

Step Credits
Your own script on a site that lets you in 0
A map, on the datacenter route 4
A map with search 4, the same
A map that finds no URL because the start page was blocked or an error 0

These prices are for this site on the datacenter route. A map on the residential route costs that route’s prices on top. Check a site first with GET /v1/tier?host=... or the MCP tool check_site; both are free. Plans are on the pricing page.

How we measured: our dev cluster on 16 Nov 2026, the Lab store on our dev domain, one run of each script above and one MCP call.

Limits

  • A sitemap is what the owner put in it. It can miss pages, and it can list old ones. Expect a few 404s when you fetch the list.
  • Not every URL is a page you want. The Lab’s sitemap lists /cart/, /checkout/ and /my-account/. Drop them before you fetch.
  • The map reads at most 10 sitemap files and returns at most 5,000 URLs. truncated: true means there may be more. A sitemap index with hundreds of files needs your own script, or several maps with search.
  • Only the same site. The map keeps the site’s own host, and its subdomains only with include_subdomains.

Fetch public pages only. Read the site’s terms and robots.txt, and follow them.

Jordan HayesDeveloper RelationsPublished 16 Nov 2026 · measured on our dev cluster, 16 Nov 2026, one sitemap index and one site map of our test store

Keep reading