MegalaniaBlog
  1. Blog
  2. How to scrape
  3. Website to CSV

How to scrape a website to CSV

Turn a product list into a CSV file. One request per page with CSS rules returns JSON, a short Python loop follows the next link and writes the rows.

Jordan HayesDeveloper Relations
5 min read
How to scrape
A stack of product cards with a price tag. A dashed arrow leads to a spreadsheet with the columns name, price and sku.

The short answer: do not parse HTML yourself. Send each list page with response_types=json and a few CSS rules in extract. You get JSON back: one object per product, plus the link to the next page. A Python loop of 30 lines follows that link to the end and writes every row with the csv module. On our test store, lab.megalania.io/shop/, that was 8 pages and 72 products, for 16 credits in all: 2 credits per page.

What you need before you start

  • A list page: a category, a search result or a shop page, where each item is a card.
  • The fields you want from each card. Here: name, price, old price, rating, SKU and link.
  • A Megalania key in MEG_API_KEY. Never put a key in a URL.

Our example is the Lab, a test store we run for exactly this. It is a normal WooCommerce shop with 9 products per page and a “next page” link. Your site will have other class names, but the steps are the same. On any WooCommerce store, try the free Store API first: How to scrape any WooCommerce store.

Check a site first

Before a big run, ask what the site costs. It is free. Over MCP, your agent calls check_site. From cURL:

cURL
curl -s -H "Authorization: Bearer $MEG_API_KEY" \
  "https://get.megalania.io/v1/tier?host=lab.megalania.io"

For the Lab, the answer said 1 credit per page on the cheapest route (plain HTTP, datacenter IP). A site that blocks that route costs more: Check a site first explains every field.

Write the CSS rules

Open one list page in your browser and look at one product card with the developer tools. On a WooCommerce store, a card looks like this (shortened):

html
<li class="product ...">
  <a href="https://lab.megalania.io/product/amber-candle-trio/" class="woocommerce-loop-product__link">
    <h2>Amber Jar Candle Trio</h2>
    <strong class="rating">4.50</strong>
    <span class="price"><del><span class="amount">$42.00</span></del>
      <ins><span class="amount">$29.95</span></ins></span>
  </a>
  <a class="button" data-product_sku="GBR-HOM-031">Add to cart</a>
</li>

Each rule is name:css. @attr reads an attribute, and a * at the end returns every match as a list. li.product{...}* returns one object per card, and the fields inside {} look only inside that card. Every rule form is in Fetch, extract and lists:

extract
items:li.product{name:h2|price:.price > .amount*|was:.price del .amount|now:.price ins .amount|rating:.rating|sku:a.button@data-product_sku|url:a.woocommerce-loop-product__link@href}*|next:a.next@href
  • price is a list: one amount for a normal product, two for a price range like “$349.00 - $353.00”.
  • was and now are filled only when the product is on sale, else they are null.
  • next is the link to the next page, or null on the last page.

A field that is missing in one card is null in that card, so your columns never shift. If a rule matches more than one element with different values, the answer carries a hints list that says which field to narrow.

One page, one request (cURL)

cURL
curl -sS -G https://get.megalania.io/v1 \
  -H "Authorization: Bearer $MEG_API_KEY" \
  --data-urlencode "url=https://lab.megalania.io/shop/" \
  --data-urlencode "response_types=json" \
  --data-urlencode "extract=items:li.product{name:h2|price:.price > .amount*|was:.price del .amount|now:.price ins .amount|rating:.rating|sku:a.button@data-product_sku|url:a.woocommerce-loop-product__link@href}*|next:a.next@href"

The answer (shortened to two products):

JSON
{
  "items": [
    {"name": "20 cm Chef’s Knife", "price": ["$112.00"], "was": null, "now": null, "rating": "4.67", "sku": "GBR-KIT-023", "url": "https://lab.megalania.io/product/chef-knife/"},
    {"name": "Amber Jar Candle Trio", "price": [], "was": "$42.00", "now": "$29.95", "rating": "4.50", "sku": "GBR-HOM-031", "url": "https://lab.megalania.io/product/amber-candle-trio/"}
  ],
  "next": "https://lab.megalania.io/shop/page/2/"
}

The headers say what happened: X-Credits-Charged: 2, X-Route: http/datacenter.

Free beta, by invite76,000 credits for 30 days, no card. Apply, and we email your invite.Apply for the beta

Python: follow next, write the CSV

The loop asks for one page, adds its rows, and moves to next until there is none. It needs only requests.

shop_to_csv.py
import csv
import os
import requests

API = "https://get.megalania.io/v1"
KEY = os.environ["MEG_API_KEY"]
RULES = (
    "items:li.product{name:h2|price:.price > .amount*|was:.price del .amount|now:.price ins .amount"
    "|rating:.rating|sku:a.button@data-product_sku|url:a.woocommerce-loop-product__link@href}*"
    "|next:a.next@href"
)

url = "https://lab.megalania.io/shop/"
rows = []
spent = 0
while url:
    r = requests.get(API, params={"url": url, "response_types": "json", "extract": RULES},
                     headers={"Authorization": f"Bearer {KEY}"}, timeout=120)
    r.raise_for_status()
    spent += int(r.headers["X-Credits-Charged"])
    page = r.json()
    for item in page["items"]:
        price = item["now"] or " - ".join(item["price"])
        rows.append({"name": item["name"], "price": price, "was": item["was"] or "",
                     "rating": item["rating"], "sku": item["sku"], "url": item["url"]})
    url = page["next"]

with open("products.csv", "w", newline="", encoding="utf-8") as f:
    w = csv.DictWriter(f, fieldnames=["name", "price", "was", "rating", "sku", "url"])
    w.writeheader()
    w.writerows(rows)
print(f"{len(rows)} products, {spent} credits")

Run it:

cURL
pip install requests
export MEG_API_KEY='meg_sk_...'
python shop_to_csv.py

It printed 72 products, 16 credits. The first lines of products.csv:

products.csv
name,price,was,rating,sku,url
20 cm Chef’s Knife,$112.00,,4.67,GBR-KIT-023,https://lab.megalania.io/product/chef-knife/
Alpine Down Parka,$349.00 - $353.00,,4.50,GBR-OUT-052,https://lab.megalania.io/product/alpine-down-parka/
Amber Jar Candle Trio,$29.95,$42.00,4.50,GBR-HOM-031,https://lab.megalania.io/product/amber-candle-trio/

Open it in Excel, Google Sheets or pandas. We write UTF-8, so names with accents and quotes stay correct. csv.DictWriter quotes a field with a comma inside it for you.

What it costs

Each list page cost 2 credits: 1 for the page on the datacenter route, and 1 for the JSON output. The 8 pages of the Lab shop cost 16 credits. The price of a page depends on the site and the route that delivered it, so check your own site first. Blocks we detect cost 0, and errors cost 0.

List pages Products (9 a page) Credits on the datacenter route
8 72 16
100 900 200
1,000 9,000 2,000

At 2 credits a page, the 76,000-credit beta grant covers half that many list pages. Plans and prices are on the pricing page.

How we measured: one run on our dev cluster on 16 Nov 2026, the Lab store on our dev domain, every page on http/datacenter.

More pages, or pages that need a browser

Being polite

Fetch public pages only. Do not log in, and do not get past a password. Read the site’s terms and robots.txt, and follow them. Keep your rate low: one page at a time is enough for a few hundred pages. Do not collect personal data.