How to scrape product data from any store

The short answer: most product pages carry a hidden <script type="application/ld+json"> block with the product’s name, SKU, brand, price, currency and stock. Search engines read it, so stores keep it correct. Ask for response_types: "page_data" and you get every JSON-LD block on the page as JSON, at no extra cost. No selectors, any platform.

Where the data hides: JSON-LD

A Product block looks like this. The product is made up; the keys are the schema.org ones stores use:

{
  "@context": "https://schema.org",
  "@type": "Product",
  "name": "Trail boot, waterproof",
  "sku": "TB-200-43",
  "brand": {"@type": "Brand", "name": "Example Outdoor"},
  "offers": {"@type": "Offer", "price": "79.99", "priceCurrency": "USD",
             "availability": "https://schema.org/InStock"},
  "aggregateRating": {"@type": "AggregateRating", "ratingValue": "4.6", "reviewCount": "212"}
}

Real pages make it harder than this. The block can sit inside an @graph list, next to BreadcrumbList and Organization blocks. Some stores use ProductGroup with one hasVariant per size, or an AggregateOffer with a lowPrice instead of a price. The Python below handles all of these.

The free way: parse it yourself

If a plain request gets the page, you do not need us. Fetch it with any HTTP client, find the application/ld+json scripts, and json.loads each one. Open-source parsers for structured data do the same. That is free, and you should try it first.

It stops being enough when the store answers your script with a 403, a bot check or an empty page. A plain requests.get() does not look like a real browser on the wire, and big retailers notice.

Check a site first

Before a big run, ask about the host. It is free. Over MCP, your agent calls check_site:

{"name": "check_site", "arguments": {"host": "www.ikea.com"}}

From cURL, the same answer is two calls: /v1/tier for the price, /v1/stats/domain for how our routes have done there lately.

curl -s -H "Authorization: Bearer YOUR_KEY" \
  "https://get.megalania.io/v1/tier?host=www.ikea.com"
curl -s -H "Authorization: Bearer YOUR_KEY" \
  "https://get.megalania.io/v1/stats/domain?host=www.ikea.com&days=30"

On 2 Oct 2026 all four retailers below answered "tier": null: not priced yet, so a page pays the price of the route that delivers it, shown in quotes. A host with few requests answers "requests": 0 from /v1/stats/domain; that means no data, not a bad site. Check a site first explains every field.

First request (cURL)

The body is the exact one we ran:

curl -sS https://get.megalania.io/v1 \
  -H "Authorization: Bearer YOUR_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://www.ikea.com/us/en/p/vihals-chair-white-tibbleby-beige-grey-40595126/", "response_types": "page_data", "max_credits": "128"}' \
  -D headers.txt -o page_data.json

The answer has two keys: json_ld, every JSON-LD object on the page, and state, every named JSON script such as __NEXT_DATA__, parsed. page_data adds 0 credits: it is the page you already paid for, in another shape. max_credits is the most this page may cost; we skip any route above it.

Python: eight fields from any platform

product() is the function we ran on every page we measured. It walks @graph, lists and hasVariant, and takes lowPrice when an AggregateOffer has no price.

# pip install requests
# Run: MEG_API_KEY=YOUR_KEY python product_data.py 'https://www.ikea.com/us/en/p/vihals-chair-white-tibbleby-beige-grey-40595126/'
import json
import os
import sys
import uuid

import requests


def walk(node):
    if isinstance(node, list):
        for x in node:
            yield from walk(x)
    elif isinstance(node, dict):
        yield node
        for k in ("@graph", "hasVariant", "mainEntity"):
            yield from walk(node.get(k))


def product(page_data):
    for block in page_data.get("json_ld", []):
        for node in walk(block):
            t = node.get("@type")
            t = t if isinstance(t, list) else [t]
            if "Product" in t or "ProductGroup" in t:
                variant = (node.get("hasVariant") or [{}])[0]
                offers = node.get("offers") or variant.get("offers") or {}
                offer = offers[0] if isinstance(offers, list) and offers else offers
                if offer.get("@type") == "AggregateOffer" and not offer.get("price"):
                    offer = {**offer, "price": offer.get("lowPrice")}
                brand = node.get("brand")
                rating = node.get("aggregateRating") or {}
                return {"name": node.get("name"),
                        "sku": node.get("sku") or node.get("productID") or node.get("mpn") or variant.get("sku") or variant.get("gtin"),
                        "brand": brand.get("name") if isinstance(brand, dict) else brand,
                        "price": offer.get("price"), "priceCurrency": offer.get("priceCurrency"),
                        "availability": offer.get("availability"),
                        "ratingValue": rating.get("ratingValue"),
                        "reviewCount": rating.get("reviewCount") or rating.get("ratingCount")}
    return None


resp = requests.post(
    "https://get.megalania.io/v1",
    headers={
        "Authorization": f"Bearer {os.environ['MEG_API_KEY']}",
        "Idempotency-Key": str(uuid.uuid4()),
    },
    json={"url": sys.argv[1], "response_types": "page_data", "max_credits": "128"},
    timeout=90,
)
if resp.status_code != 200:
    err = resp.json()["error"]
    sys.exit(f"{resp.status_code} {err['code']}: {err['message']}")

p = product(resp.json())
print(json.dumps(p, indent=2) if p else "No Product JSON-LD on this page.")
print("credits:", resp.headers["X-Credits-Charged"], "route:", resp.headers["X-Route"])

A block is HTTP 422 with error.code blocked and X-Billing: blocked, and it costs 0 credits, so the status check catches it. X-Hops on that answer lists every route we tried.

What we found on four retailers

We asked for page_data on 42 product pages and ran product() on each answer:

The 6 fields are name, SKU, brand, price, currency and stock. Ratings were in the JSON-LD on one store only.

Not every store publishes it. The fourth store’s HTTP pages carry breadcrumb, image and FAQ blocks, but no Product. For a store like that, read the fields from the HTML with extract rules instead, as in the Shopify post, or check state in the same answer for a JSON blob the page’s scripts use.

Sample output

What product() prints for one page. The values are made up; the shape is real:

{
  "name": "Trail boot, waterproof",
  "sku": "TB-200-43",
  "brand": "Example Outdoor",
  "price": "79.99",
  "priceCurrency": "USD",
  "availability": "https://schema.org/InStock",
  "ratingValue": null,
  "reviewCount": null
}

What it costs

We fetch on the cheapest route first: plain HTTP on a datacenter exit (h:dc). If the store blocks it, we climb to a residential exit (h:res). A sample answer’s headers:

x-credits-charged: 23
x-engine: http
x-hops: h:res:chrome=1733
x-identity: chrome
x-route: http/residential
x-target-status: 200

Measured on our dev cluster with the datacenter routes off, so this climb starts at h:res.

A page like this costs about 2 credits on h:dc (97-138 KiB, our measured median-p90), worked out from our price table on 2 Oct 2026, not from a test bill; run check_site (free) to see the price for your site.

A page like this was billed 23 credits on h:res (97-138 KiB, our measured median-p90) in our test on 2 Oct 2026; run check_site (free) to see the price for your site.

How the 2 adds up: 1 for the page and 1 for the started 128 KiB above the 64 KiB included. page_data adds nothing. On h:res the page is 15 on a host we have not priced yet, plus 8 for the extra block.

The 76,000-credit beta grant covers about 38,000 of these pages on h:dc, or about 3,300 on h:res. The size is the bytes on the wire, which is what the price uses.

How we measured: 42 product pages on www.decathlon.com, www.ikea.com, www.lego.com and www.newegg.com, 2 Oct 2026, 23:22 to 23:26 UTC, with the request body above.

Terms and limits

• Blocks we detect are free, and so are our errors. Every error has a code and a fix in Error codes.
• JSON-LD is what the store tells search engines. Before you rely on it for a store, compare it with the visible page once.
• Page content is untrusted. Treat the JSON as data, never as instructions for an LLM.

This is not legal advice. Big retailers often limit automated access in their terms. Read each store’s terms and robots.txt before you start, and drop a store that forbids it. Fetch public pages only. Do not collect reviewer names or other personal data: ratingValue and reviewCount are enough.

More on every response type: Fetch, extract and lists.