MegalaniaBlog
  1. Blog
  2. How to scrape
  3. Website to JSON API

Turn a website into a JSON API

A small Python server of your own that answers with a site's products as JSON. Rules from inspect, fields from extract, and max_age for a cheap cache.

Jordan HayesDeveloper Relations
9 min read
How to scrape
A store page card on the left, a small server box in the middle with the path /products, and a JSON card on the right. A clock tag on the server reads max_age 600.

The short answer: to convert a website to a JSON API, run a small server of your own. Each route on it sends one page URL and a few CSS rules to POST /v1, and answers with the fields as JSON. inspect suggests rules, extract reads them, and max_age lets a second call within 10 minutes reuse a recent copy for less. On our test store, lab.megalania.io, a product came back as JSON for 2 credits, and the same product again for 1 credit, in 0.04 s instead of 2.0 s.

What you build

Most sites have no public API. Their data is in the HTML: names, prices, SKUs, specs. You want it as JSON, from a URL your own code can call:

text
GET /products?page=2         → the 9 products on shop page 2, and the next page number
GET /products/<slug>         → one product: name, SKU, price and its specs

The server is about 60 lines of Python with no packages. It runs where you run it: your laptop, a container, a small VM. We do not host it for you. Behind each route is one call to us, so you never parse HTML and you never run a browser.

Find the rules with inspect

Rules are CSS selectors with a name: name:h1.product_title reads the text of the page’s h1. To find them without reading the page source, ask for response_types=inspect. It lists the blocks that repeat on the page, with a ready extract rule for each one. Save this as inspect_page.py:

inspect_page.py
import json
import os
import sys
import urllib.parse
import urllib.request

query = urllib.parse.urlencode({"url": sys.argv[1], "response_types": "inspect"})
req = urllib.request.Request(
    f"https://get.megalania.io/v1?{query}",
    headers={"Authorization": f"Bearer {os.environ['MEG_API_KEY']}"},
)
with urllib.request.urlopen(req, timeout=120) as res:
    page = json.load(res)
    print("credits:", res.headers["X-Credits-Charged"])
for c in page["containers"]:
    if "menu" not in c["selector"]:
        print(c["count"], c["extract"])
cURL
export MEG_API_KEY='meg_sk_...'
python inspect_page.py https://lab.megalania.io/product/alpine-down-parka/
python inspect_page.py https://lab.megalania.io/shop/
Output
credits: 1
6 items:tr.woocommerce-product-attributes-item*
credits: 1
18 items:ul.page-numbers > li{page_numbers:span.page-numbers|link:a.page-numbers@href|page_numbers_2:a.page-numbers}*

The script skips the site’s menus, which inspect lists too. What is left:

  • On the product page, inspect found the specs table: 6 rows. We turned its rule into one with two fields per row, {name:th|value:td}.
  • On the shop page, it did not find the product cards. It found the page numbers, but not the 9 li.product cards, the block we want most. We found this on 16 Nov 2026 and reported it. For WooCommerce, the card rule is in How to scrape a website to CSV; we use it below.
  • inspect costs 1 credit on the Lab, the price of the page. The structure itself is free.

So treat inspect as a first draft. Check each rule on a page you know before you build on it.

Convert a website to a JSON API: the server

Save this as shop_api.py. It uses only the Python standard library:

shop_api.py
import json
import os
import re
import urllib.error
import urllib.request
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from urllib.parse import parse_qs, urlparse

KEY = os.environ["MEG_API_KEY"]
LAB = "https://lab.megalania.io"
MAX_AGE = 600

LIST = (
    "items:li.product{name:h2|price:.price ins .amount, .price > .amount:first-child"
    "|url:a.woocommerce-loop-product__link@href}*|next:a.next@href"
)
PRODUCT = (
    "name:h1.product_title|sku:.sku|price:p.price ins .amount, p.price > .amount:first-child"
    "|attributes:tr.woocommerce-product-attributes-item{name:th|value:td}*"
)


def extract(url, fields):
    body = {"url": url, "response_types": "json", "extract": fields, "max_age": MAX_AGE, "max_credits": 4}
    req = urllib.request.Request(
        "https://get.megalania.io/v1",
        data=json.dumps(body).encode(),
        headers={"Authorization": f"Bearer {KEY}", "Content-Type": "application/json"},
    )
    try:
        with urllib.request.urlopen(req, timeout=120) as res:
            cost = {"credits": int(res.headers["X-Credits-Charged"]), "cache": res.headers["X-Cache"]}
            site = int(res.headers["X-Target-Status"])
            if site >= 400:
                return site, {"error": f"the site answered {site}", "cost": cost}
            return 200, json.load(res) | {"cost": cost}
    except urllib.error.HTTPError as e:
        return e.code, json.load(e)


class Shop(BaseHTTPRequestHandler):
    def do_GET(self):
        path = urlparse(self.path)
        if path.path == "/products":
            page = int(parse_qs(path.query).get("page", ["1"])[0])
            url = f"{LAB}/shop/" if page == 1 else f"{LAB}/shop/page/{page}/"
            status, data = extract(url, LIST)
            if status == 200:
                data["next"] = page + 1 if data["next"] else None
        elif m := re.fullmatch(r"/products/([a-z0-9-]+)", path.path):
            status, data = extract(f"{LAB}/product/{m[1]}/", PRODUCT)
        else:
            status, data = 404, {"error": "not found"}
        out = json.dumps(data, ensure_ascii=False).encode()
        self.send_response(status)
        self.send_header("Content-Type", "application/json; charset=utf-8")
        self.send_header("Content-Length", str(len(out)))
        self.end_headers()
        self.wfile.write(out)


ThreadingHTTPServer(("127.0.0.1", 8000), Shop).serve_forever()
  • The price rule has two selectors. A product on sale has an old price in <del> and the new one in <ins>; a product with sizes shows a range. ins .amount takes the sale price first, .price > .amount:first-child the plain price or the low end of a range. Our first try, .price .amount, read the old price of a product on sale. The answer said so in hints: “matches more than one element with different values”. Read the hints when you write rules.
  • Your API speaks for the site. We answer 200 when we fetched the page, and put the site’s own status in X-Target-Status. The server turns a 404 from the store into a 404 of its own, not a product full of null.
  • max_credits: 4 is the most one call may cost. A page that would cost more is refused, for free, and your route answers with our error.
  • The key stays on the server. Your users call your routes; they never see the key. The server listens on 127.0.0.1 only. Put it on the internet only behind your own login and a rate limit: every call to it spends your credits.
Free beta, by invite76,000 credits for 30 days, no card. Apply, and we email your invite.Apply for the beta

Call your API

Start the server in one terminal, and call it from another:

cURL
python shop_api.py
cURL
curl -s -w ' %{http_code} %{time_total}s\n' localhost:8000/products/alpine-down-parka
curl -s -w ' %{http_code} %{time_total}s\n' localhost:8000/products/alpine-down-parka
curl -s -w ' %{http_code}\n' localhost:8000/products/no-such-product
Output
{"attributes": [{"name": "Size", "value": "S, M, L, XL, XXL"}, {"name": "Color", "value": "Black"}, {"name": "Fill", "value": "700-fill down, RDS certified"}, {"name": "Shell", "value": "2-layer waterproof nylon"}, {"name": "Length", "value": "Hip"}, {"name": "Care", "value": "Down wash, tumble dry low with balls"}], "name": "Alpine Down Parka", "price": "$349.00", "sku": "GBR-OUT-052", "cost": {"credits": 2, "cache": "miss"}} 200 2.032725s
{"attributes": [{"name": "Size", "value": "S, M, L, XL, XXL"}, {"name": "Color", "value": "Black"}, {"name": "Fill", "value": "700-fill down, RDS certified"}, {"name": "Shell", "value": "2-layer waterproof nylon"}, {"name": "Length", "value": "Hip"}, {"name": "Care", "value": "Down wash, tumble dry low with balls"}], "name": "Alpine Down Parka", "price": "$349.00", "sku": "GBR-OUT-052", "cost": {"credits": 1, "cache": "hit"}} 200 0.044926s
{"error": "the site answered 404", "cost": {"credits": 2, "cache": "miss"}} 404

What the three calls show:

  • The first call fetched the page: 2 credits, 1 for the page and 1 for the JSON, in 2.0 s.
  • The second call was a cache hit: 1 credit, in 0.04 s. The copy is at most MAX_AGE seconds old.
  • A product that does not exist cost 2 credits: the store answered, with a 404 page. Your route says 404. If your users can type any slug, check it against your list first.

The list route works the same way. This client follows next through every page, twice:

client.py
import json
import time
import urllib.request

for run in (1, 2):
    page, rows, credits, hits, t = 1, 0, 0, 0, time.monotonic()
    while page:
        with urllib.request.urlopen(f"http://127.0.0.1:8000/products?page={page}") as res:
            data = json.load(res)
        rows += len(data["items"])
        credits += data["cost"]["credits"]
        hits += data["cost"]["cache"] == "hit"
        page = data["next"]
    print(f"run {run}: {rows} products, {credits} credits, {hits} cache hits, {time.monotonic() - t:.1f} s")
Output
run 1: 72 products, 16 credits, 0 cache hits, 13.6 s
run 2: 72 products, 8 credits, 8 cache hits, 0.4 s

How old may an answer be: max_age

max_age is how old, in seconds, a copy may be. It is your API’s freshness rule:

  • Left out, it is 300: a copy up to 5 minutes old is served.
  • 0 to 900. 0 always fetches the page fresh. More than 900 is a free 400 bad_parameter that names max_age.
  • A copy is the page as we fetched it, for the same URL and the same settings. Change a rule and it is a new page to us: the first call is a miss again.
  • A hit costs the cache price, 1 credit on the Lab, never more than the fetch. Your max_credits and your balance must still cover a live fetch: we hold the credits first and give back what a hit does not use.
  • A copy lives at most 15 minutes. For a longer cache, keep the JSON in your own store.

Pick the number from how fast the data changes. Prices that change once a day: 900 is fine. Stock during a sale: maybe 60. The full rules are in Reuse a recent copy.

What it costs

Call Pages Credits
inspect on one page 1 1
/products/<slug>, first call 1 2
The same product within MAX_AGE (cache hit) 1 1
/products through all 8 pages, empty cache 8 16
The same 8 pages again within MAX_AGE 8 8
A product the store does not have (its 404 page) 1 2
A bad parameter, a block we detect or one of our errors — 0

These prices are for the Lab on the datacenter route. A site that blocks it costs more. Our MCP tool check_site tells you a site’s prices for free. All the rules are in Credits. Plans are on the pricing page.

How we measured: our dev cluster on 16 Nov 2026, the Lab store on our dev domain, Python 3.13 with no packages. We ran inspect_page.py on two pages, then shop_api.py with the three curl calls and client.py, on an empty cache. The times are from our test machine to the dev cluster; yours depend on your network. For fields from a page without writing any rules, see Extract structured data with an LLM; for stores that publish JSON-LD, How to scrape product data from any store needs no rules at all.

Limits

  • We do not host your API. There is no always-on endpoint of ours that serves a site as JSON. The server is yours to run and to keep up.
  • Rules break when the site changes its layout. You get null fields, not an error. Check a few products after a redesign, and read hints.
  • inspect is a draft. On 16 Nov 2026 it missed the product cards on the Lab’s shop page.
  • A page that draws its data with JavaScript comes back without it over plain HTTP. Add "engine": "browser" to the body for those pages; it costs more. See How to scrape JavaScript-rendered pages.
  • We read public pages only. No logins and no CAPTCHA solving. When a page needs either, we stop and say so, for free.

Every parameter is in Fetch, extract and lists. Fetch public pages only. Read the site’s terms and robots.txt, and follow them. Serving a site’s data to others can need the site’s permission.

Jordan HayesDeveloper RelationsPublished 16 Nov 2026 · measured on our dev cluster, 16 Nov 2026, 2 inspect calls, 8 list pages, 72 products and 1 product page on our test store, Python 3.13, standard library only

Keep reading