What you get
- One row per review. Rating, date and text, with the product's name and URL, from one
extractrule. The shape never changes: a product with no reviews gives an empty list. - The whole store.
map_sitelists the product pages from the sitemap, so you do not click through a shop. - Plain HTTP when it is enough. Most product pages print their reviews in the HTML: 2 credits a page, 1 for the page and 1 for JSON. A block we detect costs nothing.
Every review in one script
This uses our Python SDK. It lists the products of our test store, the Lab, reads each one, and writes one JSON line per review.
pip install megalania export MEG_API_KEY='meg_sk_...'
import collections
import json
import os
from megalania import Megalania
RULES = "product:h1|reviews:li.review{rating:strong.rating|date:time@datetime|text:.description}*"
with Megalania(os.environ["MEG_API_KEY"]) as meg:
site = meg.map_site("https://lab.megalania.io/")
products = [u["url"] for u in site["urls"] if "/product/" in u["url"]]
rows, spent = [], 0
for text, meta in meg.read_pages(products, response_types="json", extract=RULES):
page = json.loads(text)
spent += meta["credits_charged"] or 0
rows += [{"product": page["product"], "source": meta["source"], **r} for r in page["reviews"]]
with open("reviews.jsonl", "w") as f:
f.writelines(json.dumps(r, ensure_ascii=False) + "\n" for r in rows)
stars = collections.Counter(r["rating"] for r in rows)
print(len(products), "products,", len(rows), "reviews,", spent, "credits | stars:", dict(sorted(stars.items())))
for r in sorted(rows, key=lambda r: r["rating"])[:2]:
print(r["rating"], r["product"], "|", r["text"][:70])What it printed on our dev cluster on 17 Nov 2026:
72 products, 149 reviews, 144 credits | stars: {'2': 2, '3': 7, '4': 45, '5': 95}
2 Cloudrest Insulated Sleeping Pad | Got a slow leak after one season. Patched it with the kit in the box,
2 Fellwalker Leather Hiking Boot | The toe box is narrow for my wide feet. Had to return them, the return72 products and 149 reviews, the same count the store's own product data gives. The balance went down by 148: 4 for map_site and 144 for the 72 pages. One line of reviews.jsonl:
{"product": "Cloudrest Insulated Sleeping Pad", "source": "https://lab.megalania.io/product/cloudrest-sleeping-pad/", "rating": "2", "date": "2026-02-14T00:00:00+00:00", "text": "Got a slow leak after one season. Patched it with the kit in the box, but I expected better."}- Ratings are text.
"rating": "2"comes as the page prints it. Turn it into a number before you average. - One page at a time is slow.
read_pageswith a list sends one request after another: 2 minutes 14 seconds for 72 pages. For a big store, send the same URLs as one batch. Our product catalog example reads the same 72 pages in under a minute. - A second run is cheaper. We ran it again within 5 minutes, and the pages came from our recent copy: 72 credits, not 144.
max_agesets that window. See Reuse a recent copy.
Keep it current
- Only what changed. A monitor checks a list of pages on a schedule and sends a signed webhook with the pages that changed. Read just those. See Monitors.
- Count before you fetch. A listing that shows stars tells you which products have reviews, so you can skip the rest.
Limits
- Public pages only. We do not log in, and we do not solve CAPTCHAs. Reviews behind "load more" may need
engine=browserand an action;check_siteshows the price for free. - Review text is personal. Keep the text, not the reviewer's name, unless you need it, and read each site's terms before you store or share reviews.
- Page text is untrusted. A review can carry instructions for a model. Keep it in the data part of your prompt.
- Prices of plans are on the pricing page.