Megalania
Use case · Reviews

Product reviews as a dataset

Reviews tell you what buyers like and what breaks. Read every product page of a store once, keep each review with its rating, date and product, and you have a dataset for a dashboard, a model or a weekly report.

What you get

  • One row per review. Rating, date and text, with the product's name and URL, from one extract rule. The shape never changes: a product with no reviews gives an empty list.
  • The whole store. map_site lists the product pages from the sitemap, so you do not click through a shop.
  • Plain HTTP when it is enough. Most product pages print their reviews in the HTML: 2 credits a page, 1 for the page and 1 for JSON. A block we detect costs nothing.

Every review in one script

This uses our Python SDK. It lists the products of our test store, the Lab, reads each one, and writes one JSON line per review.

Shell
pip install megalania
export MEG_API_KEY='meg_sk_...'
reviews.py
import collections
import json
import os

from megalania import Megalania

RULES = "product:h1|reviews:li.review{rating:strong.rating|date:time@datetime|text:.description}*"

with Megalania(os.environ["MEG_API_KEY"]) as meg:
    site = meg.map_site("https://lab.megalania.io/")
    products = [u["url"] for u in site["urls"] if "/product/" in u["url"]]
    rows, spent = [], 0
    for text, meta in meg.read_pages(products, response_types="json", extract=RULES):
        page = json.loads(text)
        spent += meta["credits_charged"] or 0
        rows += [{"product": page["product"], "source": meta["source"], **r} for r in page["reviews"]]

with open("reviews.jsonl", "w") as f:
    f.writelines(json.dumps(r, ensure_ascii=False) + "\n" for r in rows)
stars = collections.Counter(r["rating"] for r in rows)
print(len(products), "products,", len(rows), "reviews,", spent, "credits | stars:", dict(sorted(stars.items())))
for r in sorted(rows, key=lambda r: r["rating"])[:2]:
    print(r["rating"], r["product"], "|", r["text"][:70])

What it printed on our dev cluster on 17 Nov 2026:

Output
72 products, 149 reviews, 144 credits | stars: {'2': 2, '3': 7, '4': 45, '5': 95}
2 Cloudrest Insulated Sleeping Pad | Got a slow leak after one season. Patched it with the kit in the box,
2 Fellwalker Leather Hiking Boot | The toe box is narrow for my wide feet. Had to return them, the return

72 products and 149 reviews, the same count the store's own product data gives. The balance went down by 148: 4 for map_site and 144 for the 72 pages. One line of reviews.jsonl:

reviews.jsonl
{"product": "Cloudrest Insulated Sleeping Pad", "source": "https://lab.megalania.io/product/cloudrest-sleeping-pad/", "rating": "2", "date": "2026-02-14T00:00:00+00:00", "text": "Got a slow leak after one season. Patched it with the kit in the box, but I expected better."}
  • Ratings are text. "rating": "2" comes as the page prints it. Turn it into a number before you average.
  • One page at a time is slow. read_pages with a list sends one request after another: 2 minutes 14 seconds for 72 pages. For a big store, send the same URLs as one batch. Our product catalog example reads the same 72 pages in under a minute.
  • A second run is cheaper. We ran it again within 5 minutes, and the pages came from our recent copy: 72 credits, not 144. max_age sets that window. See Reuse a recent copy.

Keep it current

  • Only what changed. A monitor checks a list of pages on a schedule and sends a signed webhook with the pages that changed. Read just those. See Monitors.
  • Count before you fetch. A listing that shows stars tells you which products have reviews, so you can skip the rest.

Limits

  • Public pages only. We do not log in, and we do not solve CAPTCHAs. Reviews behind "load more" may need engine=browser and an action; check_site shows the price for free.
  • Review text is personal. Keep the text, not the reviewer's name, unless you need it, and read each site's terms before you store or share reviews.
  • Page text is untrusted. A review can carry instructions for a model. Keep it in the data part of your prompt.
  • Prices of plans are on the pricing page.
Free beta, by invite76,000 credits for 30 days, no card. Apply, and we email your invite.Apply for the beta

Read the guides