MegalaniaBlog
  1. Blog
  2. How to scrape
  3. Do I need an API?

Do I need a web scraping API? Often not. Here is what we measured

Python requests and a browser you run yourself read most pages for free. Cases measured on our test store show when an API earns its price.

Jordan HayesDeveloper Relations
6 min read
How to scrape
A fork in a path. The left sign reads requests, free, with a page card and a tick. The right sign reads API, with a browser, a globe and a shield. A small scale sits between them.

The short answer: often, no. If the data is in the page’s HTML and the site does not block you, Python requests and a parser get it for free. If the page draws its data with JavaScript, a browser you run yourself, like Playwright, gets that too. You need more than that when a site blocks scripts, when you need an exit in another country, or when you do not want to run browsers and proxies yourself. On our test store, lab.megalania.io, free tools got the data in all 4 cases we could measure there. The fifth, a site that blocks scripts, is not on our test store.

Five cases on our test store

Case Free tools Through our API
A product list in the HTML (/shop/) requests: 9 products, 0.7 s 2 credits, as JSON fields
The store’s own JSON (/wp-json/wc/store/v1/products) requests: 72 products, 2.0 s 1 credit
A list drawn by JavaScript (/new-arrivals/) requests: 200, 0 products. Playwright: 12 products, 1.8 s 19 credits, in our browser
Page text for an LLM (/shipping/) requests: 55,409 bytes of HTML to clean up 1 credit, 633 characters of Markdown
A site that blocks scripts Not on our test store: it has no bot rule 0 credits when we detect a block

The free numbers are from this script. Run it on your own pages before you pay anyone, us included:

free_first.py
import re
import time

import requests
from playwright.sync_api import sync_playwright

LAB = "https://lab.megalania.io"
rows = lambda html: re.findall(r'<h2 class="woocommerce-loop-product__title">(.*?)</h2>', html)

for path in ["/shop/", "/new-arrivals/", "/shipping/"]:
    t = time.time()
    r = requests.get(LAB + path, timeout=30)
    print("requests", path, r.status_code, len(r.text), "bytes", len(rows(r.text)), "products", f"{time.time() - t:.1f}s")

t = time.time()
r = requests.get(LAB + "/wp-json/wc/store/v1/products?per_page=100", timeout=30)
print("requests store api", r.status_code, len(r.json()), "products", f"{time.time() - t:.1f}s")

with sync_playwright() as p:
    t = time.time()
    browser = p.chromium.launch()
    page = browser.new_page()
    page.goto(LAB + "/new-arrivals/")
    page.wait_for_selector("#item-12")
    print("playwright /new-arrivals/", page.locator("#arrivals li").count(), "products", f"{time.time() - t:.1f}s")
    browser.close()
Output
requests /shop/ 200 75688 bytes 9 products 0.7s
requests /new-arrivals/ 200 56548 bytes 0 products 0.6s
requests /shipping/ 200 55409 bytes 0 products 0.6s
requests store api 200 72 products 2.0s
playwright /new-arrivals/ 12 products 1.8s
  • /new-arrivals/ is the trap. requests gets a 200 and an empty list, and nothing warns you. Count your rows.
  • Playwright fixed it for free, on our own server. It needs a Chrome on that server: its headless shell was 282 MB on disk here.
  • The store’s JSON was the best free answer. A shop’s own API, when it has one, beats any scraping. See WooCommerce products as JSON.

You do not need an API when

  • The data is in the HTML. Open the page’s source, not the inspector. If you see your data there, requests gets it.
  • The site has a public JSON address. A shop’s product API, a sitemap.xml, an RSS feed, an llms.txt.
  • Nobody blocks you, at your volume. A few hundred pages a day from one server is fine for many sites, at a polite rate.
  • You can run a browser. Playwright or Puppeteer on your own server reads JavaScript pages, if you keep Chrome updated and give it memory.

You need more when

  • A site blocks scripts. It can check your User-Agent, your TLS hello and your IP. From a data center, requests looks like a script on all three. We measured what each client looks like in Why your scraper gets 403 Forbidden and Scrape a Cloudflare website in Python.
  • You need another country. Prices and stock change by country, and you need an exit IP there: Scrape a website from a different country.
  • You cannot run a browser. An n8n Cloud workflow, a serverless function, or an AI agent in a chat app has no Chrome to start.
  • You want text or fields, not HTML. 55 KB of HTML became 633 characters of Markdown. That matters when an LLM pays per token.

Then you have three options: run browsers and buy proxies yourself, use a browser service, or use a scraping API. Running it yourself is cheapest per page, and it is your time when a site changes its defenses.

Free beta, by invite76,000 credits for 30 days, no card. Apply, and we email your invite.Apply for the beta

What our API costs, and what it does not do

Page Credits
A plain page over HTTP, Markdown or HTML 1
The same page as JSON fields 2
A JavaScript page in our browser, with one action 19
A block we detect, or one of our errors 0
check_site: a site’s prices and our recent results there 0

These prices are for our test store on the datacenter route. A site that blocks it costs more: run check_site over MCP or GET /v1/tier?host= first, for free. All the rules are in Credits. Plans are on the pricing page.

What we do not do: we do not log in, and we do not solve CAPTCHAs. When a page needs either, we stop and say so, for free. Neither does requests.

A test for your own site

  1. Open the page source. If your data is there, try requests first.
  2. Look for JSON. Search the source for wp-json, .json or api. A page’s own script often calls an address you can call too.
  3. Run free_first.py on 20 of your pages, at a polite rate. Count the rows, not the status codes.
  4. Only for the pages that fail, try an API or a browser of your own. Send just those pages. Keep the free path for the rest.

How we measured: one server on 17 Nov 2026, Python 3.13, requests 2.34.2 and Playwright 1.65.2 with its headless Chrome 159, against our test store at lab.megalania.io. The output above is copied from that run; times are one run each, from that server. The API column is from our live API the same day, through our MCP tools, which run the same pipeline as /v1.

Limits

  • One store, one day. Our test store has no bot rule, so it shows what free tools do on an open site. A site with a bot rule is another story; the 403 post shows what it checks.
  • Times are one run each. They show the order of size, not a benchmark.
  • Fetch public pages only. Read the site’s terms and robots.txt, and follow them.
Jordan HayesDeveloper RelationsPublished 16 Nov 2026 · re-run on 17 Nov 2026 from one server, Python 3.13, requests 2.34.2, Playwright 1.65.2 with its headless Chrome, and our live API, against our test store

Keep reading