MegalaniaBlog
  1. Blog
  2. How to fix
  3. httpx empty

httpx returns a page with no data: it is drawn by JavaScript

httpx gets a 200 and HTML with no products when the list is drawn by JavaScript. How to tell, the JSON API call with httpx, and a browser call.

Jordan HayesDeveloper Relations
2 min read
How to fix
A Python terminal card sends a requests arrow to a product page. A full JSON card comes back. A second page card shows an empty list, with a tag that reads count your rows.

The short answer: httpx returns the HTML a server sends and runs no JavaScript. On a page that draws its list in the browser, you get 200 OK, a heading, and an empty <ul id="arrivals"></ul>: raise_for_status() passes, and the products are not in the text at all. Count the rows you expected and fail loudly when there are none. Then look for wp-json or api in the page source: here the page's script calls the store's public JSON API, which httpx can call directly, over HTTP/2, for free. When a page has no such address, send it to our API with engine=browser.

What httpx gets back

Our test store's new arrivals page draws its list with JavaScript after it loads. Fetch it the plain way:

gets.py
import re

import httpx

res = httpx.get("https://lab.megalania.io/new-arrivals/", follow_redirects=True)
print(res.status_code, res.http_version, "|", len(res.text), "characters")
print(re.search(r"<ul[^>]*arrivals[^>]*>.*?</ul>", res.text, re.S).group())
Output
200 HTTP/1.1 | 56548 characters
<ul id="arrivals" class="products columns-4"></ul>
  • 56,548 characters and an empty list. The page is big because of its styles and scripts, not its products.
  • The regex prints the tag as the server sent it, attributes in their own order. No parser is needed to see that it is empty.

How httpx reports an empty page

Nothing fails, so nothing tells you. Count what you expected to get:

seen.py
import re

import httpx

res = httpx.get("https://lab.megalania.io/new-arrivals/", follow_redirects=True)
res.raise_for_status()
cards = re.findall(r'<li class="product"', res.text)
print("raise_for_status: passed | cards:", len(cards))
print("'Chef' in the HTML:", "Chef" in res.text, "| 'wp-json' in the HTML:", "wp-json" in res.text)
assert cards, "no cards in #arrivals: the list is drawn by JavaScript"
Output
raise_for_status: passed | cards: 0
'Chef' in the HTML: False | 'wp-json' in the HTML: True
AssertionError: no cards in #arrivals: the list is drawn by JavaScript
  • raise_for_status() passed: a 200 with no data is not an error to httpx.
  • A product name you can see in the browser, Chef, is not in the HTML. wp-json is: the page's script calls the store's JSON API.

The free fix: the page's own API

The page's own script fetches the list from the store's JSON API. Call that address yourself, for free:

fix.py
import httpx

params = {"orderby": "title", "order": "asc", "per_page": 12, "page": 1}
with httpx.Client(http2=True, base_url="https://lab.megalania.io") as client:
    res = client.get("/wp-json/wc/store/v1/products", params=params)
    products = res.json()
print(res.status_code, res.http_version, "| X-WP-Total:", res.headers.get("X-WP-Total"), "|", len(products), "products")
print([(p["name"], p["prices"]["price"]) for p in products[2:4]])
Output
200 HTTP/2 | X-WP-Total: 72 | 12 products
[('Amber Jar Candle Trio', '4200'), ('Bearing Baseplate Compass', '2695')]
  • params= builds the query string, and base_url keeps the paths short. With http2=True the call went over HTTP/2.
  • X-WP-Total: 72 is the full catalogue; per_page=12 is the first page of it. Prices are in cents, as strings.
Free beta, by invite76,000 credits for 30 days, no card. Apply, and we email your invite.Apply for the beta

The fix that climbs: our API

Or send the page to our API with engine=browser. A real Chrome runs the page's script, and we return the fields:

fetch.py
# Run: MEG_API_KEY=YOUR_KEY python fetch.py browser
import os
import sys

import httpx

body = {"url": "https://lab.megalania.io/new-arrivals/", "response_types": "json", "extract": "items:#arrivals li{name:h2|price:.price}*"}
if sys.argv[1:] == ["browser"]:
    body.update(engine="browser", actions="wait_for:#item-12")
res = httpx.post("https://get.megalania.io/v1", headers={"Authorization": f"Bearer {os.environ['MEG_API_KEY']}"}, json=body, timeout=100)
items = res.json().get("items", [])
print(res.status_code, res.headers.get("x-route"), "| credits", res.headers.get("x-credits-charged"), "|", len(items), "items")
print(items[2:4] or "nothing in #arrivals")
Output
200 browser/datacenter | credits 19 | 12 items
[{'name': 'Amber Jar Candle Trio', 'price': '$42.00'}, {'name': 'Bearing Baseplate Compass', 'price': '$26.95'}]
  • With browser, our Chrome ran the page's script and waited for the 12th card, then the extract rule read name and price from each.
  • The prices come as the page shows them, $26.95, not in cents. 19 credits: 15 for the browser, 1 for JSON, 3 for the wait.

The same call without the browser, on the same page:

Terminal
MEG_API_KEY=YOUR_KEY python fetch.py
Output
200 http/datacenter | credits 2 | 0 items
nothing in #arrivals
  • Without the browser, our plain HTTP fetch saw what httpx saw: an empty list, and 2 credits for it.
  • Count items on our answers the way seen.py counts cards.

One honest note: our check for JavaScript shells did not flag this page, because it has a heading and some text. So the plain call was charged for an empty list. Count your rows on every answer, ours included.

What it costs

What we ranRouteCredits
The page in our browser, 12 rows as JSONc:dc19
The same page over plain HTTP: no rowsh:dc2
gets.py, seen.py and fix.py, straight from httpxnone0

Every rule is in Credits. Plans are on the pricing page.

How we measured: httpx 0.28.1 with h2 4.4.1 on Python 3.13.5, one Linux server, 17 Nov 2026. Our test store's new arrivals page, which draws 12 products with JavaScript, for the first three scripts. Our API on our dev cluster for fetch.py, against the same store on our dev domain. The outputs are copied from those runs.

Limits

  • Public pages only. We do not log in, and we do not solve CAPTCHAs. When a page needs either, we stop and say so, for free.
  • A page built by JavaScript needs engine=browser. It costs more. See How to scrape JavaScript-rendered pages.
  • Read the site's terms. A page's JSON API is public when the page calls it with no key, but follow the site's terms and robots.txt.

More ways to tell which pages need a browser are in How to scrape JavaScript-rendered pages.