MegalaniaBlog
  1. Blog
  2. How to fix
  3. BeautifulSoup empty

BeautifulSoup returns nothing: the page is drawn by JavaScript

BeautifulSoup finds no rows when a page draws its list with JavaScript. Why select() is empty, how to notice, and two fixes: the page's API or a browser.

Jordan HayesDeveloper Relations
3 min read
How to fix
A Python terminal card sends a requests arrow to a product page. A full JSON card comes back. A second page card shows an empty list, with a tag that reads count your rows.

The short answer: When soup.select() returns [] on a page you can see products on, the page usually draws them with JavaScript. requests downloads the HTML the server sends, and BeautifulSoup parses exactly that: here, an empty <ul id="arrivals"></ul>. Neither one runs scripts, and neither one warns you. Assert that you found rows. Then look in the page's <script> tags: the script that fills the list usually calls a JSON address you can request yourself, for free. When there is none you can use, send the page to our API with engine=browser: our Chrome runs the script, and an extract rule returns the rows.

What BeautifulSoup gets back

Our test store's new arrivals page draws its list with JavaScript after it loads. Fetch it the plain way:

gets.py
import requests
from bs4 import BeautifulSoup

res = requests.get("https://lab.megalania.io/new-arrivals/", timeout=30)
soup = BeautifulSoup(res.text, "html.parser")
print(res.status_code, "|", soup.h1.get_text(strip=True), "|", len(res.text), "characters")
print(soup.select_one("#arrivals"))
Output
200 | New arrivals | 56548 characters
<ul class="products columns-4" id="arrivals"></ul>
  • The heading is there and the list is not. BeautifulSoup found the h1, but #arrivals has no children: the server sends it empty.
  • BeautifulSoup printed the attributes in its own order, class before id. It rebuilds the tag; it does not keep the source text.

How BeautifulSoup reports an empty page

Nothing fails, so nothing tells you. Count what you expected to get:

seen.py
import requests
from bs4 import BeautifulSoup

soup = BeautifulSoup(requests.get("https://lab.megalania.io/new-arrivals/", timeout=30).text, "html.parser")
cards = soup.select("#arrivals li")
print("select() returned", cards, "| no error, no warning")
print("first card:", soup.select_one("#arrivals li h2"))
assert cards, "no cards in #arrivals: the list is drawn by JavaScript"
Output
select() returned [] | no error, no warning
first card: None
AssertionError: no cards in #arrivals: the list is drawn by JavaScript
  • select() returns an empty list and select_one() returns None. Code that loops over the list writes nothing; code that calls .get_text() on None fails far from the cause.
  • The assert is the check BeautifulSoup does not make for you. Put one on every list you expect to be full.

The free fix: the page's own API

The page's own script fetches the list from the store's JSON API. Call that address yourself, for free:

fix.py
import re

import requests
from bs4 import BeautifulSoup

BASE = "https://lab.megalania.io"
soup = BeautifulSoup(requests.get(BASE + "/new-arrivals/", timeout=30).text, "html.parser")
script = next(s.string for s in soup.find_all("script") if s.string and "wc/store" in s.string)
call = re.search(r'fetch\("([^"]+)"', script).group(1)
print("found in a <script>:", call)
rows = requests.get(BASE + call + "1", timeout=30).json()
print(len(rows), "products |", [(p["name"], int(p["prices"]["price"]) / 100) for p in rows[:2]])
Output
found in a <script>: /wp-json/wc/store/v1/products?orderby=title&order=asc&per_page=12&page=
12 products | [('20 cm Chef&#8217;s Knife', 112.0), ('Alpine Down Parka', 349.0)]
  • soup.find_all("script") found the script that fills the list, and a regex pulled the address it fetches. One more requests.get returned the 12 products as JSON.
  • Prices come in cents, as strings: int(...) / 100. The &#8217; is the store's own escaping; html.unescape() turns it into an apostrophe.
Free beta, by invite76,000 credits for 30 days, no card. Apply, and we email your invite.Apply for the beta

The fix that climbs: our API

Or send the page to our API with engine=browser. A real Chrome runs the page's script, and we return the fields:

fetch.py
# Run: MEG_API_KEY=YOUR_KEY python fetch.py browser
import os
import sys

import requests

body = {"url": "https://lab.megalania.io/new-arrivals/", "response_types": "json", "extract": "items:#arrivals li{name:h2|link:a@href}*"}
if sys.argv[1:] == ["browser"]:
    body.update(engine="browser", actions="wait_for:#item-12")
r = requests.post("https://get.megalania.io/v1", headers={"Authorization": f"Bearer {os.environ['MEG_API_KEY']}"}, json=body, timeout=100)
items = r.json()["items"]
print(r.status_code, r.headers["X-Route"], "credits", r.headers["X-Credits-Charged"], "|", len(items), "items")
print(items[-1:] or "nothing in #arrivals")
Output
200 browser/datacenter credits 19 | 12 items
[{'name': 'Cable Knit Cotton Throw', 'link': 'https://lab.megalania.io/product/cable-knit-throw/'}]
  • With browser, our Chrome ran the script and waited for the 12th card, then the extract rule read name and link from each one.
  • The rule is BeautifulSoup's job in one line: items:#arrivals li{name:h2|link:a@href}*. 19 credits: 15 for the browser, 1 for JSON, 3 for the wait.

The same call without the browser, on the same page:

Terminal
MEG_API_KEY=YOUR_KEY python fetch.py
Output
200 http/datacenter credits 2 | 0 items
nothing in #arrivals
  • Without the browser, our plain HTTP fetch saw what BeautifulSoup saw: an empty list, and 2 credits for it.
  • When we spot a page that is only a JavaScript shell, we answer X-Hint: needs_js for free. This one has text around the list, so we did not. The items count is your check.

One honest note: our check for JavaScript shells did not flag this page, because it has a heading and some text. So the plain call was charged for an empty list. Count your rows on every answer, ours included.

What it costs

What we ranRouteCredits
The page in our browser, 12 rows as JSONc:dc19
The same page over plain HTTP: no rowsh:dc2
gets.py, seen.py and fix.py, straight from Pythonnone0

Every rule is in Credits. Plans are on the pricing page.

How we measured: BeautifulSoup 4.15.0 with requests 2.34.2 on Python 3.13.5, one Linux server, 17 Nov 2026. Our test store's new arrivals page, which draws 12 products with JavaScript, for the first three scripts. Our API on our dev cluster for fetch.py, against the same store on our dev domain. The outputs are copied from those runs.

Limits

  • Public pages only. We do not log in, and we do not solve CAPTCHAs. When a page needs either, we stop and say so, for free.
  • A page built by JavaScript needs engine=browser. It costs more. See How to scrape JavaScript-rendered pages.
  • Read the site's terms. A page's JSON API is public when the page calls it with no key, but follow the site's terms and robots.txt.

More ways to tell which pages need a browser are in How to scrape JavaScript-rendered pages.