The short answer: When soup.select() returns [] on a page you can see products on, the page usually draws them with JavaScript. requests downloads the HTML the server sends, and BeautifulSoup parses exactly that: here, an empty <ul id="arrivals"></ul>. Neither one runs scripts, and neither one warns you. Assert that you found rows. Then look in the page's <script> tags: the script that fills the list usually calls a JSON address you can request yourself, for free. When there is none you can use, send the page to our API with engine=browser: our Chrome runs the script, and an extract rule returns the rows.
What BeautifulSoup gets back
Our test store's new arrivals page draws its list with JavaScript after it loads. Fetch it the plain way:
import requests
from bs4 import BeautifulSoup
res = requests.get("https://lab.megalania.io/new-arrivals/", timeout=30)
soup = BeautifulSoup(res.text, "html.parser")
print(res.status_code, "|", soup.h1.get_text(strip=True), "|", len(res.text), "characters")
print(soup.select_one("#arrivals"))200 | New arrivals | 56548 characters <ul class="products columns-4" id="arrivals"></ul>
- The heading is there and the list is not. BeautifulSoup found the
h1, but#arrivalshas no children: the server sends it empty. - BeautifulSoup printed the attributes in its own order,
classbeforeid. It rebuilds the tag; it does not keep the source text.
How BeautifulSoup reports an empty page
Nothing fails, so nothing tells you. Count what you expected to get:
import requests
from bs4 import BeautifulSoup
soup = BeautifulSoup(requests.get("https://lab.megalania.io/new-arrivals/", timeout=30).text, "html.parser")
cards = soup.select("#arrivals li")
print("select() returned", cards, "| no error, no warning")
print("first card:", soup.select_one("#arrivals li h2"))
assert cards, "no cards in #arrivals: the list is drawn by JavaScript"select() returned [] | no error, no warning first card: None AssertionError: no cards in #arrivals: the list is drawn by JavaScript
select()returns an empty list andselect_one()returnsNone. Code that loops over the list writes nothing; code that calls.get_text()onNonefails far from the cause.- The
assertis the check BeautifulSoup does not make for you. Put one on every list you expect to be full.
The free fix: the page's own API
The page's own script fetches the list from the store's JSON API. Call that address yourself, for free:
import re
import requests
from bs4 import BeautifulSoup
BASE = "https://lab.megalania.io"
soup = BeautifulSoup(requests.get(BASE + "/new-arrivals/", timeout=30).text, "html.parser")
script = next(s.string for s in soup.find_all("script") if s.string and "wc/store" in s.string)
call = re.search(r'fetch\("([^"]+)"', script).group(1)
print("found in a <script>:", call)
rows = requests.get(BASE + call + "1", timeout=30).json()
print(len(rows), "products |", [(p["name"], int(p["prices"]["price"]) / 100) for p in rows[:2]])found in a <script>: /wp-json/wc/store/v1/products?orderby=title&order=asc&per_page=12&page=
12 products | [('20 cm Chef’s Knife', 112.0), ('Alpine Down Parka', 349.0)]soup.find_all("script")found the script that fills the list, and a regex pulled the address it fetches. One morerequests.getreturned the 12 products as JSON.- Prices come in cents, as strings:
int(...) / 100. The’is the store's own escaping;html.unescape()turns it into an apostrophe.
The fix that climbs: our API
Or send the page to our API with engine=browser. A real Chrome runs the page's script, and we return the fields:
# Run: MEG_API_KEY=YOUR_KEY python fetch.py browser
import os
import sys
import requests
body = {"url": "https://lab.megalania.io/new-arrivals/", "response_types": "json", "extract": "items:#arrivals li{name:h2|link:a@href}*"}
if sys.argv[1:] == ["browser"]:
body.update(engine="browser", actions="wait_for:#item-12")
r = requests.post("https://get.megalania.io/v1", headers={"Authorization": f"Bearer {os.environ['MEG_API_KEY']}"}, json=body, timeout=100)
items = r.json()["items"]
print(r.status_code, r.headers["X-Route"], "credits", r.headers["X-Credits-Charged"], "|", len(items), "items")
print(items[-1:] or "nothing in #arrivals")200 browser/datacenter credits 19 | 12 items
[{'name': 'Cable Knit Cotton Throw', 'link': 'https://lab.megalania.io/product/cable-knit-throw/'}]- With
browser, our Chrome ran the script and waited for the 12th card, then theextractrule read name and link from each one. - The rule is BeautifulSoup's job in one line:
items:#arrivals li{name:h2|link:a@href}*. 19 credits: 15 for the browser, 1 for JSON, 3 for the wait.
The same call without the browser, on the same page:
MEG_API_KEY=YOUR_KEY python fetch.py
200 http/datacenter credits 2 | 0 items nothing in #arrivals
- Without the browser, our plain HTTP fetch saw what BeautifulSoup saw: an empty list, and 2 credits for it.
- When we spot a page that is only a JavaScript shell, we answer
X-Hint: needs_jsfor free. This one has text around the list, so we did not. Theitemscount is your check.
One honest note: our check for JavaScript shells did not flag this page, because it has a heading and some text. So the plain call was charged for an empty list. Count your rows on every answer, ours included.
What it costs
| What we ran | Route | Credits |
|---|---|---|
| The page in our browser, 12 rows as JSON | c:dc | 19 |
| The same page over plain HTTP: no rows | h:dc | 2 |
gets.py, seen.py and fix.py, straight from Python | none | 0 |
Every rule is in Credits. Plans are on the pricing page.
How we measured: BeautifulSoup 4.15.0 with requests 2.34.2 on Python 3.13.5, one Linux server, 17 Nov 2026. Our test store's new arrivals page, which draws 12 products with JavaScript, for the first three scripts. Our API on our dev cluster for fetch.py, against the same store on our dev domain. The outputs are copied from those runs.
Limits
- Public pages only. We do not log in, and we do not solve CAPTCHAs. When a page needs either, we stop and say so, for free.
- A page built by JavaScript needs
engine=browser. It costs more. See How to scrape JavaScript-rendered pages. - Read the site's terms. A page's JSON API is public when the page calls it with no key, but follow the site's terms and robots.txt.
More ways to tell which pages need a browser are in How to scrape JavaScript-rendered pages.