MegalaniaBlog
  1. Blog
  2. How to fix
  3. Scrapy empty

Scrapy returns no items: the page is drawn by JavaScript

A Scrapy spider finishes with no items and no error when the list is drawn by JavaScript. How to notice, the JSON API fix, and a browser call.

Jordan HayesDeveloper Relations
3 min read
How to fix
A Python terminal card sends a requests arrow to a product page. A full JSON card comes back. A second page card shows an empty list, with a tag that reads count your rows.

The short answer: A spider that yields nothing from a page full of products in your browser is usually reading a list that JavaScript fills in later. Scrapy keeps the response exactly as the server sent it and runs no scripts, so here it gets an empty <ul id="arrivals"></ul>. The spider ends with finish_reason: finished and no items, and nothing in the log says why. Check for rows in parse and raise CloseSpider with a reason when there are none. Then request the JSON address the page's script calls with a JsonRequest: free, and faster than HTML. When a site has no such address, let api.py ask our API for engine=browser.

What Scrapy gets back

Our test store's new arrivals page draws its list with JavaScript after it loads. Fetch it the plain way:

gets.py
import scrapy


class Arrivals(scrapy.Spider):
    name = "arrivals"
    start_urls = ["https://lab.megalania.io/new-arrivals/"]
    custom_settings = {"LOG_LEVEL": "ERROR"}

    def parse(self, response):
        print(response.status, "|", response.css("h1::text").get(), "|", len(response.text), "characters")
        print(response.css("#arrivals").get())
Output
200 | New arrivals | 56548 characters
<ul id="arrivals" class="products columns-4"></ul>
  • A 200 with the heading and an empty list. The server sends #arrivals with no children; a script fills it in the browser.
  • Scrapy did nothing wrong: it returned the page exactly as the server sent it.

How Scrapy reports an empty page

Nothing fails, so nothing tells you. Count what you expected to get:

seen.py
import scrapy
from scrapy.exceptions import CloseSpider


class Arrivals(scrapy.Spider):
    name = "arrivals"
    start_urls = ["https://lab.megalania.io/new-arrivals/"]

    def parse(self, response):
        cards = response.css("#arrivals li")
        if not cards:
            raise CloseSpider("no cards in #arrivals: the list is drawn by JavaScript")
        for card in cards:
            yield {"name": card.css("h2::text").get()}
Output
Closing spider (no cards in #arrivals: the list is drawn by JavaScript)
'downloader/response_status_count/200': 1,
'finish_reason': 'no cards in #arrivals: the list is drawn by JavaScript',
  • Without the check, the same spider ended with finish_reason: finished and no item_scraped_count at all. A scheduled crawl reports success and writes nothing.
  • CloseSpider puts your reason in the log and in the stats, where a monitor can read it.

The free fix: the page's own API

The page's own script fetches the list from the store's JSON API. Call that address yourself, for free:

fix.py
import scrapy

API = "https://lab.megalania.io/wp-json/wc/store/v1/products?orderby=title&order=asc&per_page=12&page=1"


class Arrivals(scrapy.Spider):
    name = "arrivals"
    custom_settings = {"LOG_LEVEL": "ERROR"}

    async def start(self):
        yield scrapy.http.JsonRequest(API)

    def parse(self, response):
        products = response.json()
        print(response.status, "|", len(products), "products |", products[-1]["name"], int(products[-1]["prices"]["price"]) / 100)
        for p in products:
            yield {"name": p["name"], "price": int(p["prices"]["price"]) / 100}
Output
200 | 12 products | Cable Knit Cotton Throw 78.0
  • The address is in the page's own <script>: the store's public Store API, the same call the page makes. A JsonRequest and response.json() replace every CSS selector.
  • The Store API sends each price as a string of cents, so int(...) / 100 gives 78.0.
Free beta, by invite76,000 credits for 30 days, no card. Apply, and we email your invite.Apply for the beta

The fix that climbs: our API

Or send the page to our API with engine=browser. A real Chrome runs the page's script, and we return the fields:

api.py
# Run: MEG_API_KEY=YOUR_KEY scrapy runspider api.py --nolog -a engine=browser -a actions=wait_for:#item-12
import os

import scrapy


class ViaApi(scrapy.Spider):
    name = "via_api"
    custom_settings = {"DOWNLOAD_TIMEOUT": 100}

    async def start(self):
        body = {"url": "https://lab.megalania.io/new-arrivals/", "response_types": "json", "extract": "items:#arrivals li{name:h2|link:a@href}*"}
        body.update({k: v for k, v in vars(self).items() if k in ("engine", "actions")})
        yield scrapy.http.JsonRequest(
            "https://get.megalania.io/v1",
            data=body,
            headers={"Authorization": f"Bearer {os.environ['MEG_API_KEY']}"},
            meta={"handle_httpstatus_list": [422, 503]},
        )

    def parse(self, response):
        items = response.json().get("items", [])
        h = lambda k: (response.headers.get(k) or b"-").decode()
        print("status:", response.status, "route:", h("X-Route"), "credits:", h("X-Credits-Charged"), "items:", len(items))
        yield from items
Output
status: 200 route: browser/datacenter credits: 19 items: 12
{"name": "Cable Knit Cotton Throw", "link": "https://lab.megalania.io/product/cable-knit-throw/"}
  • With -a engine=browser -a actions=wait_for:#item-12, we loaded the page in Chrome, held on until card 12 existed, and the extract rule turned each card into a row.
  • yield from items hands our rows to your pipelines: -O items.jsonl wrote 12 lines. The browser route is 15 of the 19 credits; JSON is 1 and the wait 3.

The same call without the browser, on the same page:

Terminal
MEG_API_KEY=YOUR_KEY scrapy runspider api.py --nolog
Output
status: 200 route: http/datacenter credits: 2 items: 0
  • No engine argument, so we fetched over plain HTTP and met the same empty #arrivals as gets.py. We still charged 2 credits.
  • The items count is the same check as seen.py. Put it on our answers too.

One honest note: our check for JavaScript shells did not flag this page, because it has a heading and some text. So the plain call was charged for an empty list. Count your rows on every answer, ours included.

What it costs

What we ranRouteCredits
api.py with engine=browser: 12 itemsc:dc19
api.py without it: 0 itemsh:dc2
gets.py, seen.py and fix.py, straight from Scrapynone0

Every rule is in Credits. Plans are on the pricing page.

How we measured: Scrapy 2.18.0 with Twisted 26.10 on Python 3.13.5, one Linux server, 17 Nov 2026. The first three spiders crawl the new arrivals page of our test store, where a script fills in 12 products after load. api.py calls our API on our dev cluster, against the same store on our dev domain. The outputs are copied from those runs; log lines are cut to the message.

Limits

  • Public pages only. We do not log in, and we do not solve CAPTCHAs. When a page needs either, we stop and say so, for free.
  • A page built by JavaScript needs engine=browser. It costs more. See How to scrape JavaScript-rendered pages.
  • Read the site's terms. A page's JSON API is public when the page calls it with no key, but follow the site's terms and robots.txt.

More ways to tell which pages need a browser are in How to scrape JavaScript-rendered pages.