MegalaniaBlog
  1. Blog
  2. How to scrape
  3. Scrapy

Scrapy JavaScript rendering without a browser on your side

Keep your Scrapy spider and its selectors. Send the pages that need JavaScript through one API call and get rendered HTML back. Measured on our test store.

Jordan HayesDeveloper Relations
6 min read
How to scrape
A Scrapy spider card sends a request arrow to a page card with an empty list. A browser card in the middle fills it. A HTML card with 12 rows comes back to the spider.

The short answer: Scrapy does not run JavaScript, so a page that draws its list in the browser comes back as an empty list. For Scrapy JavaScript rendering you do not need a browser of your own or a new plugin. Send that page’s URL to GET /v1 with engine=browser, and the response your callback gets is the rendered HTML. Your response.css(...) selectors stay the same. On our test store, lab.megalania.io, a JavaScript page with 12 products cost 18 credits in the browser, and all 8 shop pages, 72 products, cost 8 credits over plain HTTP.

What Scrapy sees on a JavaScript page

A plain spider, two pages from the Lab:

plain.py
import scrapy


class Plain(scrapy.Spider):
    name = "plain"
    start_urls = ["https://lab.megalania.io/shop/", "https://lab.megalania.io/new-arrivals/"]
    custom_settings = {"LOG_LEVEL": "WARNING"}

    def parse(self, response):
        names = response.css("li.product h2::text, #arrivals li h2::text").getall()
        print(response.url.split("/")[-2], response.status, len(names), names[:1])
cURL
pip install scrapy
scrapy runspider plain.py
Output
new-arrivals 200 0 []
shop 200 9 ['20 cm Chef’s Knife']

Both pages answer 200. /shop/ has its 9 products in the HTML. /new-arrivals/ has an empty #arrivals list: the page fills it with JavaScript after it loads, and Scrapy only downloads the HTML. Nothing in the response says it failed.

If all your pages look like /shop/, plain Scrapy is free and fast. Keep it. The rest of this post is for the pages that look like /new-arrivals/.

Scrapy JavaScript rendering with one request

A small helper builds a scrapy.Request to our API. The page goes in the url parameter, and the key goes in a header:

arrivals.py
import os
from urllib.parse import urlencode

import scrapy

KEY = os.environ["MEG_API_KEY"]


def meg(url, callback, **params):
    query = urlencode({"url": url, "response_types": "html", **params})
    return scrapy.Request(f"https://get.megalania.io/v1?{query}", callback, headers={"Authorization": f"Bearer {KEY}"}, cb_kwargs={"page": url})


class Arrivals(scrapy.Spider):
    name = "arrivals"
    custom_settings = {
        "ROBOTSTXT_OBEY": False,
        "DOWNLOAD_TIMEOUT": 130,
        "HTTPERROR_ALLOW_ALL": True,
        "LOG_LEVEL": "WARNING",
    }

    async def start(self):
        url = "https://lab.megalania.io/new-arrivals/"
        yield meg(url, self.parse)
        yield meg(url, self.parse, engine="browser", actions="wait_for:#item-12")

    def parse(self, response, page):
        if response.status != 200:
            print("error", response.status, response.text[:200])
            return
        prices = response.css("#arrivals li .price::text").getall()
        print(response.headers["X-Route"].decode(), response.headers["X-Credits-Charged"].decode(), len(prices), prices[:3])
cURL
export MEG_API_KEY='meg_sk_...'
scrapy runspider arrivals.py
Output
http/datacenter 1 0 []
browser/datacenter 18 12 ['$112.00', '$349.00', '$42.00']
  • The first request is plain HTTP on our side. Same empty list as Scrapy alone, and it cost 1 credit. Our check for JavaScript shells did not flag this page: it has a heading and a paragraph, so it does not look empty. When we do flag a shell, the answer has X-Hint: needs_js and costs nothing. Here it did not, so count your rows.
  • The second request ran a real Chrome on our side. 12 products, 18 credits: 15 for the browser route and 3 for the page action.
  • wait_for:#item-12 waits until the 12th card exists. The list loads after the page’s load event, so without it the browser can answer before the cards are there.
  • response_types=html makes the body plain HTML, so response.css works on it as on any page.

Three settings matter:

  • ROBOTSTXT_OBEY: False. Scrapy would ask our API host for /robots.txt and get a 404. The site’s own robots.txt still applies to you: read it and follow it.
  • DOWNLOAD_TIMEOUT: 130. A page in a browser can take tens of seconds. Our own limit is 60 seconds by default and 90 at most.
  • HTTPERROR_ALLOW_ALL: True. An error from us is JSON with a 4xx status, and it is free. With this setting your callback sees it and can print it. Without it, Scrapy drops it.
Free beta, by invite76,000 credits for 30 days, no card. Apply, and we email your invite.Apply for the beta

Most spiders follow links. Here the spider reads all 8 shop pages, one after the other, and yields one item per product:

shop.py
import os
from urllib.parse import urlencode, urljoin

import scrapy

KEY = os.environ["MEG_API_KEY"]


def meg(url, callback, **params):
    query = urlencode({"url": url, "response_types": "html", **params})
    return scrapy.Request(f"https://get.megalania.io/v1?{query}", callback, headers={"Authorization": f"Bearer {KEY}"}, cb_kwargs={"page": url})


class Shop(scrapy.Spider):
    name = "shop"
    custom_settings = {
        "ROBOTSTXT_OBEY": False,
        "DOWNLOAD_TIMEOUT": 130,
        "CONCURRENT_REQUESTS": 5,
        "LOG_LEVEL": "WARNING",
    }
    credits = 0

    async def start(self):
        yield meg("https://lab.megalania.io/shop/", self.parse)

    def parse(self, response, page):
        self.credits += int(response.headers["X-Credits-Charged"])
        for card in response.css("li.product"):
            price = card.css(".price ins .amount, .price > .amount")
            yield {
                "name": card.css("h2::text").get(),
                "price": "".join(price[0].css("::text").getall()) if price else None,
                "url": card.css("a.woocommerce-loop-product__link::attr(href)").get(),
            }
        next_page = response.css("a.next::attr(href)").get()
        if next_page:
            yield meg(urljoin(page, next_page), self.parse)

    def closed(self, reason):
        print("credits", self.credits)
cURL
scrapy runspider shop.py -o shop.json
Output
credits 8

shop.json has 72 products. The first one:

JSON
{"name": "20 cm Chef’s Knife", "price": "$112.00", "url": "https://lab.megalania.io/product/chef-knife/"}
  • urljoin(page, next_page), not response.urljoin. response.url is our API URL, not the shop page. The helper passes the page’s own URL in cb_kwargs, so a relative link joins to the right site.
  • CONCURRENT_REQUESTS: 5. A beta account runs 5 requests at once. One more answers 429 over_concurrency with a Retry-After header, for free. Scrapy’s retry middleware retries a 429 but does not wait for Retry-After, so stay at 5 and it does not happen.
  • The price rule reads the sale price. A product on sale has the old price in <del> and the new one in <ins>. .price ins .amount, .price > .amount picks the one a shopper pays.
  • This crawl needs no browser. /shop/ is in the HTML, so it goes over plain HTTP, 1 credit a page. Add engine="browser" only to the requests for pages that need it.

How to tell, before you pay, which of your pages need the browser is in How to scrape JavaScript-rendered pages. The same crawl without Scrapy, as JSON fields, is in Web scraping with Python requests.

What it costs

Step Pages Credits
Scrapy alone, both pages 2 0
/new-arrivals/ through the API over plain HTTP (empty) 1 1
/new-arrivals/ in the browser, with one action 1 18
The 8 shop pages as HTML 8 8
A bad parameter, a block we detect or one of our errors — 0

These prices are for the Lab on the datacenter route. A site that blocks it costs more. Our MCP tool check_site tells you a site’s prices for free. All the rules are in Credits. Plans are on the pricing page.

How we measured: our dev cluster on 16 Nov 2026, the Lab store on our dev domain, Python 3.13 and Scrapy 2.18.0. We ran plain.py, arrivals.py and shop.py, then ran them again within 5 minutes. HTML from the cache costs the same 1 credit a page, so shop.py cost 8 both times. The browser call is fetched fresh every time: a request with page actions does not use the cache.

Limits

  • An empty list can still be a 200, from Scrapy and from us. Check the item count on pages you know.
  • Selectors break when the site changes its layout. You get None fields, not an error. Check a few items after a redesign.
  • We read public pages only. No logins and no CAPTCHA solving. When a page needs either, we stop and say so, for free.
  • Scrapy’s own features still run on your side: the duplicate filter, item pipelines and feed exports work as before. Only the download goes through us.

Every parameter is in Fetch, extract and lists. Fetch public pages only. Read the site’s terms and robots.txt, and follow them.