MegalaniaBlog
  1. Blog
  2. How to fix
  3. Scrapy 403

Scrapy 403 Forbidden: why your spider gets it, and the fix

Scrapy says Scrapy in every request and drops 403s before parse. What it sends, where the 403 goes, the settings that fix it. Tested on Scrapy 2.18.

Jordan HayesDeveloper Relations
3 min read
How to fix
A padlock cut open. Inside, four pins named UA, headers, cookies and TLS. Three sit on the line, one sits low.

The short answer: Scrapy gets 403 Forbidden when a site's bot filter sees its default request: User-Agent: Scrapy/2.18.0 (+https://scrapy.org) and Accept-Language: en. Then the 403 never reaches your parse(): HttpErrorMiddleware logs Ignoring response and moves on, so a spider can finish with no items and no error. Set USER_AGENT and DEFAULT_REQUEST_HEADERS to browser values, and add handle_httpstatus_list where you want to see the 403 yourself. Scrapy's connection stays HTTP/1.1 with Python's TLS. For sites that check those, yield one JsonRequest to our API per page instead. A block we detect costs nothing.

What Scrapy sends

Start a listener in one terminal with nc -l 8799. It prints every byte a client sends and never answers, so the client times out. Run this in a second terminal:

Terminal 2
scrapy fetch --nolog -s DOWNLOAD_TIMEOUT=3 -s RETRY_ENABLED=False http://127.0.0.1:8799/
Output
GET / HTTP/1.1
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en
User-Agent: Scrapy/2.18.0 (+https://scrapy.org)
Accept-Encoding: gzip, deflate, br, zstd
Host: 127.0.0.1:8799
  • The User-Agent is Scrapy/2.18.0 with a link to the project. It is polite and honest, and it is the first thing a bot filter blocks.
  • Accept-Language: en, a bare language with no region and no weights. Browsers send something like en-US,en;q=0.9.
  • Host comes last. Scrapy's Accept looks like a browser's, but the order of the lines does not.

How Scrapy reports a 403

Our test store answers 403 on its uploads folder, because folder listing is off. That is a real "no", not a bot filter, and a handy way to see what your code gets back:

seen.py
import scrapy


class Uploads(scrapy.Spider):
    name = "uploads"
    start_urls = ["https://lab.megalania.io/wp-content/uploads/"]

    def parse(self, response):
        print("parse got", response.status)
Output
Crawled (403) <GET https://lab.megalania.io/wp-content/uploads/> (referer: None)
Ignoring response <403 https://lab.megalania.io/wp-content/uploads/>: HTTP status code is not handled or not allowed
'httperror/response_ignored_count': 1,
'httperror/response_ignored_status_count/403': 1,
  • parse() never ran. parse got is not in the output: the 403 stopped at HttpErrorMiddleware, which logged it at INFO and counted it in the stats.
  • Search your stats for httperror/response_ignored_status_count/403. It is the only place a quiet run shows that a site said no.

The free fix: browser headers

Send the headers a browser sends. This fixes sites that read only the headers:

fix.py
import scrapy


class About(scrapy.Spider):
    name = "about"
    start_urls = ["https://lab.megalania.io/about/"]
    custom_settings = {
        "USER_AGENT": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/154.0 Safari/537.36",
        "DEFAULT_REQUEST_HEADERS": {
            "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
            "Accept-Language": "en-US,en;q=0.9",
        },
        "LOG_LEVEL": "ERROR",
    }

    def parse(self, response):
        print(response.status, response.css("h1::text").get(), "|", response.request.headers.get("User-Agent").decode()[:40])
Output
200 About | Mozilla/5.0 (Windows NT 10.0; Win64; x64
  • USER_AGENT replaces the Scrapy name on every request. DEFAULT_REQUEST_HEADERS replaces the default Accept and Accept-Language pair.
  • response.request.headers shows what was really sent, so you can check the fix from inside the spider.
  • A scrapy startproject project also obeys robots.txt by default (ROBOTSTXT_OBEY = True). Keep it on.
Free beta, by invite76,000 credits for 30 days, no card. Apply, and we email your invite.Apply for the beta

The fix that climbs: our API

Send the page to our API instead. We fetch it like a real browser, move to a residential IP or a real Chrome when the site blocks us, and a block we detect costs nothing:

api.py
# Run: MEG_API_KEY=YOUR_KEY scrapy runspider api.py -a page=https://lab.megalania.io/returns/
import os

import scrapy


class ViaApi(scrapy.Spider):
    name = "via_api"
    custom_settings = {"DOWNLOAD_TIMEOUT": 100, "LOG_LEVEL": "ERROR"}

    async def start(self):
        body = {"url": self.page, "response_types": "markdown", "max_credits": "64"}
        body.update({k: v for k, v in vars(self).items() if k in ("engine", "actions", "timeout")})
        yield scrapy.http.JsonRequest(
            "https://get.megalania.io/v1",
            data=body,
            headers={"Authorization": f"Bearer {os.environ['MEG_API_KEY']}"},
            meta={"handle_httpstatus_list": [422, 503]},
        )

    def parse(self, response):
        h = lambda k: (response.headers.get(k) or b"-").decode()
        print("status:", response.status, "route:", h("X-Route"), "credits:", h("X-Credits-Charged"))
        print(response.text[:150])
Output
status: 200 route: http/datacenter credits: 1
![Wooden shelves with jars in a small shop](https://lab.megalania.io/wp-content/uploads/gibber/about-shop.webp)

# About

Gibber & Co. is a demo store
  • One JsonRequest per page. handle_httpstatus_list lets our 422 and 503 reach parse, so you can read error.code instead of a log line.
  • The about page came back as Markdown, its first image as a Markdown image.

The same call on the uploads folder that answers 403 to everyone:

Terminal
MEG_API_KEY=YOUR_KEY scrapy runspider api.py -a page=https://lab.megalania.io/wp-content/uploads/
Output
status: 422 route: browser/residential credits: -
{"error":{"code":"blocked","message":"The site blocked us and there is nothing to show. You are not charged. X-Untried lists the routes left to try; n
  • This time the answer reached parse, because 422 is in handle_httpstatus_list. Four routes, four 403s, one blocked.
  • credits: - means no charge header: count it as 0. Our error JSON says what to try next.

One honest note: our test store blocks nobody, so these runs show the calls, not a bot filter being beaten. On a site that blocks, the call is the same; the price depends on the route that worked, and our MCP tool check_site tells you that price for free.

What it costs

What we ranRouteCredits
The about page as Markdownh:dc1
The uploads folder: 403 on every routeh:dc h:res c:dc c:res0
seen.py and fix.py, straight from Scrapynone0

Every rule is in Credits. Plans are on the pricing page.

How we measured: Scrapy 2.18.0 with Twisted 26.10 on Python 3.13.5, one Linux server, 17 Nov 2026. nc -lk 8799 (OpenBSD netcat) for what Scrapy sends. Our test store for the 403 and for fix.py. Our API on our dev cluster for api.py, against the same store on our dev domain. The outputs are copied from those runs; log lines are cut to the message.

Limits

  • Public pages only. We do not log in, and we do not solve CAPTCHAs. When a page needs either, we stop and say so, for free.
  • A page built by JavaScript needs engine=browser. It costs more. See How to scrape JavaScript-rendered pages.
  • Read the site's terms. A 403 is sometimes the site saying no. Fetch public pages only, and follow the site's terms and robots.txt.

Why sites send 403 at all, and the other layers they check, are in Why your scraper gets 403 Forbidden.