The short answer: Scrapy gets 403 Forbidden when a site's bot filter sees its default request: User-Agent: Scrapy/2.18.0 (+https://scrapy.org) and Accept-Language: en. Then the 403 never reaches your parse(): HttpErrorMiddleware logs Ignoring response and moves on, so a spider can finish with no items and no error. Set USER_AGENT and DEFAULT_REQUEST_HEADERS to browser values, and add handle_httpstatus_list where you want to see the 403 yourself. Scrapy's connection stays HTTP/1.1 with Python's TLS. For sites that check those, yield one JsonRequest to our API per page instead. A block we detect costs nothing.
What Scrapy sends
Start a listener in one terminal with nc -l 8799. It prints every byte a client sends and never answers, so the client times out. Run this in a second terminal:
scrapy fetch --nolog -s DOWNLOAD_TIMEOUT=3 -s RETRY_ENABLED=False http://127.0.0.1:8799/
GET / HTTP/1.1 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Language: en User-Agent: Scrapy/2.18.0 (+https://scrapy.org) Accept-Encoding: gzip, deflate, br, zstd Host: 127.0.0.1:8799
- The
User-AgentisScrapy/2.18.0with a link to the project. It is polite and honest, and it is the first thing a bot filter blocks. Accept-Language: en, a bare language with no region and no weights. Browsers send something likeen-US,en;q=0.9.Hostcomes last. Scrapy'sAcceptlooks like a browser's, but the order of the lines does not.
How Scrapy reports a 403
Our test store answers 403 on its uploads folder, because folder listing is off. That is a real "no", not a bot filter, and a handy way to see what your code gets back:
import scrapy
class Uploads(scrapy.Spider):
name = "uploads"
start_urls = ["https://lab.megalania.io/wp-content/uploads/"]
def parse(self, response):
print("parse got", response.status)Crawled (403) <GET https://lab.megalania.io/wp-content/uploads/> (referer: None) Ignoring response <403 https://lab.megalania.io/wp-content/uploads/>: HTTP status code is not handled or not allowed 'httperror/response_ignored_count': 1, 'httperror/response_ignored_status_count/403': 1,
parse()never ran.parse gotis not in the output: the 403 stopped atHttpErrorMiddleware, which logged it at INFO and counted it in the stats.- Search your stats for
httperror/response_ignored_status_count/403. It is the only place a quiet run shows that a site said no.
The free fix: browser headers
Send the headers a browser sends. This fixes sites that read only the headers:
import scrapy
class About(scrapy.Spider):
name = "about"
start_urls = ["https://lab.megalania.io/about/"]
custom_settings = {
"USER_AGENT": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/154.0 Safari/537.36",
"DEFAULT_REQUEST_HEADERS": {
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"Accept-Language": "en-US,en;q=0.9",
},
"LOG_LEVEL": "ERROR",
}
def parse(self, response):
print(response.status, response.css("h1::text").get(), "|", response.request.headers.get("User-Agent").decode()[:40])200 About | Mozilla/5.0 (Windows NT 10.0; Win64; x64
USER_AGENTreplaces the Scrapy name on every request.DEFAULT_REQUEST_HEADERSreplaces the defaultAcceptandAccept-Languagepair.response.request.headersshows what was really sent, so you can check the fix from inside the spider.- A
scrapy startprojectproject also obeys robots.txt by default (ROBOTSTXT_OBEY = True). Keep it on.
The fix that climbs: our API
Send the page to our API instead. We fetch it like a real browser, move to a residential IP or a real Chrome when the site blocks us, and a block we detect costs nothing:
# Run: MEG_API_KEY=YOUR_KEY scrapy runspider api.py -a page=https://lab.megalania.io/returns/
import os
import scrapy
class ViaApi(scrapy.Spider):
name = "via_api"
custom_settings = {"DOWNLOAD_TIMEOUT": 100, "LOG_LEVEL": "ERROR"}
async def start(self):
body = {"url": self.page, "response_types": "markdown", "max_credits": "64"}
body.update({k: v for k, v in vars(self).items() if k in ("engine", "actions", "timeout")})
yield scrapy.http.JsonRequest(
"https://get.megalania.io/v1",
data=body,
headers={"Authorization": f"Bearer {os.environ['MEG_API_KEY']}"},
meta={"handle_httpstatus_list": [422, 503]},
)
def parse(self, response):
h = lambda k: (response.headers.get(k) or b"-").decode()
print("status:", response.status, "route:", h("X-Route"), "credits:", h("X-Credits-Charged"))
print(response.text[:150])status: 200 route: http/datacenter credits: 1  # About Gibber & Co. is a demo store
- One
JsonRequestper page.handle_httpstatus_listlets our422and503reachparse, so you can readerror.codeinstead of a log line. - The about page came back as Markdown, its first image as a Markdown image.
The same call on the uploads folder that answers 403 to everyone:
MEG_API_KEY=YOUR_KEY scrapy runspider api.py -a page=https://lab.megalania.io/wp-content/uploads/
status: 422 route: browser/residential credits: -
{"error":{"code":"blocked","message":"The site blocked us and there is nothing to show. You are not charged. X-Untried lists the routes left to try; n- This time the answer reached
parse, because422is inhandle_httpstatus_list. Four routes, four 403s, oneblocked. credits: -means no charge header: count it as 0. Our error JSON says what to try next.
One honest note: our test store blocks nobody, so these runs show the calls, not a bot filter being beaten. On a site that blocks, the call is the same; the price depends on the route that worked, and our MCP tool check_site tells you that price for free.
What it costs
| What we ran | Route | Credits |
|---|---|---|
| The about page as Markdown | h:dc | 1 |
| The uploads folder: 403 on every route | h:dc h:res c:dc c:res | 0 |
seen.py and fix.py, straight from Scrapy | none | 0 |
Every rule is in Credits. Plans are on the pricing page.
How we measured: Scrapy 2.18.0 with Twisted 26.10 on Python 3.13.5, one Linux server, 17 Nov 2026. nc -lk 8799 (OpenBSD netcat) for what Scrapy sends. Our test store for the 403 and for fix.py. Our API on our dev cluster for api.py, against the same store on our dev domain. The outputs are copied from those runs; log lines are cut to the message.
Limits
- Public pages only. We do not log in, and we do not solve CAPTCHAs. When a page needs either, we stop and say so, for free.
- A page built by JavaScript needs
engine=browser. It costs more. See How to scrape JavaScript-rendered pages. - Read the site's terms. A 403 is sometimes the site saying no. Fetch public pages only, and follow the site's terms and robots.txt.
Why sites send 403 at all, and the other layers they check, are in Why your scraper gets 403 Forbidden.