MegalaniaBlog
  1. Blog
  2. How to fix
  3. requests 403

Python requests 403 Forbidden: why, and the fix

Python requests tells every site it is a script. What it sends, how it reports a 403, what headers fix, and a call that climbs. With requests 2.32.3.

Jordan HayesDeveloper Relations
3 min read
How to fix
A padlock cut open. Inside, four pins named UA, headers, cookies and TLS. Three sit on the line, one sits low.

The short answer: requests gets a 403 Forbidden when the site's bot filter decides it is not a browser, and requests says so itself: User-Agent: python-requests/2.32.3. A browser User-Agent and Accept-Language fix sites that read only the headers. Sites that also read the connection still answer 403: requests speaks HTTP/1.1 only, with Python's TLS, and no header changes that. For those, keep requests and post the URL to our API: we fetch it like a browser and climb to a residential IP or a real Chrome when the site blocks us. A block we detect costs nothing.

What Python requests sends

Start a listener in one terminal with nc -l 8799. It prints every byte a client sends and never answers, so the client times out. Run this in a second terminal:

Terminal 2
python3 -c 'import requests; requests.get("http://127.0.0.1:8799/", timeout=3)'
Output
GET / HTTP/1.1
Host: 127.0.0.1:8799
User-Agent: python-requests/2.32.3
Accept-Encoding: gzip, deflate
Accept: */*
Connection: keep-alive
  • User-Agent: python-requests/2.32.3 names the library and its version. It is the first header a bot filter reads, and no browser sends it.
  • Five headers, nothing else. No Accept-Language and no Sec-Fetch-* group. Accept: */* is a script's Accept: a browser asking for a page leads with text/html.
  • Accept-Encoding: gzip, deflate has no br or zstd. requests adds those only when the brotli or zstandard package is installed; current browsers offer all four.
  • HTTP/1.1 only. requests has no HTTP/2. On our test store, resp.raw.version was 11 while the same page over curl was HTTP/2.

How Python requests reports a 403

Our test store answers 403 on its uploads folder, because folder listing is off. That is a real "no", not a bot filter, and a handy way to see what your code gets back:

seen.py
import requests

resp = requests.get("https://lab.megalania.io/wp-content/uploads/", timeout=30)
print(resp.status_code, resp.reason, resp.ok, len(resp.content))
resp.raise_for_status()
Output
403 Forbidden False 239
Traceback (most recent call last):
  ...
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://lab.megalania.io/wp-content/uploads/
  • requests does not raise on a 403. get() returns normally and resp.ok is False. Code that skips raise_for_status() parses the 239-byte error page as if it were the data.
  • raise_for_status() raises requests.exceptions.HTTPError with 403 Client Error: Forbidden for url: .... The error page is still in err.response.text, and the headers in err.response.headers.

The free fix: browser headers

Send the headers a browser sends. This fixes sites that read only the headers:

fix.py
# pip install requests
import requests

session = requests.Session()
session.headers.update({
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/154.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.9",
})
resp = session.get("https://lab.megalania.io/shipping/", timeout=30)
print(resp.status_code, "HTTP version:", resp.raw.version)
Output
200 HTTP version: 11
  • A Session keeps cookies between calls. Sites that set a cookie on the first page and check it on the next need that.
  • We sent fix.py to nc too. requests keeps its own order: Host, User-Agent, Accept-Encoding, Accept, Connection, then Accept-Language last. Accept-Encoding is still gzip, deflate.
  • The version is still 11, HTTP/1.1, and the TLS hello is still Python's. A site that checks the connection still sees requests. That is the point to stop adding headers.
Free beta, by invite76,000 credits for 30 days, no card. Apply, and we email your invite.Apply for the beta

The fix that climbs: our API

Send the page to our API instead. We fetch it like a real browser, move to a residential IP or a real Chrome when the site blocks us, and a block we detect costs nothing:

fetch.py
# pip install requests
# Run: MEG_API_KEY=YOUR_KEY python fetch.py https://lab.megalania.io/shipping/
import os
import sys

import requests

resp = requests.post(
    "https://get.megalania.io/v1",
    headers={"Authorization": f"Bearer {os.environ['MEG_API_KEY']}"},
    json={"url": sys.argv[1], "response_types": "markdown", "max_credits": "64"},
    timeout=90,
)
print("status:", resp.status_code, "target:", resp.headers.get("X-Target-Status"))
print("route:", resp.headers.get("X-Route"), "credits:", resp.headers.get("X-Credits-Charged"), "billing:", resp.headers.get("X-Billing"))
print("tries:", resp.headers.get("X-Hops"))
print(resp.text[:160])
Output
status: 200 target: 200
route: http/datacenter credits: 1 billing: None
tries: [{"rung":"h:dc","identity":"chrome","ms":1770,"outcome":"ok"}]
# Shipping

We pack every order by hand in recycled cardboard and paper tape. No plastic fillers.

| Service | Time | Price |
| --- | --- | --- |
| Standard | 3
  • resp.status_code is our answer to you; X-Target-Status is what the site answered us. Both were 200 on the shipping page.
  • X-Hops shows one try: plain HTTP from a datacenter IP with a Chrome fingerprint, 1,770 ms. X-Credits-Charged: 1. The body is the page as Markdown, its table kept.

The same call on the uploads folder that answers 403 to everyone:

Terminal
MEG_API_KEY=YOUR_KEY python fetch.py https://lab.megalania.io/wp-content/uploads/
Output
status: 422 target: 403
route: browser/residential credits: None billing: blocked
tries: [{"rung":"h:dc","identity":"firefox","ms":4728,"outcome":"blocked"},{"rung":"h:res","identity":"firefox","ms":4854,"outcome":"blocked"},{"rung":"c:dc","identity":"chrome","ms":1343,"outcome":"blocked"},{"rung":"c:res","identity":"chrome","ms":1991,"outcome":"blocked"}]
{"error":{"code":"blocked","message":"The site blocked us and there is nothing to show. You are not charged. X-Untried lists the routes left to try; no X-Untrie
  • We tried all four routes, and the folder said 403 to each, so the answer is 422 with error.code blocked. X-Billing: blocked and no X-Credits-Charged: it cost 0.
  • In Python, resp.headers.get("X-Credits-Charged") is None on a block. Treat None as 0 when you add up a run's cost.

One honest note: our test store blocks nobody, so these runs show the calls, not a bot filter being beaten. On a site that blocks, the call is the same; the price depends on the route that worked, and our MCP tool check_site tells you that price for free.

What it costs

What we ranRouteCredits
The shipping page as Markdownh:dc1
The uploads folder: 403 on every routeh:dc h:res c:dc c:res0
seen.py and fix.py, straight from your machinenone0

Every rule is in Credits. Plans are on the pricing page.

How we measured: requests 2.32.3 on Python 3.13.5, one Linux server, 16 Nov 2026. nc -l 8799 (OpenBSD netcat) for what requests sends, with and without the browser headers. Our test store for the 403 and for fix.py. Our API on our dev cluster for fetch.py, against the same store on our dev domain. The outputs are copied from those runs; the traceback is shortened.

Limits

  • Public pages only. We do not log in, and we do not solve CAPTCHAs. When a page needs either, we stop and say so, for free.
  • A page built by JavaScript needs engine=browser. It costs more. See How to scrape JavaScript-rendered pages.
  • Read the site's terms. A 403 is sometimes the site saying no. Fetch public pages only, and follow the site's terms and robots.txt.

Why sites send 403 at all, and the other layers they check, are in Why your scraper gets 403 Forbidden.