The short answer: requests gets a 403 Forbidden when the site's bot filter decides it is not a browser, and requests says so itself: User-Agent: python-requests/2.32.3. A browser User-Agent and Accept-Language fix sites that read only the headers. Sites that also read the connection still answer 403: requests speaks HTTP/1.1 only, with Python's TLS, and no header changes that. For those, keep requests and post the URL to our API: we fetch it like a browser and climb to a residential IP or a real Chrome when the site blocks us. A block we detect costs nothing.
What Python requests sends
Start a listener in one terminal with nc -l 8799. It prints every byte a client sends and never answers, so the client times out. Run this in a second terminal:
python3 -c 'import requests; requests.get("http://127.0.0.1:8799/", timeout=3)'GET / HTTP/1.1 Host: 127.0.0.1:8799 User-Agent: python-requests/2.32.3 Accept-Encoding: gzip, deflate Accept: */* Connection: keep-alive
User-Agent: python-requests/2.32.3names the library and its version. It is the first header a bot filter reads, and no browser sends it.- Five headers, nothing else. No
Accept-Languageand noSec-Fetch-*group.Accept: */*is a script's Accept: a browser asking for a page leads withtext/html. Accept-Encoding: gzip, deflatehas nobrorzstd. requests adds those only when thebrotliorzstandardpackage is installed; current browsers offer all four.- HTTP/1.1 only. requests has no HTTP/2. On our test store,
resp.raw.versionwas11while the same page over curl was HTTP/2.
How Python requests reports a 403
Our test store answers 403 on its uploads folder, because folder listing is off. That is a real "no", not a bot filter, and a handy way to see what your code gets back:
import requests
resp = requests.get("https://lab.megalania.io/wp-content/uploads/", timeout=30)
print(resp.status_code, resp.reason, resp.ok, len(resp.content))
resp.raise_for_status()403 Forbidden False 239 Traceback (most recent call last): ... requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://lab.megalania.io/wp-content/uploads/
- requests does not raise on a 403.
get()returns normally andresp.okisFalse. Code that skipsraise_for_status()parses the 239-byte error page as if it were the data. raise_for_status()raisesrequests.exceptions.HTTPErrorwith403 Client Error: Forbidden for url: .... The error page is still inerr.response.text, and the headers inerr.response.headers.
The free fix: browser headers
Send the headers a browser sends. This fixes sites that read only the headers:
# pip install requests
import requests
session = requests.Session()
session.headers.update({
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/154.0 Safari/537.36",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"Accept-Language": "en-US,en;q=0.9",
})
resp = session.get("https://lab.megalania.io/shipping/", timeout=30)
print(resp.status_code, "HTTP version:", resp.raw.version)200 HTTP version: 11
- A
Sessionkeeps cookies between calls. Sites that set a cookie on the first page and check it on the next need that. - We sent
fix.pytonctoo. requests keeps its own order:Host,User-Agent,Accept-Encoding,Accept,Connection, thenAccept-Languagelast.Accept-Encodingis stillgzip, deflate. - The version is still
11, HTTP/1.1, and the TLS hello is still Python's. A site that checks the connection still sees requests. That is the point to stop adding headers.
The fix that climbs: our API
Send the page to our API instead. We fetch it like a real browser, move to a residential IP or a real Chrome when the site blocks us, and a block we detect costs nothing:
# pip install requests
# Run: MEG_API_KEY=YOUR_KEY python fetch.py https://lab.megalania.io/shipping/
import os
import sys
import requests
resp = requests.post(
"https://get.megalania.io/v1",
headers={"Authorization": f"Bearer {os.environ['MEG_API_KEY']}"},
json={"url": sys.argv[1], "response_types": "markdown", "max_credits": "64"},
timeout=90,
)
print("status:", resp.status_code, "target:", resp.headers.get("X-Target-Status"))
print("route:", resp.headers.get("X-Route"), "credits:", resp.headers.get("X-Credits-Charged"), "billing:", resp.headers.get("X-Billing"))
print("tries:", resp.headers.get("X-Hops"))
print(resp.text[:160])status: 200 target: 200
route: http/datacenter credits: 1 billing: None
tries: [{"rung":"h:dc","identity":"chrome","ms":1770,"outcome":"ok"}]
# Shipping
We pack every order by hand in recycled cardboard and paper tape. No plastic fillers.
| Service | Time | Price |
| --- | --- | --- |
| Standard | 3resp.status_codeis our answer to you;X-Target-Statusis what the site answered us. Both were 200 on the shipping page.X-Hopsshows one try: plain HTTP from a datacenter IP with a Chrome fingerprint, 1,770 ms.X-Credits-Charged: 1. The body is the page as Markdown, its table kept.
The same call on the uploads folder that answers 403 to everyone:
MEG_API_KEY=YOUR_KEY python fetch.py https://lab.megalania.io/wp-content/uploads/
status: 422 target: 403
route: browser/residential credits: None billing: blocked
tries: [{"rung":"h:dc","identity":"firefox","ms":4728,"outcome":"blocked"},{"rung":"h:res","identity":"firefox","ms":4854,"outcome":"blocked"},{"rung":"c:dc","identity":"chrome","ms":1343,"outcome":"blocked"},{"rung":"c:res","identity":"chrome","ms":1991,"outcome":"blocked"}]
{"error":{"code":"blocked","message":"The site blocked us and there is nothing to show. You are not charged. X-Untried lists the routes left to try; no X-Untrie- We tried all four routes, and the folder said 403 to each, so the answer is
422witherror.codeblocked.X-Billing: blockedand noX-Credits-Charged: it cost 0. - In Python,
resp.headers.get("X-Credits-Charged")isNoneon a block. TreatNoneas 0 when you add up a run's cost.
One honest note: our test store blocks nobody, so these runs show the calls, not a bot filter being beaten. On a site that blocks, the call is the same; the price depends on the route that worked, and our MCP tool check_site tells you that price for free.
What it costs
| What we ran | Route | Credits |
|---|---|---|
| The shipping page as Markdown | h:dc | 1 |
| The uploads folder: 403 on every route | h:dc h:res c:dc c:res | 0 |
seen.py and fix.py, straight from your machine | none | 0 |
Every rule is in Credits. Plans are on the pricing page.
How we measured: requests 2.32.3 on Python 3.13.5, one Linux server, 16 Nov 2026. nc -l 8799 (OpenBSD netcat) for what requests sends, with and without the browser headers. Our test store for the 403 and for fix.py. Our API on our dev cluster for fetch.py, against the same store on our dev domain. The outputs are copied from those runs; the traceback is shortened.
Limits
- Public pages only. We do not log in, and we do not solve CAPTCHAs. When a page needs either, we stop and say so, for free.
- A page built by JavaScript needs
engine=browser. It costs more. See How to scrape JavaScript-rendered pages. - Read the site's terms. A 403 is sometimes the site saying no. Fetch public pages only, and follow the site's terms and robots.txt.
Why sites send 403 at all, and the other layers they check, are in Why your scraper gets 403 Forbidden.