MegalaniaBlog
  1. Blog
  2. How to fix
  3. httpx 403

httpx 403 Forbidden: why it happens, and the fix

httpx names itself in every request. What it sends, how a 403 comes back, how HTTP/2 helps, and the call that climbs. Tested with httpx 0.28.1.

Jordan HayesDeveloper Relations
2 min read
How to fix
A padlock cut open. Inside, four pins named UA, headers, cookies and TLS. Three sit on the line, one sits low.

The short answer: An httpx request that gets 403 Forbidden was most likely read as a script, and httpx does not hide it: User-Agent: python-httpx/0.28.1, Accept: */*, nothing about language. Unlike requests, httpx can speak HTTP/2, the protocol browsers use, once you install h2 and pass http2=True. Add browser header values to that and you pass sites that check headers and protocol. httpx keeps Python's TLS hello, and no setting changes it. When a site checks that too, send the page to our API with the same httpx call shape. Our answer tells you which route worked, and a block we detect costs 0 credits.

What httpx sends

Start a listener in one terminal with nc -l 8799. It prints every byte a client sends and never answers, so the client times out. Run this in a second terminal:

Terminal 2
python -c 'import httpx; httpx.get("http://127.0.0.1:8799/", timeout=3)'
Output
GET / HTTP/1.1
Host: 127.0.0.1:8799
Accept: */*
Accept-Encoding: gzip, deflate, br
Connection: keep-alive
User-Agent: python-httpx/0.28.1
  • User-Agent: python-httpx/0.28.1 names the library and its exact version. A filter that keeps a list of script names stops it on that line alone.
  • **No Accept-Language and Accept: */*.** A page load in a browser asks for text/html first and names a language.
  • Host comes first, as in a browser, but User-Agent comes last, after Connection. Browsers put it near the top.

How httpx reports a 403

Our test store answers 403 on its uploads folder, because folder listing is off. That is a real "no", not a bot filter, and a handy way to see what your code gets back:

seen.py
import httpx

res = httpx.get("https://lab.megalania.io/wp-content/uploads/")
print(res.status_code, res.reason_phrase, res.http_version, "|", len(res.text), "characters")
try:
    res.raise_for_status()
except httpx.HTTPStatusError as err:
    print(type(err).__name__, "|", str(err).splitlines()[0])
Output
403 Forbidden HTTP/1.1 | 239 characters
HTTPStatusError | Client error '403 Forbidden' for url 'https://lab.megalania.io/wp-content/uploads/'
  • httpx returns the 403 as a normal response. Nothing is raised until you call res.raise_for_status(), which raises httpx.HTTPStatusError.
  • Without http2=True, httpx spoke HTTP/1.1 to a store that also serves HTTP/2.

The free fix: browser headers

Send the headers a browser sends. This fixes sites that read only the headers:

fix.py
import httpx

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/154.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.9",
}
with httpx.Client(http2=True, headers=HEADERS) as client:
    res = client.get("https://lab.megalania.io/shipping/")
    print(res.status_code, res.http_version, res.headers["content-type"])
Output
200 HTTP/2 text/html; charset=UTF-8
  • http2=True needs pip install 'httpx[http2]' (the h2 package). With it, httpx agreed HTTP/2 with the store, as a browser does. requests cannot do that.
  • We sent the same client to nc: your User-Agent, Accept and Accept-Language replaced the defaults, but Accept-Encoding and Connection stayed in httpx's order, before them.
  • The TLS hello is still Python's. A filter that reads it sees the same client as before.
Free beta, by invite76,000 credits for 30 days, no card. Apply, and we email your invite.Apply for the beta

The fix that climbs: our API

Send the page to our API instead. We fetch it like a real browser, move to a residential IP or a real Chrome when the site blocks us, and a block we detect costs nothing:

fetch.py
# Run: MEG_API_KEY=YOUR_KEY python fetch.py https://lab.megalania.io/shipping/
import os
import sys

import httpx

res = httpx.post(
    "https://get.megalania.io/v1",
    headers={"Authorization": f"Bearer {os.environ['MEG_API_KEY']}"},
    json={"url": sys.argv[1], "response_types": "markdown", "max_credits": "64"},
    timeout=100,
)
print(res.status_code, "| target", res.headers.get("x-target-status"), "| route", res.headers.get("x-route"))
print("credits", res.headers.get("x-credits-charged"), "| billing", res.headers.get("x-billing"), "| hops", res.headers.get("x-hops"))
print(res.text[:150])
Output
200 | target 200 | route http/datacenter
credits 1 | billing None | hops [{"rung":"h:dc","identity":"chrome","ms":1526,"outcome":"ok"}]
# Shipping

We pack every order by hand in recycled cardboard and paper tape. No plastic fillers.

| Service | Time | Price |
| --- | --- | --- |
| St
  • x-hops lists each try: one, on h:dc, plain HTTP from a data center, with a Chrome identity, in 1,526 ms.
  • The shipping table came back as a Markdown table, ready for a parser or a model.

The same call on the uploads folder that answers 403 to everyone:

Terminal
MEG_API_KEY=YOUR_KEY python fetch.py https://lab.megalania.io/wp-content/uploads/
Output
422 | target 403 | route browser/residential
credits None | billing blocked | hops [{"rung":"h:dc","identity":"firefox","ms":4448,"outcome":"blocked"},{"rung":"h:res","identity":"firefox","ms":4938,"outcome":"blocked"},{"rung":"c:dc","identity":"chrome","ms":1099,"outcome":"blocked"},{"rung":"c:res","identity":"chrome","ms":1520,"outcome":"blocked"}]
{"error":{"code":"blocked","message":"The site blocked us and there is nothing to show. You are not charged. X-Untried lists the routes left to try; n
  • We climbed from a data center to a residential IP, then to a real Chrome on each: four 403s, and one 422 with x-billing: blocked.
  • credits None: httpx returns None for a header that is not there, and a block has no charge header.

One honest note: our test store blocks nobody, so these runs show the calls, not a bot filter being beaten. On a site that blocks, the call is the same; the price depends on the route that worked, and our MCP tool check_site tells you that price for free.

What it costs

What we ranRouteCredits
The shipping page as Markdownh:dc1
The uploads folder: 403 on every routeh:dc h:res c:dc c:res0
seen.py and fix.py, straight from httpxnone0

Every rule is in Credits. Plans are on the pricing page.

How we measured: httpx 0.28.1 with h2 4.4.1 on Python 3.13.5, one Linux server, 17 Nov 2026. nc -l 8799 (OpenBSD netcat) for what httpx sends, with and without the browser headers. Our test store for the 403 and for fix.py. Our API on our dev cluster for fetch.py, against the same store on our dev domain. The outputs are copied from those runs.

Limits

  • Public pages only. We do not log in, and we do not solve CAPTCHAs. When a page needs either, we stop and say so, for free.
  • A page built by JavaScript needs engine=browser. It costs more. See How to scrape JavaScript-rendered pages.
  • Read the site's terms. A 403 is sometimes the site saying no. Fetch public pages only, and follow the site's terms and robots.txt.

Why sites send 403 at all, and the other layers they check, are in Why your scraper gets 403 Forbidden.