The short answer: An httpx request that gets 403 Forbidden was most likely read as a script, and httpx does not hide it: User-Agent: python-httpx/0.28.1, Accept: */*, nothing about language. Unlike requests, httpx can speak HTTP/2, the protocol browsers use, once you install h2 and pass http2=True. Add browser header values to that and you pass sites that check headers and protocol. httpx keeps Python's TLS hello, and no setting changes it. When a site checks that too, send the page to our API with the same httpx call shape. Our answer tells you which route worked, and a block we detect costs 0 credits.
What httpx sends
Start a listener in one terminal with nc -l 8799. It prints every byte a client sends and never answers, so the client times out. Run this in a second terminal:
python -c 'import httpx; httpx.get("http://127.0.0.1:8799/", timeout=3)'GET / HTTP/1.1 Host: 127.0.0.1:8799 Accept: */* Accept-Encoding: gzip, deflate, br Connection: keep-alive User-Agent: python-httpx/0.28.1
User-Agent: python-httpx/0.28.1names the library and its exact version. A filter that keeps a list of script names stops it on that line alone.- **No
Accept-LanguageandAccept: */*.** A page load in a browser asks fortext/htmlfirst and names a language. Hostcomes first, as in a browser, butUser-Agentcomes last, afterConnection. Browsers put it near the top.
How httpx reports a 403
Our test store answers 403 on its uploads folder, because folder listing is off. That is a real "no", not a bot filter, and a handy way to see what your code gets back:
import httpx
res = httpx.get("https://lab.megalania.io/wp-content/uploads/")
print(res.status_code, res.reason_phrase, res.http_version, "|", len(res.text), "characters")
try:
res.raise_for_status()
except httpx.HTTPStatusError as err:
print(type(err).__name__, "|", str(err).splitlines()[0])403 Forbidden HTTP/1.1 | 239 characters HTTPStatusError | Client error '403 Forbidden' for url 'https://lab.megalania.io/wp-content/uploads/'
- httpx returns the 403 as a normal response. Nothing is raised until you call
res.raise_for_status(), which raiseshttpx.HTTPStatusError. - Without
http2=True, httpx spoke HTTP/1.1 to a store that also serves HTTP/2.
The free fix: browser headers
Send the headers a browser sends. This fixes sites that read only the headers:
import httpx
HEADERS = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/154.0 Safari/537.36",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"Accept-Language": "en-US,en;q=0.9",
}
with httpx.Client(http2=True, headers=HEADERS) as client:
res = client.get("https://lab.megalania.io/shipping/")
print(res.status_code, res.http_version, res.headers["content-type"])200 HTTP/2 text/html; charset=UTF-8
http2=Trueneedspip install 'httpx[http2]'(theh2package). With it, httpx agreed HTTP/2 with the store, as a browser does. requests cannot do that.- We sent the same client to
nc: yourUser-Agent,AcceptandAccept-Languagereplaced the defaults, butAccept-EncodingandConnectionstayed in httpx's order, before them. - The TLS hello is still Python's. A filter that reads it sees the same client as before.
The fix that climbs: our API
Send the page to our API instead. We fetch it like a real browser, move to a residential IP or a real Chrome when the site blocks us, and a block we detect costs nothing:
# Run: MEG_API_KEY=YOUR_KEY python fetch.py https://lab.megalania.io/shipping/
import os
import sys
import httpx
res = httpx.post(
"https://get.megalania.io/v1",
headers={"Authorization": f"Bearer {os.environ['MEG_API_KEY']}"},
json={"url": sys.argv[1], "response_types": "markdown", "max_credits": "64"},
timeout=100,
)
print(res.status_code, "| target", res.headers.get("x-target-status"), "| route", res.headers.get("x-route"))
print("credits", res.headers.get("x-credits-charged"), "| billing", res.headers.get("x-billing"), "| hops", res.headers.get("x-hops"))
print(res.text[:150])200 | target 200 | route http/datacenter
credits 1 | billing None | hops [{"rung":"h:dc","identity":"chrome","ms":1526,"outcome":"ok"}]
# Shipping
We pack every order by hand in recycled cardboard and paper tape. No plastic fillers.
| Service | Time | Price |
| --- | --- | --- |
| Stx-hopslists each try: one, onh:dc, plain HTTP from a data center, with a Chrome identity, in 1,526 ms.- The shipping table came back as a Markdown table, ready for a parser or a model.
The same call on the uploads folder that answers 403 to everyone:
MEG_API_KEY=YOUR_KEY python fetch.py https://lab.megalania.io/wp-content/uploads/
422 | target 403 | route browser/residential
credits None | billing blocked | hops [{"rung":"h:dc","identity":"firefox","ms":4448,"outcome":"blocked"},{"rung":"h:res","identity":"firefox","ms":4938,"outcome":"blocked"},{"rung":"c:dc","identity":"chrome","ms":1099,"outcome":"blocked"},{"rung":"c:res","identity":"chrome","ms":1520,"outcome":"blocked"}]
{"error":{"code":"blocked","message":"The site blocked us and there is nothing to show. You are not charged. X-Untried lists the routes left to try; n- We climbed from a data center to a residential IP, then to a real Chrome on each: four 403s, and one
422withx-billing: blocked. credits None: httpx returnsNonefor a header that is not there, and a block has no charge header.
One honest note: our test store blocks nobody, so these runs show the calls, not a bot filter being beaten. On a site that blocks, the call is the same; the price depends on the route that worked, and our MCP tool check_site tells you that price for free.
What it costs
| What we ran | Route | Credits |
|---|---|---|
| The shipping page as Markdown | h:dc | 1 |
| The uploads folder: 403 on every route | h:dc h:res c:dc c:res | 0 |
seen.py and fix.py, straight from httpx | none | 0 |
Every rule is in Credits. Plans are on the pricing page.
How we measured: httpx 0.28.1 with h2 4.4.1 on Python 3.13.5, one Linux server, 17 Nov 2026. nc -l 8799 (OpenBSD netcat) for what httpx sends, with and without the browser headers. Our test store for the 403 and for fix.py. Our API on our dev cluster for fetch.py, against the same store on our dev domain. The outputs are copied from those runs.
Limits
- Public pages only. We do not log in, and we do not solve CAPTCHAs. When a page needs either, we stop and say so, for free.
- A page built by JavaScript needs
engine=browser. It costs more. See How to scrape JavaScript-rendered pages. - Read the site's terms. A 403 is sometimes the site saying no. Fetch public pages only, and follow the site's terms and robots.txt.
Why sites send 403 at all, and the other layers they check, are in Why your scraper gets 403 Forbidden.