The short answer: urllib.request.urlopen gets HTTP Error 403: Forbidden when a site reads its request as a script, and urllib makes that easy: User-Agent: Python-urllib/3.13, Accept-Encoding: identity, Connection: close, and no Accept at all. Unlike requests, urllib raises HTTPError on a 403. The error is also the response, so err.code, err.headers and err.read() all work. Build a Request with browser headers to pass sites that read only headers. urllib speaks HTTP/1.1 with Python's TLS, and no setting changes that. For sites that check more, post the page to our API with urllib too: a block we detect costs nothing.
What urllib sends
Start a listener in one terminal with nc -l 8799. It prints every byte a client sends and never answers, so the client times out. Run this in a second terminal:
python -c 'import urllib.request; urllib.request.urlopen("http://127.0.0.1:8799/", timeout=3)'GET / HTTP/1.1 Accept-Encoding: identity Host: 127.0.0.1:8799 User-Agent: Python-urllib/3.13 Connection: close
User-Agent: Python-urllib/3.13names the language and its version. It is the oldest script name on the web, and every bot list has it.- No
Acceptline, andAccept-Encoding: identity: urllib asks for no compression, because it cannot undo it. A browser always asks for gzip or better. Connection: close: urllib opens a new connection for every request.
How urllib reports a 403
Our test store answers 403 on its uploads folder, because folder listing is off. That is a real "no", not a bot filter, and a handy way to see what your code gets back:
from urllib.error import HTTPError
from urllib.request import urlopen
try:
urlopen("https://lab.megalania.io/wp-content/uploads/", timeout=10)
except HTTPError as err:
print(type(err).__name__, "|", err)
print("code:", err.code, "| content-type:", err.headers["Content-Type"], "| body:", len(err.read()), "bytes")HTTPError | HTTP Error 403: Forbidden code: 403 | content-type: text/html; charset=iso-8859-1 | body: 239 bytes
urlopenraised. urllib turns every 4xx and 5xx answer intoHTTPError, unlike requests, which returns it.- The error doubles as the response:
err.read()gives the 239-byte error page,err.headersthe headers. Read them before you retry.
The free fix: browser headers
Send the headers a browser sends. This fixes sites that read only the headers:
import html
import re
from urllib.request import Request, urlopen
HEADERS = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/154.0 Safari/537.36",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"Accept-Language": "en-US,en;q=0.9",
}
req = Request("https://lab.megalania.io/journal/", headers=HEADERS)
with urlopen(req, timeout=10) as res:
page = res.read().decode()
print(res.status, res.headers["Content-Type"], "|", html.unescape(re.search(r"<title>(.*?)</title>", page).group(1)))200 text/html; charset=UTF-8 | Journal – Gibber & Co.
Request(url, headers=...)replaces urllib'sUser-Agentand addsAcceptandAccept-Language.- Leave
Accept-Encodingout: urllib does not decompress, so asking for gzip gets you bytes you cannot read withoutgzip.decompress.
The fix that climbs: our API
Send the page to our API instead. We fetch it like a real browser, move to a residential IP or a real Chrome when the site blocks us, and a block we detect costs nothing:
# Run: MEG_API_KEY=YOUR_KEY python fetch.py https://lab.megalania.io/shipping/
import json
import os
import sys
from urllib.error import HTTPError
from urllib.request import Request, urlopen
body = json.dumps({"url": sys.argv[1], "response_types": "markdown"}).encode()
req = Request(
"https://get.megalania.io/v1",
data=body,
headers={"Authorization": f"Bearer {os.environ['MEG_API_KEY']}", "Content-Type": "application/json"},
)
try:
with urlopen(req, timeout=100) as res:
print(res.status, res.headers["X-Route"], "credits", res.headers["X-Credits-Charged"])
print(res.read().decode()[:110])
except HTTPError as err:
error = json.load(err)["error"]
print(err.code, err.headers["X-Target-Status"], "credits", err.headers.get("X-Credits-Charged", 0), "|", error["code"])
print(error["message"][:110])200 http/datacenter credits 1 . X-Target-Status: 403: the site said no on every route we tried. A block costs 0 credits.
One honest note: our test store blocks nobody, so these runs show the calls, not a bot filter being beaten. On a site that blocks, the call is the same; the price depends on the route that worked, and our MCP tool check_site tells you that price for free.
What it costs
| What we ran | Route | Credits |
|---|---|---|
| The about page as Markdown | h:dc | 1 |
| The uploads folder: 403 on every route | h:dc h:res c:dc c:res | 0 |
seen.py and fix.py, straight from urllib | none | 0 |
Every rule is in Credits. Plans are on the pricing page.
How we measured: urllib from Python 3.13.5, standard library only, one Linux server, 17 Nov 2026. nc -l 8799 (OpenBSD netcat) for what urllib sends. Our test store for the 403 and for fix.py. Our API on our dev cluster for fetch.py, against the same store on our dev domain. The outputs are copied from those runs.
Limits
- Public pages only. We do not log in, and we do not solve CAPTCHAs. When a page needs either, we stop and say so, for free.
- A page built by JavaScript needs
engine=browser. It costs more. See How to scrape JavaScript-rendered pages. - Read the site's terms. A 403 is sometimes the site saying no. Fetch public pages only, and follow the site's terms and robots.txt.
Why sites send 403 at all, and the other layers they check, are in Why your scraper gets 403 Forbidden.