MegalaniaBlog
  1. Blog
  2. How to fix
  3. urllib 403

urllib HTTP Error 403: Forbidden: why it happens, and the fix

urllib names itself Python-urllib and raises HTTPError on a 403. What it sends, how to read the error, the Request headers fix, and our API.

Jordan HayesDeveloper Relations
2 min read
How to fix
A padlock cut open. Inside, four pins named UA, headers, cookies and TLS. Three sit on the line, one sits low.

The short answer: urllib.request.urlopen gets HTTP Error 403: Forbidden when a site reads its request as a script, and urllib makes that easy: User-Agent: Python-urllib/3.13, Accept-Encoding: identity, Connection: close, and no Accept at all. Unlike requests, urllib raises HTTPError on a 403. The error is also the response, so err.code, err.headers and err.read() all work. Build a Request with browser headers to pass sites that read only headers. urllib speaks HTTP/1.1 with Python's TLS, and no setting changes that. For sites that check more, post the page to our API with urllib too: a block we detect costs nothing.

What urllib sends

Start a listener in one terminal with nc -l 8799. It prints every byte a client sends and never answers, so the client times out. Run this in a second terminal:

Terminal 2
python -c 'import urllib.request; urllib.request.urlopen("http://127.0.0.1:8799/", timeout=3)'
Output
GET / HTTP/1.1
Accept-Encoding: identity
Host: 127.0.0.1:8799
User-Agent: Python-urllib/3.13
Connection: close
  • User-Agent: Python-urllib/3.13 names the language and its version. It is the oldest script name on the web, and every bot list has it.
  • No Accept line, and Accept-Encoding: identity: urllib asks for no compression, because it cannot undo it. A browser always asks for gzip or better.
  • Connection: close: urllib opens a new connection for every request.

How urllib reports a 403

Our test store answers 403 on its uploads folder, because folder listing is off. That is a real "no", not a bot filter, and a handy way to see what your code gets back:

seen.py
from urllib.error import HTTPError
from urllib.request import urlopen

try:
    urlopen("https://lab.megalania.io/wp-content/uploads/", timeout=10)
except HTTPError as err:
    print(type(err).__name__, "|", err)
    print("code:", err.code, "| content-type:", err.headers["Content-Type"], "| body:", len(err.read()), "bytes")
Output
HTTPError | HTTP Error 403: Forbidden
code: 403 | content-type: text/html; charset=iso-8859-1 | body: 239 bytes
  • urlopen raised. urllib turns every 4xx and 5xx answer into HTTPError, unlike requests, which returns it.
  • The error doubles as the response: err.read() gives the 239-byte error page, err.headers the headers. Read them before you retry.

The free fix: browser headers

Send the headers a browser sends. This fixes sites that read only the headers:

fix.py
import html
import re
from urllib.request import Request, urlopen

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/154.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.9",
}
req = Request("https://lab.megalania.io/journal/", headers=HEADERS)
with urlopen(req, timeout=10) as res:
    page = res.read().decode()
print(res.status, res.headers["Content-Type"], "|", html.unescape(re.search(r"<title>(.*?)</title>", page).group(1)))
Output
200 text/html; charset=UTF-8 | Journal – Gibber & Co.
  • Request(url, headers=...) replaces urllib's User-Agent and adds Accept and Accept-Language.
  • Leave Accept-Encoding out: urllib does not decompress, so asking for gzip gets you bytes you cannot read without gzip.decompress.
Free beta, by invite76,000 credits for 30 days, no card. Apply, and we email your invite.Apply for the beta

The fix that climbs: our API

Send the page to our API instead. We fetch it like a real browser, move to a residential IP or a real Chrome when the site blocks us, and a block we detect costs nothing:

fetch.py
# Run: MEG_API_KEY=YOUR_KEY python fetch.py https://lab.megalania.io/shipping/
import json
import os
import sys
from urllib.error import HTTPError
from urllib.request import Request, urlopen

body = json.dumps({"url": sys.argv[1], "response_types": "markdown"}).encode()
req = Request(
    "https://get.megalania.io/v1",
    data=body,
    headers={"Authorization": f"Bearer {os.environ['MEG_API_KEY']}", "Content-Type": "application/json"},
)
try:
    with urlopen(req, timeout=100) as res:
        print(res.status, res.headers["X-Route"], "credits", res.headers["X-Credits-Charged"])
        print(res.read().decode()[:110])
except HTTPError as err:
    error = json.load(err)["error"]
    print(err.code, err.headers["X-Target-Status"], "credits", err.headers.get("X-Credits-Charged", 0), "|", error["code"])
    print(error["message"][:110])
Output
200 http/datacenter credits 1
![Wooden shelves with jars in a small shop](https://lab.megalania.io/wp-content/uploads/gibber/about-shop.webp
  • Standard library only: json.dumps for the body, Content-Type set by hand, because urllib does not do it for you.
  • The about page came back as Markdown, its first image as a Markdown image.

The same call on the uploads folder that answers 403 to everyone:

Terminal
MEG_API_KEY=YOUR_KEY python fetch.py https://lab.megalania.io/wp-content/uploads/
Output
422 403 credits 0 | blocked
The site blocked us and there is nothing to show. You are not charged. X-Untried lists the routes left to try;
  • Our 422 raised HTTPError too, so fetch.py reads our error JSON from the exception with json.load(err).
  • X-Target-Status: 403: the site said no on every route we tried. A block costs 0 credits.

One honest note: our test store blocks nobody, so these runs show the calls, not a bot filter being beaten. On a site that blocks, the call is the same; the price depends on the route that worked, and our MCP tool check_site tells you that price for free.

What it costs

What we ranRouteCredits
The about page as Markdownh:dc1
The uploads folder: 403 on every routeh:dc h:res c:dc c:res0
seen.py and fix.py, straight from urllibnone0

Every rule is in Credits. Plans are on the pricing page.

How we measured: urllib from Python 3.13.5, standard library only, one Linux server, 17 Nov 2026. nc -l 8799 (OpenBSD netcat) for what urllib sends. Our test store for the 403 and for fix.py. Our API on our dev cluster for fetch.py, against the same store on our dev domain. The outputs are copied from those runs.

Limits

  • Public pages only. We do not log in, and we do not solve CAPTCHAs. When a page needs either, we stop and say so, for free.
  • A page built by JavaScript needs engine=browser. It costs more. See How to scrape JavaScript-rendered pages.
  • Read the site's terms. A 403 is sometimes the site saying no. Fetch public pages only, and follow the site's terms and robots.txt.

Why sites send 403 at all, and the other layers they check, are in Why your scraper gets 403 Forbidden.