MegalaniaBlog
  1. Blog
  2. How to fix
  3. aiohttp 403

aiohttp 403 Forbidden: why it happens, and the fix

aiohttp sends Python/3.13 aiohttp/3.14.5 as its User-Agent. What it sends, how a 403 comes back, the session headers fix, and the call that climbs.

Jordan HayesDeveloper Relations
2 min read
How to fix
A padlock cut open. Inside, four pins named UA, headers, cookies and TLS. Three sit on the line, one sits low.

The short answer: aiohttp gets 403 Forbidden when a bot filter reads its default request: User-Agent: Python/3.13 aiohttp/3.14.5 and Accept: */*, with no language. A 403 is a normal response in aiohttp: res.status is 403 and nothing is raised until you call res.raise_for_status(), or create the session with raise_for_status=True. Pass browser headers to the ClientSession and every request carries them. aiohttp speaks HTTP/1.1 only, with Python's TLS, so sites that check the protocol or the TLS hello still see a script. Send those pages to our API from the same session: a block we detect costs nothing.

What aiohttp sends

Start a listener in one terminal with nc -l 8799. It prints every byte a client sends and never answers, so the client times out. Run this in a second terminal:

Terminal 2
python -c '
import asyncio, aiohttp
async def main():
    async with aiohttp.ClientSession(timeout=aiohttp.ClientTimeout(total=3)) as s:
        await s.get("http://127.0.0.1:8799/")
asyncio.run(main())'
Output
GET / HTTP/1.1
Host: 127.0.0.1:8799
Accept: */*
Accept-Encoding: gzip, deflate, br, zstd
User-Agent: Python/3.13 aiohttp/3.14.5
  • User-Agent: Python/3.13 aiohttp/3.14.5 names the language, the library and both versions.
  • **Accept: */* and no Accept-Language.** A browser asks for text/html first and names a language.
  • aiohttp asks for zstd as well as gzip and brotli, which no browser sends in this order.

How aiohttp reports a 403

Our test store answers 403 on its uploads folder, because folder listing is off. That is a real "no", not a bot filter, and a handy way to see what your code gets back:

seen.py
import asyncio

import aiohttp


async def main():
    async with aiohttp.ClientSession() as session:
        async with session.get("https://lab.megalania.io/wp-content/uploads/") as res:
            print(res.status, res.reason, "| ok:", res.ok, "|", len(await res.text()), "characters")
            try:
                res.raise_for_status()
            except aiohttp.ClientResponseError as err:
                print(type(err).__name__, "|", err)


asyncio.run(main())
Output
403 Forbidden | ok: False | 239 characters
ClientResponseError | 403, message='Forbidden', url='https://lab.megalania.io/wp-content/uploads/'
  • The 403 came back as a normal response. res.ok is False, and the error page is still readable with await res.text().
  • raise_for_status() raises ClientResponseError, with the status, the reason and the URL in its message.

The free fix: browser headers

Send the headers a browser sends. This fixes sites that read only the headers:

fix.py
import asyncio

import aiohttp

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/154.0 Safari/537.36",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.9",
}
PAGES = ["https://lab.megalania.io/faq/", "https://lab.megalania.io/journal/"]


async def main():
    async with aiohttp.ClientSession(headers=HEADERS, raise_for_status=True) as session:
        for url in PAGES:
            async with session.get(url) as res:
                print(res.status, f"HTTP/{res.version.major}.{res.version.minor}", url.rsplit("/", 2)[1], "|", res.request_info.headers["User-Agent"][:40])


asyncio.run(main())
Output
200 HTTP/1.1 faq | Mozilla/5.0 (Windows NT 10.0; Win64; x64
200 HTTP/1.1 journal | Mozilla/5.0 (Windows NT 10.0; Win64; x64
  • Headers on the ClientSession go with every request, and res.request_info.headers shows what was really sent.
  • raise_for_status=True on the session turns every 4xx and 5xx into ClientResponseError, so a 403 stops the task instead of passing quietly.
  • HTTP/1.1 on both pages. aiohttp has no HTTP/2, though the store offers it.
Free beta, by invite76,000 credits for 30 days, no card. Apply, and we email your invite.Apply for the beta

The fix that climbs: our API

Send the page to our API instead. We fetch it like a real browser, move to a residential IP or a real Chrome when the site blocks us, and a block we detect costs nothing:

fetch.py
# Run: MEG_API_KEY=YOUR_KEY python fetch.py https://lab.megalania.io/product/bearing-compass/
import asyncio
import os
import sys

import aiohttp


async def main():
    body = {"url": sys.argv[1], "response_types": "json", "extract": "name:h1|was:.summary del .amount|now:.summary ins .amount"}
    headers = {"Authorization": f"Bearer {os.environ['MEG_API_KEY']}"}
    async with aiohttp.ClientSession(headers=headers, timeout=aiohttp.ClientTimeout(total=100)) as session:
        async with session.post("https://get.megalania.io/v1", json=body) as res:
            print(res.status, res.headers.get("X-Target-Status"), res.headers.get("X-Route"), "credits", res.headers.get("X-Credits-Charged", "0"))
            print((await res.text())[:120])


asyncio.run(main())
Output
200 200 http/datacenter credits 2
{"name":"Bearing Baseplate Compass","now":"$26.95","was":"$34.00"}
  • Three fields as JSON from plain HTTP: 1 credit for the page, 1 for JSON.
  • The compass is on sale, so its price block holds two prices and screen-reader text. del and ins pick the old and the new price apart.

The same call on the uploads folder that answers 403 to everyone:

Terminal
MEG_API_KEY=YOUR_KEY python fetch.py https://lab.megalania.io/wp-content/uploads/
Output
422 403 browser/residential credits 0
{"error":{"code":"blocked","message":"The site blocked us and there is nothing to show. You are not charged. X-Untried l
  • 422 from us, 403 from the site, after our last route, our own Chrome on a residential IP.
  • No charge header came back, so the script printed its default, 0. A block is free.

One honest note: our test store blocks nobody, so these runs show the calls, not a bot filter being beaten. On a site that blocks, the call is the same; the price depends on the route that worked, and our MCP tool check_site tells you that price for free.

What it costs

What we ranRouteCredits
A product's name and two prices as JSONh:dc2
The uploads folder: 403 on every routeh:dc h:res c:dc c:res0
seen.py and fix.py, straight from aiohttpnone0

Every rule is in Credits. Plans are on the pricing page.

How we measured: aiohttp 3.14.5 on Python 3.13.5, one Linux server, 17 Nov 2026. nc -l 8799 (OpenBSD netcat) for what aiohttp sends. Our test store for the 403 and for fix.py. Our API on our dev cluster for fetch.py, against the same store on our dev domain. The outputs are copied from those runs.

Limits

  • Public pages only. We do not log in, and we do not solve CAPTCHAs. When a page needs either, we stop and say so, for free.
  • A page built by JavaScript needs engine=browser. It costs more. See How to scrape JavaScript-rendered pages.
  • Read the site's terms. A 403 is sometimes the site saying no. Fetch public pages only, and follow the site's terms and robots.txt.

Why sites send 403 at all, and the other layers they check, are in Why your scraper gets 403 Forbidden.