The short answer: aiohttp gets 403 Forbidden when a bot filter reads its default request: User-Agent: Python/3.13 aiohttp/3.14.5 and Accept: */*, with no language. A 403 is a normal response in aiohttp: res.status is 403 and nothing is raised until you call res.raise_for_status(), or create the session with raise_for_status=True. Pass browser headers to the ClientSession and every request carries them. aiohttp speaks HTTP/1.1 only, with Python's TLS, so sites that check the protocol or the TLS hello still see a script. Send those pages to our API from the same session: a block we detect costs nothing.
What aiohttp sends
Start a listener in one terminal with nc -l 8799. It prints every byte a client sends and never answers, so the client times out. Run this in a second terminal:
python -c '
import asyncio, aiohttp
async def main():
async with aiohttp.ClientSession(timeout=aiohttp.ClientTimeout(total=3)) as s:
await s.get("http://127.0.0.1:8799/")
asyncio.run(main())'GET / HTTP/1.1 Host: 127.0.0.1:8799 Accept: */* Accept-Encoding: gzip, deflate, br, zstd User-Agent: Python/3.13 aiohttp/3.14.5
User-Agent: Python/3.13 aiohttp/3.14.5names the language, the library and both versions.- **
Accept: */*and noAccept-Language.** A browser asks fortext/htmlfirst and names a language. - aiohttp asks for
zstdas well as gzip and brotli, which no browser sends in this order.
How aiohttp reports a 403
Our test store answers 403 on its uploads folder, because folder listing is off. That is a real "no", not a bot filter, and a handy way to see what your code gets back:
import asyncio
import aiohttp
async def main():
async with aiohttp.ClientSession() as session:
async with session.get("https://lab.megalania.io/wp-content/uploads/") as res:
print(res.status, res.reason, "| ok:", res.ok, "|", len(await res.text()), "characters")
try:
res.raise_for_status()
except aiohttp.ClientResponseError as err:
print(type(err).__name__, "|", err)
asyncio.run(main())403 Forbidden | ok: False | 239 characters ClientResponseError | 403, message='Forbidden', url='https://lab.megalania.io/wp-content/uploads/'
- The 403 came back as a normal response.
res.okisFalse, and the error page is still readable withawait res.text(). raise_for_status()raisesClientResponseError, with the status, the reason and the URL in its message.
The free fix: browser headers
Send the headers a browser sends. This fixes sites that read only the headers:
import asyncio
import aiohttp
HEADERS = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/154.0 Safari/537.36",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"Accept-Language": "en-US,en;q=0.9",
}
PAGES = ["https://lab.megalania.io/faq/", "https://lab.megalania.io/journal/"]
async def main():
async with aiohttp.ClientSession(headers=HEADERS, raise_for_status=True) as session:
for url in PAGES:
async with session.get(url) as res:
print(res.status, f"HTTP/{res.version.major}.{res.version.minor}", url.rsplit("/", 2)[1], "|", res.request_info.headers["User-Agent"][:40])
asyncio.run(main())200 HTTP/1.1 faq | Mozilla/5.0 (Windows NT 10.0; Win64; x64 200 HTTP/1.1 journal | Mozilla/5.0 (Windows NT 10.0; Win64; x64
- Headers on the
ClientSessiongo with every request, andres.request_info.headersshows what was really sent. raise_for_status=Trueon the session turns every 4xx and 5xx intoClientResponseError, so a 403 stops the task instead of passing quietly.HTTP/1.1on both pages. aiohttp has no HTTP/2, though the store offers it.
The fix that climbs: our API
Send the page to our API instead. We fetch it like a real browser, move to a residential IP or a real Chrome when the site blocks us, and a block we detect costs nothing:
# Run: MEG_API_KEY=YOUR_KEY python fetch.py https://lab.megalania.io/product/bearing-compass/
import asyncio
import os
import sys
import aiohttp
async def main():
body = {"url": sys.argv[1], "response_types": "json", "extract": "name:h1|was:.summary del .amount|now:.summary ins .amount"}
headers = {"Authorization": f"Bearer {os.environ['MEG_API_KEY']}"}
async with aiohttp.ClientSession(headers=headers, timeout=aiohttp.ClientTimeout(total=100)) as session:
async with session.post("https://get.megalania.io/v1", json=body) as res:
print(res.status, res.headers.get("X-Target-Status"), res.headers.get("X-Route"), "credits", res.headers.get("X-Credits-Charged", "0"))
print((await res.text())[:120])
asyncio.run(main())200 200 http/datacenter credits 2
{"name":"Bearing Baseplate Compass","now":"$26.95","was":"$34.00"}- Three fields as JSON from plain HTTP: 1 credit for the page, 1 for JSON.
- The compass is on sale, so its price block holds two prices and screen-reader text.
delandinspick the old and the new price apart.
The same call on the uploads folder that answers 403 to everyone:
MEG_API_KEY=YOUR_KEY python fetch.py https://lab.megalania.io/wp-content/uploads/
422 403 browser/residential credits 0
{"error":{"code":"blocked","message":"The site blocked us and there is nothing to show. You are not charged. X-Untried l422from us,403from the site, after our last route, our own Chrome on a residential IP.- No charge header came back, so the script printed its default, 0. A block is free.
One honest note: our test store blocks nobody, so these runs show the calls, not a bot filter being beaten. On a site that blocks, the call is the same; the price depends on the route that worked, and our MCP tool check_site tells you that price for free.
What it costs
| What we ran | Route | Credits |
|---|---|---|
| A product's name and two prices as JSON | h:dc | 2 |
| The uploads folder: 403 on every route | h:dc h:res c:dc c:res | 0 |
seen.py and fix.py, straight from aiohttp | none | 0 |
Every rule is in Credits. Plans are on the pricing page.
How we measured: aiohttp 3.14.5 on Python 3.13.5, one Linux server, 17 Nov 2026. nc -l 8799 (OpenBSD netcat) for what aiohttp sends. Our test store for the 403 and for fix.py. Our API on our dev cluster for fetch.py, against the same store on our dev domain. The outputs are copied from those runs.
Limits
- Public pages only. We do not log in, and we do not solve CAPTCHAs. When a page needs either, we stop and say so, for free.
- A page built by JavaScript needs
engine=browser. It costs more. See How to scrape JavaScript-rendered pages. - Read the site's terms. A 403 is sometimes the site saying no. Fetch public pages only, and follow the site's terms and robots.txt.
Why sites send 403 at all, and the other layers they check, are in Why your scraper gets 403 Forbidden.