The short answer: Headless Playwright gets 403 Forbidden because it says what it is. Its Chrome sends HeadlessChrome/159 in the User-Agent and "HeadlessChrome" in the sec-ch-ua header, and a bot filter needs only one of them. page.goto() does not raise on a 403: it returns a response with status 403 and ok False. A context with user_agent= fixes the first header but not sec-ch-ua, which still says HeadlessChrome, and sec-ch-ua-platform still says Linux under a Windows user agent. When a site reads those, skip the local browser and post the page to our API. A block we detect costs nothing.
What Playwright sends
Start a listener in one terminal with nc -l 8799. It prints every byte a client sends and never answers, so the client times out. Run this in a second terminal:
from playwright.sync_api import sync_playwright
with sync_playwright() as p:
browser = p.chromium.launch()
page = browser.new_page()
try:
page.goto("http://127.0.0.1:8799/", timeout=3000)
except Exception as err:
print(type(err).__name__, "|", str(err).splitlines()[0])
browser.close()GET / HTTP/1.1 Host: 127.0.0.1:8799 Connection: keep-alive sec-ch-ua: "HeadlessChrome";v="159", "Not/A)Brand";v="8", "Chromium";v="159" sec-ch-ua-mobile: ?0 sec-ch-ua-platform: "Linux" Upgrade-Insecure-Requests: 1 User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/159 Safari/537.36 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/jxl,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7 Sec-Fetch-Site: none Sec-Fetch-Mode: navigate Sec-Fetch-User: ?1 Sec-Fetch-Dest: document Accept-Encoding: gzip, deflate, br, zstd
HeadlessChrome/159in theUser-Agentand"HeadlessChrome";v="159"insec-ch-ua. The rest is a real Chrome page load:Sec-Fetch-*,Upgrade-Insecure-Requests, a browserAccept.- No
Accept-Languageat all. A desktop Chrome always sends one. - We cut Chrome's build number to
159in this output. The full number has four dotted parts.
How Playwright reports a 403
Our test store answers 403 on its uploads folder, because folder listing is off. That is a real "no", not a bot filter, and a handy way to see what your code gets back:
from playwright.sync_api import sync_playwright
with sync_playwright() as p:
browser = p.chromium.launch()
page = browser.new_page()
res = page.goto("https://lab.megalania.io/wp-content/uploads/")
print("goto returned:", res.status, res.status_text, "| ok:", res.ok)
print("page title:", page.title())
browser.close()goto returned: 403 | ok: False page title: 403 Forbidden
gotoreturned normally. A 403 is a page load that worked, as far as the browser is concerned. Checkres.okorres.status.status_textis empty: over HTTP/2 there is no reason phrase, so there are two spaces after403.
The free fix: browser headers
Send the headers a browser sends. This fixes sites that read only the headers:
from playwright.sync_api import sync_playwright
UA = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/154.0 Safari/537.36"
with sync_playwright() as p:
browser = p.chromium.launch()
context = browser.new_context(user_agent=UA, locale="en-US")
page = context.new_page()
res = page.goto("https://lab.megalania.io/journal/")
print(res.status, page.title(), "|", page.evaluate("navigator.userAgent")[:40])
browser.close()200 Journal – Gibber & Co. | Mozilla/5.0 (Windows NT 10.0; Win64; x64
navigator.userAgentinside the page now says Windows Chrome, andlocale="en-US"addsAccept-Language: en-US.sec-ch-uadid not change. We sentfix.pytonc: it still said"HeadlessChrome";v="159", andsec-ch-ua-platformstill said"Linux". A filter that compares them sees the mismatch.- Playwright's headed Chrome (
headless=False) drops the HeadlessChrome name, but it needs a display on the server.
The fix that climbs: our API
Send the page to our API instead. We fetch it like a real browser, move to a residential IP or a real Chrome when the site blocks us, and a block we detect costs nothing:
# Run: MEG_API_KEY=YOUR_KEY python fetch.py https://lab.megalania.io/product/amber-candle-trio/
import os
import sys
import requests
r = requests.post(
"https://get.megalania.io/v1",
headers={"Authorization": f"Bearer {os.environ['MEG_API_KEY']}"},
json={"url": sys.argv[1], "response_types": "markdown", "max_credits": "64"},
timeout=100,
)
print(f"{r.status_code} target={r.headers.get('X-Target-Status')} route={r.headers.get('X-Route')} credits={r.headers.get('X-Credits-Charged', 0)}")
print(r.text[:140])200 target=200 route=http/datacenter credits=1 [](https://lab.megalania.io/wp-co
- A product page did not need a browser: plain HTTP, 1 credit. Use your own Playwright only for the pages that need one.
X-Credits-Chargeddefaults to 0 in the script, so a block printscredits=0.
The same call on the uploads folder that answers 403 to everyone:
MEG_API_KEY=YOUR_KEY python fetch.py https://lab.megalania.io/wp-content/uploads/
422 target=403 route=browser/residential credits=0
{"error":{"code":"blocked","message":"The site blocked us and there is nothing to show. You are not charged. X-Untried lists the routes left- We ended on
browser/residential: our own Chrome on a residential IP, after plain HTTP from both. All four got 403. credits=0, andX-Untriedin our answer names any route we did not try.
One honest note: our test store blocks nobody, so these runs show the calls, not a bot filter being beaten. On a site that blocks, the call is the same; the price depends on the route that worked, and our MCP tool check_site tells you that price for free.
What it costs
| What we ran | Route | Credits |
|---|---|---|
| A product page as Markdown | h:dc | 1 |
| The uploads folder: 403 on every route | h:dc h:res c:dc c:res | 0 |
sends.py, seen.py and fix.py, in your own Chromium | none | 0 |
Every rule is in Credits. Plans are on the pricing page.
How we measured: Playwright 1.65.2 for Python with its Chrome headless shell 159, Python 3.13.5, one Linux server, 17 Nov 2026. nc -l 8799 (OpenBSD netcat) for what the browser sends, with and without user_agent; we cut Chrome's build number to 159. Our test store for the 403 and for fix.py. Our API on our dev cluster for fetch.py, against the same store on our dev domain. The outputs are copied from those runs.
Limits
- Public pages only. We do not log in, and we do not solve CAPTCHAs. When a page needs either, we stop and say so, for free.
- A page built by JavaScript needs
engine=browser. It costs more. See How to scrape JavaScript-rendered pages. - Read the site's terms. A 403 is sometimes the site saying no. Fetch public pages only, and follow the site's terms and robots.txt.
Why sites send 403 at all, and the other layers they check, are in Why your scraper gets 403 Forbidden.