MegalaniaBlog
  1. Blog
  2. How to fix
  3. Playwright 403

Playwright 403 Forbidden: why headless Chrome gets it

Headless Playwright tells sites it is HeadlessChrome, twice. What it sends, how goto reports a 403, what user_agent fixes and what it leaves.

Jordan HayesDeveloper Relations
2 min read
How to fix
A padlock cut open. Inside, four pins named UA, headers, cookies and TLS. Three sit on the line, one sits low.

The short answer: Headless Playwright gets 403 Forbidden because it says what it is. Its Chrome sends HeadlessChrome/159 in the User-Agent and "HeadlessChrome" in the sec-ch-ua header, and a bot filter needs only one of them. page.goto() does not raise on a 403: it returns a response with status 403 and ok False. A context with user_agent= fixes the first header but not sec-ch-ua, which still says HeadlessChrome, and sec-ch-ua-platform still says Linux under a Windows user agent. When a site reads those, skip the local browser and post the page to our API. A block we detect costs nothing.

What Playwright sends

Start a listener in one terminal with nc -l 8799. It prints every byte a client sends and never answers, so the client times out. Run this in a second terminal:

sends.py
from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    try:
        page.goto("http://127.0.0.1:8799/", timeout=3000)
    except Exception as err:
        print(type(err).__name__, "|", str(err).splitlines()[0])
    browser.close()
Output
GET / HTTP/1.1
Host: 127.0.0.1:8799
Connection: keep-alive
sec-ch-ua: "HeadlessChrome";v="159", "Not/A)Brand";v="8", "Chromium";v="159"
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "Linux"
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/159 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/jxl,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7
Sec-Fetch-Site: none
Sec-Fetch-Mode: navigate
Sec-Fetch-User: ?1
Sec-Fetch-Dest: document
Accept-Encoding: gzip, deflate, br, zstd
  • HeadlessChrome/159 in the User-Agent and "HeadlessChrome";v="159" in sec-ch-ua. The rest is a real Chrome page load: Sec-Fetch-*, Upgrade-Insecure-Requests, a browser Accept.
  • No Accept-Language at all. A desktop Chrome always sends one.
  • We cut Chrome's build number to 159 in this output. The full number has four dotted parts.

How Playwright reports a 403

Our test store answers 403 on its uploads folder, because folder listing is off. That is a real "no", not a bot filter, and a handy way to see what your code gets back:

seen.py
from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    res = page.goto("https://lab.megalania.io/wp-content/uploads/")
    print("goto returned:", res.status, res.status_text, "| ok:", res.ok)
    print("page title:", page.title())
    browser.close()
Output
goto returned: 403  | ok: False
page title: 403 Forbidden
  • goto returned normally. A 403 is a page load that worked, as far as the browser is concerned. Check res.ok or res.status.
  • status_text is empty: over HTTP/2 there is no reason phrase, so there are two spaces after 403.

The free fix: browser headers

Send the headers a browser sends. This fixes sites that read only the headers:

fix.py
from playwright.sync_api import sync_playwright

UA = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/154.0 Safari/537.36"
with sync_playwright() as p:
    browser = p.chromium.launch()
    context = browser.new_context(user_agent=UA, locale="en-US")
    page = context.new_page()
    res = page.goto("https://lab.megalania.io/journal/")
    print(res.status, page.title(), "|", page.evaluate("navigator.userAgent")[:40])
    browser.close()
Output
200 Journal – Gibber & Co. | Mozilla/5.0 (Windows NT 10.0; Win64; x64
  • navigator.userAgent inside the page now says Windows Chrome, and locale="en-US" adds Accept-Language: en-US.
  • sec-ch-ua did not change. We sent fix.py to nc: it still said "HeadlessChrome";v="159", and sec-ch-ua-platform still said "Linux". A filter that compares them sees the mismatch.
  • Playwright's headed Chrome (headless=False) drops the HeadlessChrome name, but it needs a display on the server.
Free beta, by invite76,000 credits for 30 days, no card. Apply, and we email your invite.Apply for the beta

The fix that climbs: our API

Send the page to our API instead. We fetch it like a real browser, move to a residential IP or a real Chrome when the site blocks us, and a block we detect costs nothing:

fetch.py
# Run: MEG_API_KEY=YOUR_KEY python fetch.py https://lab.megalania.io/product/amber-candle-trio/
import os
import sys

import requests

r = requests.post(
    "https://get.megalania.io/v1",
    headers={"Authorization": f"Bearer {os.environ['MEG_API_KEY']}"},
    json={"url": sys.argv[1], "response_types": "markdown", "max_credits": "64"},
    timeout=100,
)
print(f"{r.status_code} target={r.headers.get('X-Target-Status')} route={r.headers.get('X-Route')} credits={r.headers.get('X-Credits-Charged', 0)}")
print(r.text[:140])
Output
200 target=200 route=http/datacenter credits=1
[![Amber Jar Candle Trio](https://lab.megalania.io/wp-content/uploads/gibber/amber-candle-trio-416x624.webp)](https://lab.megalania.io/wp-co
  • A product page did not need a browser: plain HTTP, 1 credit. Use your own Playwright only for the pages that need one.
  • X-Credits-Charged defaults to 0 in the script, so a block prints credits=0.

The same call on the uploads folder that answers 403 to everyone:

Terminal
MEG_API_KEY=YOUR_KEY python fetch.py https://lab.megalania.io/wp-content/uploads/
Output
422 target=403 route=browser/residential credits=0
{"error":{"code":"blocked","message":"The site blocked us and there is nothing to show. You are not charged. X-Untried lists the routes left
  • We ended on browser/residential: our own Chrome on a residential IP, after plain HTTP from both. All four got 403.
  • credits=0, and X-Untried in our answer names any route we did not try.

One honest note: our test store blocks nobody, so these runs show the calls, not a bot filter being beaten. On a site that blocks, the call is the same; the price depends on the route that worked, and our MCP tool check_site tells you that price for free.

What it costs

What we ranRouteCredits
A product page as Markdownh:dc1
The uploads folder: 403 on every routeh:dc h:res c:dc c:res0
sends.py, seen.py and fix.py, in your own Chromiumnone0

Every rule is in Credits. Plans are on the pricing page.

How we measured: Playwright 1.65.2 for Python with its Chrome headless shell 159, Python 3.13.5, one Linux server, 17 Nov 2026. nc -l 8799 (OpenBSD netcat) for what the browser sends, with and without user_agent; we cut Chrome's build number to 159. Our test store for the 403 and for fix.py. Our API on our dev cluster for fetch.py, against the same store on our dev domain. The outputs are copied from those runs.

Limits

  • Public pages only. We do not log in, and we do not solve CAPTCHAs. When a page needs either, we stop and say so, for free.
  • A page built by JavaScript needs engine=browser. It costs more. See How to scrape JavaScript-rendered pages.
  • Read the site's terms. A 403 is sometimes the site saying no. Fetch public pages only, and follow the site's terms and robots.txt.

Why sites send 403 at all, and the other layers they check, are in Why your scraper gets 403 Forbidden.