MegalaniaBlog
  1. Blog
  2. How to fix
  3. Selenium 403

Selenium 403 Forbidden: why headless Chrome gets refused

Selenium never shows you the 403: get() returns None. What headless Chrome sends, how to spot a refused page, the user-agent fix and what it leaves.

Jordan HayesDeveloper Relations
2 min read
How to fix
A padlock cut open. Inside, four pins named UA, headers, cookies and TLS. Three sit on the line, one sits low.

The short answer: Headless Chrome under Selenium gets 403 Forbidden when a site reads its User-Agent: HeadlessChrome/156 on X11; Linux. Selenium does not tell you about the 403: driver.get() returns None and the driver has no status code, so the only sign is the page itself, a title like 403 Forbidden. Check the title or a known element after every get(). Chrome's --user-agent switch replaces the name, but sec-ch-ua-platform still says Linux, which a filter can compare. When a site checks more than headers, send the page to our API, which tries other IPs and its own Chrome, and returns a free blocked when nothing works.

What Selenium sends

Start a listener in one terminal with nc -l 8799. It prints every byte a client sends and never answers, so the client times out. Run this in a second terminal:

sends.py
from selenium import webdriver

opts = webdriver.ChromeOptions()
opts.add_argument("--headless=new")
driver = webdriver.Chrome(options=opts)
driver.set_page_load_timeout(3)
try:
    driver.get("http://127.0.0.1:8799/")
except Exception as err:
    print(type(err).__name__)
driver.quit()
Output
TimeoutException
GET / HTTP/1.1
Host: 127.0.0.1:8799
Connection: keep-alive
sec-ch-ua: "Not:A-Brand";v="8", "Chromium";v="156"
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "Linux"
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/156 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/jxl,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7
Sec-Fetch-Site: none
Sec-Fetch-Mode: navigate
Sec-Fetch-User: ?1
Sec-Fetch-Dest: document
Accept-Encoding: gzip, deflate, br, zstd
Accept-Language: en-US,en;q=0.9
  • HeadlessChrome/156 in the User-Agent, on X11; Linux x86_64. Many filters stop at that word.
  • sec-ch-ua does not say headless in Chrome 156, but it names only Chromium and a filler brand. An installed Chrome adds Google Chrome.
  • Selenium sent Accept-Language on its own, unlike a bare HTTP library. Chrome's version is cut to 156 here.

How Selenium reports a 403

Our test store answers 403 on its uploads folder, because folder listing is off. That is a real "no", not a bot filter, and a handy way to see what your code gets back:

seen.py
from selenium import webdriver
from selenium.webdriver.common.by import By

opts = webdriver.ChromeOptions()
opts.add_argument("--headless=new")
driver = webdriver.Chrome(options=opts)
print("get() returned:", driver.get("https://lab.megalania.io/wp-content/uploads/"))
print("title:", driver.title, "| h1:", driver.find_element(By.TAG_NAME, "h1").text)
print("has status_code:", hasattr(driver, "status_code"))
driver.quit()
Output
get() returned: None
title: 403 Forbidden | h1: Forbidden
has status_code: False
  • get() returns None on every page, 200 or 403. WebDriver has no status code, so Selenium cannot give you one.
  • Check driver.title or wait for an element only the real page has. Without that check, your next find_element fails with NoSuchElementException, far from the cause.

The free fix: browser headers

Send the headers a browser sends. This fixes sites that read only the headers:

fix.py
from selenium import webdriver

UA = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/154.0 Safari/537.36"
opts = webdriver.ChromeOptions()
opts.add_argument("--headless=new")
opts.add_argument(f"--user-agent={UA}")
driver = webdriver.Chrome(options=opts)
driver.get("https://lab.megalania.io/faq/")
print(driver.title, "|", driver.execute_script("return navigator.userAgent")[:40])
driver.quit()
Output
FAQ – Gibber & Co. | Mozilla/5.0 (Windows NT 10.0; Win64; x64
  • --user-agent changes the header and navigator.userAgent, so scripts in the page see the same value.
  • sec-ch-ua-platform did not change. We sent fix.py to nc: it still said "Linux" under a Windows user agent, a mismatch a filter can spot.
Free beta, by invite76,000 credits for 30 days, no card. Apply, and we email your invite.Apply for the beta

The fix that climbs: our API

Send the page to our API instead. We fetch it like a real browser, move to a residential IP or a real Chrome when the site blocks us, and a block we detect costs nothing:

fetch.py
# Run: MEG_API_KEY=YOUR_KEY python fetch.py https://lab.megalania.io/faq/
import json
import os
import sys

import requests

r = requests.post(
    "https://get.megalania.io/v1",
    headers={"Authorization": f"Bearer {os.environ['MEG_API_KEY']}"},
    json={"url": sys.argv[1], "response_types": "markdown"},
    timeout=100,
)
hops = json.loads(r.headers.get("X-Hops", "[]"))
print(r.status_code, "| target", r.headers.get("X-Target-Status"), "| credits", r.headers.get("X-Credits-Charged", 0))
print(" -> ".join(f"{h['rung']} {h['outcome']}" for h in hops))
print(r.text[:70])
Output
200 | target 200 | credits 1
h:dc ok
# FAQ

Is this a real shop?

No. Gibber & Co. is a demo store by Megal
  • Our first try, plain HTTP from a data center, got the FAQ for 1 credit. Keep Selenium for pages where you click and scroll.
  • X-Hops lists every try with its route and outcome: here one, h:dc ok.

The same call on the uploads folder that answers 403 to everyone:

Terminal
MEG_API_KEY=YOUR_KEY python fetch.py https://lab.megalania.io/wp-content/uploads/
Output
422 | target 403 | credits 0
h:dc blocked -> h:res blocked -> c:dc blocked -> c:res blocked
{"error":{"code":"blocked","message":"The site blocked us and there is
  • Four routes, four 403s: plain HTTP from a data center and a residential IP, then our Chrome on each. One 422, blocked.
  • Unlike driver.get(), our answer says the site refused, in the status and in X-Target-Status.

One honest note: our test store blocks nobody, so these runs show the calls, not a bot filter being beaten. On a site that blocks, the call is the same; the price depends on the route that worked, and our MCP tool check_site tells you that price for free.

What it costs

What we ranRouteCredits
The FAQ page as Markdownh:dc1
The uploads folder: 403 on every routeh:dc h:res c:dc c:res0
sends.py, seen.py and fix.py, in your own Chromenone0

Every rule is in Credits. Plans are on the pricing page.

How we measured: Selenium 4.52.0 for Python with Chrome 156 headless, Python 3.13.5, one Linux server, 17 Nov 2026. nc -l 8799 (OpenBSD netcat) for what the browser sends, with and without --user-agent; we cut Chrome's version to 156. Our test store for the 403 and for fix.py. Our API on our dev cluster for fetch.py, against the same store on our dev domain. The outputs are copied from those runs.

Limits

  • Public pages only. We do not log in, and we do not solve CAPTCHAs. When a page needs either, we stop and say so, for free.
  • A page built by JavaScript needs engine=browser. It costs more. See How to scrape JavaScript-rendered pages.
  • Read the site's terms. A 403 is sometimes the site saying no. Fetch public pages only, and follow the site's terms and robots.txt.

Why sites send 403 at all, and the other layers they check, are in Why your scraper gets 403 Forbidden.