The short answer: Headless Chrome under Selenium gets 403 Forbidden when a site reads its User-Agent: HeadlessChrome/156 on X11; Linux. Selenium does not tell you about the 403: driver.get() returns None and the driver has no status code, so the only sign is the page itself, a title like 403 Forbidden. Check the title or a known element after every get(). Chrome's --user-agent switch replaces the name, but sec-ch-ua-platform still says Linux, which a filter can compare. When a site checks more than headers, send the page to our API, which tries other IPs and its own Chrome, and returns a free blocked when nothing works.
What Selenium sends
Start a listener in one terminal with nc -l 8799. It prints every byte a client sends and never answers, so the client times out. Run this in a second terminal:
from selenium import webdriver
opts = webdriver.ChromeOptions()
opts.add_argument("--headless=new")
driver = webdriver.Chrome(options=opts)
driver.set_page_load_timeout(3)
try:
driver.get("http://127.0.0.1:8799/")
except Exception as err:
print(type(err).__name__)
driver.quit()TimeoutException GET / HTTP/1.1 Host: 127.0.0.1:8799 Connection: keep-alive sec-ch-ua: "Not:A-Brand";v="8", "Chromium";v="156" sec-ch-ua-mobile: ?0 sec-ch-ua-platform: "Linux" Upgrade-Insecure-Requests: 1 User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/156 Safari/537.36 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/jxl,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7 Sec-Fetch-Site: none Sec-Fetch-Mode: navigate Sec-Fetch-User: ?1 Sec-Fetch-Dest: document Accept-Encoding: gzip, deflate, br, zstd Accept-Language: en-US,en;q=0.9
HeadlessChrome/156in theUser-Agent, onX11; Linux x86_64. Many filters stop at that word.sec-ch-uadoes not say headless in Chrome 156, but it names onlyChromiumand a filler brand. An installed Chrome addsGoogle Chrome.- Selenium sent
Accept-Languageon its own, unlike a bare HTTP library. Chrome's version is cut to156here.
How Selenium reports a 403
Our test store answers 403 on its uploads folder, because folder listing is off. That is a real "no", not a bot filter, and a handy way to see what your code gets back:
from selenium import webdriver
from selenium.webdriver.common.by import By
opts = webdriver.ChromeOptions()
opts.add_argument("--headless=new")
driver = webdriver.Chrome(options=opts)
print("get() returned:", driver.get("https://lab.megalania.io/wp-content/uploads/"))
print("title:", driver.title, "| h1:", driver.find_element(By.TAG_NAME, "h1").text)
print("has status_code:", hasattr(driver, "status_code"))
driver.quit()get() returned: None title: 403 Forbidden | h1: Forbidden has status_code: False
get()returnsNoneon every page, 200 or 403. WebDriver has no status code, so Selenium cannot give you one.- Check
driver.titleor wait for an element only the real page has. Without that check, your nextfind_elementfails withNoSuchElementException, far from the cause.
The free fix: browser headers
Send the headers a browser sends. This fixes sites that read only the headers:
from selenium import webdriver
UA = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/154.0 Safari/537.36"
opts = webdriver.ChromeOptions()
opts.add_argument("--headless=new")
opts.add_argument(f"--user-agent={UA}")
driver = webdriver.Chrome(options=opts)
driver.get("https://lab.megalania.io/faq/")
print(driver.title, "|", driver.execute_script("return navigator.userAgent")[:40])
driver.quit()FAQ – Gibber & Co. | Mozilla/5.0 (Windows NT 10.0; Win64; x64
--user-agentchanges the header andnavigator.userAgent, so scripts in the page see the same value.sec-ch-ua-platformdid not change. We sentfix.pytonc: it still said"Linux"under a Windows user agent, a mismatch a filter can spot.
The fix that climbs: our API
Send the page to our API instead. We fetch it like a real browser, move to a residential IP or a real Chrome when the site blocks us, and a block we detect costs nothing:
# Run: MEG_API_KEY=YOUR_KEY python fetch.py https://lab.megalania.io/faq/
import json
import os
import sys
import requests
r = requests.post(
"https://get.megalania.io/v1",
headers={"Authorization": f"Bearer {os.environ['MEG_API_KEY']}"},
json={"url": sys.argv[1], "response_types": "markdown"},
timeout=100,
)
hops = json.loads(r.headers.get("X-Hops", "[]"))
print(r.status_code, "| target", r.headers.get("X-Target-Status"), "| credits", r.headers.get("X-Credits-Charged", 0))
print(" -> ".join(f"{h['rung']} {h['outcome']}" for h in hops))
print(r.text[:70])200 | target 200 | credits 1 h:dc ok # FAQ Is this a real shop? No. Gibber & Co. is a demo store by Megal
- Our first try, plain HTTP from a data center, got the FAQ for 1 credit. Keep Selenium for pages where you click and scroll.
X-Hopslists every try with its route and outcome: here one,h:dc ok.
The same call on the uploads folder that answers 403 to everyone:
MEG_API_KEY=YOUR_KEY python fetch.py https://lab.megalania.io/wp-content/uploads/
422 | target 403 | credits 0
h:dc blocked -> h:res blocked -> c:dc blocked -> c:res blocked
{"error":{"code":"blocked","message":"The site blocked us and there is- Four routes, four 403s: plain HTTP from a data center and a residential IP, then our Chrome on each. One
422,blocked. - Unlike
driver.get(), our answer says the site refused, in the status and inX-Target-Status.
One honest note: our test store blocks nobody, so these runs show the calls, not a bot filter being beaten. On a site that blocks, the call is the same; the price depends on the route that worked, and our MCP tool check_site tells you that price for free.
What it costs
| What we ran | Route | Credits |
|---|---|---|
| The FAQ page as Markdown | h:dc | 1 |
| The uploads folder: 403 on every route | h:dc h:res c:dc c:res | 0 |
sends.py, seen.py and fix.py, in your own Chrome | none | 0 |
Every rule is in Credits. Plans are on the pricing page.
How we measured: Selenium 4.52.0 for Python with Chrome 156 headless, Python 3.13.5, one Linux server, 17 Nov 2026. nc -l 8799 (OpenBSD netcat) for what the browser sends, with and without --user-agent; we cut Chrome's version to 156. Our test store for the 403 and for fix.py. Our API on our dev cluster for fetch.py, against the same store on our dev domain. The outputs are copied from those runs.
Limits
- Public pages only. We do not log in, and we do not solve CAPTCHAs. When a page needs either, we stop and say so, for free.
- A page built by JavaScript needs
engine=browser. It costs more. See How to scrape JavaScript-rendered pages. - Read the site's terms. A 403 is sometimes the site saying no. Fetch public pages only, and follow the site's terms and robots.txt.
Why sites send 403 at all, and the other layers they check, are in Why your scraper gets 403 Forbidden.