MegalaniaBlog
  1. Blog
  2. How to fix
  3. Puppeteer 403

Puppeteer 403 Forbidden: why a headless page gets refused

Puppeteer's Chrome puts HeadlessChrome in its User-Agent. What it sends, how goto reports a 403, the setUserAgent fix, and the call that climbs.

Jordan HayesDeveloper Relations
2 min read
How to fix
A padlock cut open. Inside, four pins named UA, headers, cookies and TLS. Three sit on the line, one sits low.

The short answer: Puppeteer's Chrome gets 403 Forbidden when a site spots that it runs headless, and the User-Agent says it outright: HeadlessChrome/157. Its sec-ch-ua brands are only Chromium and a filler, with no Google Chrome, which an installed Chrome sends. page.goto() resolves on a 403, so check res.ok(). page.setUserAgent() and setExtraHTTPHeaders() change what the page sends, and that is enough for sites that read headers. Sites that read the browser more closely still see a test build. Then post the page to our API, which tries other IPs and its own Chrome, and returns a free blocked when nothing works.

What Puppeteer sends

Start a listener in one terminal with nc -l 8799. It prints every byte a client sends and never answers, so the client times out. Run this in a second terminal:

sends.mjs
// npm install puppeteer
import puppeteer from "puppeteer";

const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.goto("http://127.0.0.1:8799/", { timeout: 3000 }).catch((e) => console.log(e.name, "|", e.message));
await browser.close();
Output
TimeoutError | Navigation timeout of 3000 ms exceeded
GET / HTTP/1.1
Host: 127.0.0.1:8799
Connection: keep-alive
sec-ch-ua: "Chromium";v="157", "Not-A.Brand";v="99"
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "Linux"
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/157 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/jxl,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7
Sec-Fetch-Site: none
Sec-Fetch-Mode: navigate
Sec-Fetch-User: ?1
Sec-Fetch-Dest: document
Accept-Encoding: gzip, deflate, br, zstd
Accept-Language: en-US,en;q=0.9
  • HeadlessChrome/157 in the User-Agent. That one word is enough for many filters.
  • sec-ch-ua lists Chromium and Not-A.Brand, not Google Chrome. Puppeteer downloads Chrome for Testing, and a desktop Chrome sends three brands.
  • Puppeteer did send Accept-Language: en-US,en;q=0.9, last. We cut Chrome's version to 157 in this output.

How Puppeteer reports a 403

Our test store answers 403 on its uploads folder, because folder listing is off. That is a real "no", not a bot filter, and a handy way to see what your code gets back:

seen.mjs
// npm install puppeteer
import puppeteer from "puppeteer";

const browser = await puppeteer.launch();
const page = await browser.newPage();
const res = await page.goto("https://lab.megalania.io/wp-content/uploads/");
console.log("goto resolved:", res.status(), "| ok:", res.ok(), "| title:", await page.title());
await browser.close();
Output
goto resolved: 403 | ok: false | title: 403 Forbidden
  • goto resolved with the 403 page loaded in the tab. Only network errors and timeouts reject it.
  • Your next $eval runs on the error page and throws when the element is missing, far from the real cause. Check res.ok() first.

The free fix: browser headers

Send the headers a browser sends. This fixes sites that read only the headers:

fix.mjs
// npm install puppeteer
import puppeteer from "puppeteer";

const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.setUserAgent({ userAgent: "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/154.0 Safari/537.36" });
await page.setExtraHTTPHeaders({ "Accept-Language": "en-US,en;q=0.9" });
const res = await page.goto("https://lab.megalania.io/product/chef-knife/");
console.log(res.status(), "|", await page.$eval("h1", (h) => h.textContent), "|", await page.$eval(".price", (p) => p.textContent.trim()));
await browser.close();
Output
200 | 20 cm Chef’s Knife | $112.00
  • setUserAgent takes an object in Puppeteer 25. It also changes navigator.userAgent, so scripts in the page see the same value.
  • setExtraHTTPHeaders adds headers to every request from the page, images and API calls included.
Free beta, by invite76,000 credits for 30 days, no card. Apply, and we email your invite.Apply for the beta

The fix that climbs: our API

Send the page to our API instead. We fetch it like a real browser, move to a residential IP or a real Chrome when the site blocks us, and a block we detect costs nothing:

fetch.mjs
// Run: MEG_API_KEY=YOUR_KEY node fetch.mjs https://lab.megalania.io/product/chef-knife/
const res = await fetch("https://get.megalania.io/v1", {
  method: "POST",
  headers: { Authorization: `Bearer ${process.env.MEG_API_KEY}`, "Content-Type": "application/json" },
  body: JSON.stringify({ url: process.argv[2], response_types: "json", extract: "name:h1|price:.summary .price", max_credits: "64" }),
});
console.log(res.status, res.headers.get("x-target-status"), res.headers.get("x-route"), "credits:", res.headers.get("x-credits-charged") ?? 0);
console.log((await res.text()).slice(0, 150));
Output
200 200 http/datacenter credits: 2
{"name":"20 cm Chef’s Knife","price":"$112.00"}
  • Two fields as JSON, from plain HTTP: 1 credit for the page, 1 for JSON. No Chrome needed for this page.
  • .summary .price picks the product's own price, not the prices of related products further down.

The same call on the uploads folder that answers 403 to everyone:

Terminal
MEG_API_KEY=YOUR_KEY node fetch.mjs https://lab.megalania.io/wp-content/uploads/
Output
422 403 browser/residential credits: 0
{"error":{"code":"blocked","message":"The site blocked us and there is nothing to show. You are not charged. X-Untried lists the routes left to try; n
  • 422 from us, 403 from the site, after our last route, browser/residential.
  • ?? 0 prints 0 when there is no charge header: a block is free.

One honest note: our test store blocks nobody, so these runs show the calls, not a bot filter being beaten. On a site that blocks, the call is the same; the price depends on the route that worked, and our MCP tool check_site tells you that price for free.

What it costs

What we ranRouteCredits
A product's name and price as JSONh:dc2
The uploads folder: 403 on every routeh:dc h:res c:dc c:res0
sends.mjs, seen.mjs and fix.mjs, in your own Chromenone0

Every rule is in Credits. Plans are on the pricing page.

How we measured: Puppeteer 25.17.0 with its Chrome 157 on Node 24.21, one Linux server, 17 Nov 2026. nc -l 8799 (OpenBSD netcat) for what the browser sends; we cut Chrome's version to 157. Our test store for the 403 and for fix.mjs. Our API on our dev cluster for fetch.mjs, against the same store on our dev domain. The outputs are copied from those runs.

Limits

  • Public pages only. We do not log in, and we do not solve CAPTCHAs. When a page needs either, we stop and say so, for free.
  • A page built by JavaScript needs engine=browser. It costs more. See How to scrape JavaScript-rendered pages.
  • Read the site's terms. A 403 is sometimes the site saying no. Fetch public pages only, and follow the site's terms and robots.txt.

Why sites send 403 at all, and the other layers they check, are in Why your scraper gets 403 Forbidden.