The short answer: A 403 Forbidden from node-fetch usually means the site looked at the request and saw a script. node-fetch makes that easy: it writes user-agent: node-fetch, and its header names come out in lower case, sorted a to z, with Host added last. Node's built-in fetch gives itself away too, with user-agent: node, accept-language: * and sec-fetch-mode: cors. Neither one rejects the promise on a 403, so check res.ok before you read the body. Real browser header values help on sites that only look at headers; the lower-case order and Node's HTTP/1.1 connection remain. For sites that look deeper, post the page to our API, which retries from other IPs and in a real Chrome. If every route is blocked, you pay nothing.
What node-fetch sends
Start a listener in one terminal with nc -l 8799. It prints every byte a client sends and never answers, so the client times out. Run this in a second terminal:
npm install node-fetch
node -e 'import("node-fetch").then(({ default: f }) => f("http://127.0.0.1:8799/", { signal: AbortSignal.timeout(2000) })).catch((e) => console.log(e.name))'
node -e 'fetch("http://127.0.0.1:8799/", { signal: AbortSignal.timeout(2000) }).catch((e) => console.log(e.name))'GET / HTTP/1.1 accept: */* accept-encoding: gzip, deflate, br user-agent: node-fetch Host: 127.0.0.1:8799 Connection: keep-alive GET / HTTP/1.1 host: 127.0.0.1:8799 connection: keep-alive accept: */* accept-language: * sec-fetch-mode: cors user-agent: node accept-encoding: gzip, deflate
- node-fetch 3.3.2 (first request) writes
user-agent: node-fetch, no version. Its own headers are in lower case and in a to z order, then Node addsHostandConnectionlast. A browser on HTTP/1.1 sendsHostfirst, with capitals. - Node's built-in
fetch(second request) comes from undici:user-agent: node,accept-language: *andsec-fetch-mode: cors. No browser sends a star for its language, andcorsis what a page's script sends, not a page load. - **
accept: */*** in both. A browser loading a page asks fortext/htmlfirst.
How node-fetch reports a 403
Our test store answers 403 on its uploads folder, because folder listing is off. That is a real "no", not a bot filter, and a handy way to see what your code gets back:
// npm install node-fetch
import fetch from "node-fetch";
const res = await fetch("https://lab.megalania.io/wp-content/uploads/");
console.log("ok:", res.ok, "status:", res.status, res.statusText);
console.log((await res.text()).length, "characters of HTML in the body");ok: false status: 403 Forbidden 239 characters of HTML in the body
- node-fetch does not throw on a 403. The promise resolves,
res.okisfalse, and the site's error page is the body. Code that never checksres.oksaves that page as data. - The built-in
fetchbehaves the same way: only a network failure or an abort rejects the promise.
The free fix: browser headers
Send the headers a browser sends. This fixes sites that read only the headers:
// npm install node-fetch
import fetch from "node-fetch";
const res = await fetch("https://lab.megalania.io/faq/", {
headers: {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/154.0 Safari/537.36",
Accept: "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"Accept-Language": "en-US,en;q=0.9",
},
});
console.log(res.status, res.headers.get("content-type"));200 text/html; charset=UTF-8
- We sent
fix.mjstonc: node-fetch wroteaccept,accept-encoding: gzip, deflate, br,accept-language,user-agent, thenHostandConnection. Your values are in, but the names are still lower case and sorted, so the order still gives it away. - node-fetch uses Node's
httpmodule: HTTP/1.1 only, and Node's own TLS hello. No header changes either.
The fix that climbs: our API
Send the page to our API instead. We fetch it like a real browser, move to a residential IP or a real Chrome when the site blocks us, and a block we detect costs nothing:
// npm install node-fetch
// Run: MEG_API_KEY=YOUR_KEY node fetch.mjs https://lab.megalania.io/faq/
import fetch from "node-fetch";
const res = await fetch("https://get.megalania.io/v1", {
method: "POST",
headers: { Authorization: `Bearer ${process.env.MEG_API_KEY}`, "Content-Type": "application/json" },
body: JSON.stringify({ url: process.argv[2], response_types: "markdown", max_credits: "64" }),
signal: AbortSignal.timeout(90000),
});
console.log("status:", res.status, "target:", res.headers.get("x-target-status"));
console.log("route:", res.headers.get("x-route"), "credits:", res.headers.get("x-credits-charged"), "billing:", res.headers.get("x-billing"));
console.log("tries:", res.headers.get("x-hops"));
console.log((await res.text()).slice(0, 160));status: 200 target: 200
route: http/datacenter credits: 1 billing: null
tries: [{"rung":"h:dc","identity":"chrome","ms":1793,"outcome":"ok"}]
# FAQ
Is this a real shop?
No. Gibber & Co. is a demo store by Megalania for testing web scraping. Nothing here is for sale. Checkout is switched off and no pres.headers.get()returnsnullfor a header we did not send, sobilling: nullon a normal page.x-credits-chargedwas1andx-routewashttp/datacenter: one try, 1,793 ms.AbortSignal.timeout(90000)gives up after 90 seconds, and node-fetch 3 accepts it like the built-infetch.
The same call on the uploads folder that answers 403 to everyone:
MEG_API_KEY=YOUR_KEY node fetch.mjs https://lab.megalania.io/wp-content/uploads/
status: 422 target: 403
route: browser/residential credits: null billing: blocked
tries: [{"rung":"h:dc","identity":"firefox","ms":4097,"outcome":"blocked"},{"rung":"h:res","identity":"firefox","ms":5373,"outcome":"blocked"},{"rung":"c:dc","identity":"chrome","ms":1515,"outcome":"blocked"},{"rung":"c:res","identity":"chrome","ms":1152,"outcome":"blocked"}]
{"error":{"code":"blocked","message":"The site blocked us and there is nothing to show. You are not charged. X-Untried lists the routes left to try; no X-Untrie- Four tries, four 403s, one answer:
422,error.codeblocked,x-billing: blocked. The promise resolves, so readres.statusbefore the body. credits: null: there is no charge header on a block. Count it as 0.
One honest note: our test store blocks nobody, so these runs show the calls, not a bot filter being beaten. On a site that blocks, the call is the same; the price depends on the route that worked, and our MCP tool check_site tells you that price for free.
What it costs
| What we ran | Route | Credits |
|---|---|---|
| The FAQ page as Markdown | h:dc | 1 |
| The uploads folder: 403 on every route | h:dc h:res c:dc c:res | 0 |
seen.mjs and fix.mjs, straight from Node | none | 0 |
Every rule is in Credits. Plans are on the pricing page.
How we measured: node-fetch 3.3.2 and the built-in fetch on Node 24.21, one Linux server, 16 Nov 2026. nc -l 8799 (OpenBSD netcat) for what each sends, with and without the browser headers. Our test store for the 403 and for fix.mjs. Our API on our dev cluster for fetch.mjs, against the same store on our dev domain. The outputs are copied from those runs.
Limits
- Public pages only. We do not log in, and we do not solve CAPTCHAs. When a page needs either, we stop and say so, for free.
- A page built by JavaScript needs
engine=browser. It costs more. See How to scrape JavaScript-rendered pages. - Read the site's terms. A 403 is sometimes the site saying no. Fetch public pages only, and follow the site's terms and robots.txt.
Why sites send 403 at all, and the other layers they check, are in Why your scraper gets 403 Forbidden.