The short answer: In Node, axios gets a 403 Forbidden when the site's bot filter decides it is not a browser. axios gives itself away three times: User-Agent: axios/1.21.1, an Accept that asks for JSON first, and compress in Accept-Encoding, a format no current browser offers. Set all three headers yourself and the request reads like a browser's. Node's HTTP/1.1 and TLS stay, and no header changes them. When the site reads those, send the URL to our API instead: we fetch it like a browser and climb to a residential IP or a real Chrome when the site blocks us. A block we detect costs nothing.
What axios sends
Start a listener in one terminal with nc -l 8799. It prints every byte a client sends and never answers, so the client times out. Run this in a second terminal:
npm install axios
node -e 'require("axios").get("http://127.0.0.1:8799/", { timeout: 3000 }).catch((e) => console.log(e.message))'GET / HTTP/1.1 Accept: application/json, text/plain, */* User-Agent: axios/1.21.1 Accept-Encoding: gzip, compress, deflate, br Host: 127.0.0.1:8799 Connection: keep-alive
- **
Accept: application/json, text/plain, */*** comes first. That is a JSON API client talking. A browser loading a page asks fortext/htmlfirst. User-Agent: axios/1.21.1names the library and its version.compressinAccept-Encoding. It is the old Unixcompressformat. No current browser offers it, so this header alone marks the request as axios.Hostcomes fourth. Browsers on HTTP/1.1 send it first. Header order is one more thing a bot filter can score.
How axios reports a 403
Our test store answers 403 on its uploads folder, because folder listing is off. That is a real "no", not a bot filter, and a handy way to see what your code gets back:
// npm install axios
import axios from "axios";
try {
await axios.get("https://lab.megalania.io/wp-content/uploads/");
} catch (err) {
console.log(err.message);
console.log(err.name, err.code, err.response.status, err.response.statusText);
console.log(err.response.data.length, "characters of HTML in err.response.data");
}Request failed with status code 403 AxiosError ERR_BAD_REQUEST 403 Forbidden 239 characters of HTML in err.response.data
- axios throws on a 403. Any status outside 200 to 299 rejects the promise, so an
awaitwithouttryends the script with a stack trace. err.codeisERR_BAD_REQUESTfor every 4xx, so branch onerr.response.status, not on the code. The page the site sent is inerr.response.data.
The free fix: browser headers
Send the headers a browser sends. This fixes sites that read only the headers:
// npm install axios
import axios from "axios";
const resp = await axios.get("https://lab.megalania.io/returns/", {
headers: {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/154.0 Safari/537.36",
Accept: "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"Accept-Language": "en-US,en;q=0.9",
"Accept-Encoding": "gzip, deflate, br",
},
});
console.log(resp.status, "HTTP version:", resp.request.res.httpVersion);200 HTTP version: 1.1
- Your headers replace the axios ones. We sent
fix.mjstonc:Accept,User-Agent,Accept-Language,Accept-Encoding: gzip, deflate, br, thenHostandConnection. - Set
Accept-Encodingyourself. axios adds its own only when you did not, so without that line thecompresstell stays in the fixed request. resp.request.res.httpVersionis1.1: axios in Node uses Node'shttpmodule, so no HTTP/2, and the TLS hello is Node's.
The fix that climbs: our API
Send the page to our API instead. We fetch it like a real browser, move to a residential IP or a real Chrome when the site blocks us, and a block we detect costs nothing:
// npm install axios
// Run: MEG_API_KEY=YOUR_KEY node fetch.mjs https://lab.megalania.io/returns/
import axios from "axios";
const resp = await axios.post(
"https://get.megalania.io/v1",
{ url: process.argv[2], response_types: "markdown", max_credits: "64" },
{ headers: { Authorization: `Bearer ${process.env.MEG_API_KEY}` }, timeout: 90000, responseType: "text", validateStatus: () => true },
);
console.log("status:", resp.status, "target:", resp.headers["x-target-status"]);
console.log("route:", resp.headers["x-route"], "credits:", resp.headers["x-credits-charged"], "billing:", resp.headers["x-billing"]);
console.log("tries:", resp.headers["x-hops"]);
console.log(resp.data.slice(0, 160));status: 200 target: 200
route: http/datacenter credits: 1 billing: undefined
tries: [{"rung":"h:dc","identity":"chrome","ms":1667,"outcome":"ok"}]
# Returns
Changed your mind? Send it back within 60 days for a full refund, as long as it is unused and has its tags.
## How to return
1. Write to us throughvalidateStatus: () => truestops axios from throwing on our own errors, so you read our JSON instead of a stack trace.responseType: "text"keeps the Markdown a string.- axios gives you header names in lower case:
x-credits-chargedwas1andx-routewashttp/datacenter, one try in 1,667 ms.
The same call on the uploads folder that answers 403 to everyone:
MEG_API_KEY=YOUR_KEY node fetch.mjs https://lab.megalania.io/wp-content/uploads/
status: 422 target: 403
route: browser/residential credits: undefined billing: blocked
tries: [{"rung":"h:dc","identity":"firefox","ms":4601,"outcome":"blocked"},{"rung":"h:res","identity":"firefox","ms":4576,"outcome":"blocked"},{"rung":"c:dc","identity":"chrome","ms":1441,"outcome":"blocked"},{"rung":"c:res","identity":"chrome","ms":2500,"outcome":"blocked"}]
{"error":{"code":"blocked","message":"The site blocked us and there is nothing to show. You are not charged. X-Untried lists the routes left to try; no X-Untrie- Four tries, four 403s, one answer:
422,error.codeblocked,x-billing: blocked. WithvalidateStatusit is a normal response, not a thrown error. x-credits-chargedisundefinedon a block because we send no charge header. Count it as 0.
One honest note: our test store blocks nobody, so these runs show the calls, not a bot filter being beaten. On a site that blocks, the call is the same; the price depends on the route that worked, and our MCP tool check_site tells you that price for free.
What it costs
| What we ran | Route | Credits |
|---|---|---|
| The returns page as Markdown | h:dc | 1 |
| The uploads folder: 403 on every route | h:dc h:res c:dc c:res | 0 |
seen.mjs and fix.mjs, straight from Node | none | 0 |
Every rule is in Credits. Plans are on the pricing page.
How we measured: axios 1.21.1 on Node 24.21, one Linux server, 16 Nov 2026. nc -l 8799 (OpenBSD netcat) for what axios sends, with and without the browser headers. Our test store for the 403 and for fix.mjs. Our API on our dev cluster for fetch.mjs, against the same store on our dev domain. The outputs are copied from those runs.
Limits
- Public pages only. We do not log in, and we do not solve CAPTCHAs. When a page needs either, we stop and say so, for free.
- A page built by JavaScript needs
engine=browser. It costs more. See How to scrape JavaScript-rendered pages. - Read the site's terms. A 403 is sometimes the site saying no. Fetch public pages only, and follow the site's terms and robots.txt.
Why sites send 403 at all, and the other layers they check, are in Why your scraper gets 403 Forbidden.