The short answer: curl gets a 403 Forbidden when the site's bot filter decides it is not a browser, and curl's request is the shortest there is: Host, User-Agent: curl/8.14.1 and Accept: */*. -A with a browser User-Agent, Accept-Language and --compressed fix sites that read only the headers. curl already speaks HTTP/2, but its TLS hello is OpenSSL's, not Chrome's, and no flag changes that. And by default curl exits 0 on a 403, so scripts miss it. When headers are not enough, send the URL to our API: same curl, a different URL. A block we detect costs nothing.
What curl sends
Start a listener in one terminal with nc -l 8799. It prints every byte a client sends and never answers, so the client times out. Run this in a second terminal:
curl -m 3 http://127.0.0.1:8799/
GET / HTTP/1.1 Host: 127.0.0.1:8799 User-Agent: curl/8.14.1 Accept: */*
- Three headers. That is the whole request: no
Accept-Language, noSec-Fetch-*, not even aConnectionheader. - No
Accept-Encoding. curl asks for compression only with--compressed. Every browser asks for it, so its absence is a tell. - HTTP/2 over HTTPS. Built with nghttp2, curl 8.14.1 got our test store over HTTP/2, as a browser does. The TLS hello is OpenSSL's, which a bot filter can tell from Chrome's.
How curl reports a 403
Our test store answers 403 on its uploads folder, because folder listing is off. That is a real "no", not a bot filter, and a handy way to see what your code gets back:
curl -s https://lab.megalania.io/wp-content/uploads/ | head -c 120; echo; echo "exit code: ${PIPESTATUS[0]}"
curl -sS --fail https://lab.megalania.io/wp-content/uploads/; echo "exit code: $?"<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd"> <html><head> <title>403 Forbi exit code: 0 curl: (22) The requested URL returned error: 403 exit code: 22
- Without
--fail, curl exits 0 on a 403 and prints the error page as if it were the page. A shell script that checks$?goes on with an error page saved as data. - With
--fail(-f), curl printscurl: (22) The requested URL returned error: 403and exits 22.-sSkeeps the progress meter off but the error line on.
The free fix: browser headers
Send the headers a browser sends. This fixes sites that read only the headers:
curl -sS -o /dev/null -w '%{http_code} HTTP version: %{http_version}\n' --compressed \
-A 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/154.0 Safari/537.36' \
-H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' \
-H 'Accept-Language: en-US,en;q=0.9' \
https://lab.megalania.io/about/200 HTTP version: 2
-Asets theUser-Agent.--compressedasks for compression and unpacks the answer.- We sent the same flags to
nc: curl 8.14.1 wroteAccept-Encoding: deflate, gzip, br, zstd. Browsers writegzip, deflate, br, zstd, so even the fixed request has curl's order. - In your browser's developer tools, Copy as cURL on a request gives every header the browser sent, in one paste.
The fix that climbs: our API
Send the page to our API instead. We fetch it like a real browser, move to a residential IP or a real Chrome when the site blocks us, and a block we detect costs nothing:
curl -sS https://get.megalania.io/v1 \
-H "Authorization: Bearer $MEG_API_KEY" \
-H "Content-Type: application/json" \
-D - -o page.md \
-d '{"url": "https://lab.megalania.io/about/", "response_types": "markdown", "max_credits": "64"}' \
| grep -iE '^(HTTP|x-(target-status|route|credits-charged|billing|hops):)'
head -c 160 page.mdHTTP/2 200
x-credits-charged: 1
x-hops: [{"rung":"h:dc","identity":"chrome","ms":1896,"outcome":"ok"}]
x-route: http/datacenter
x-target-status: 200

# About
Gibber & Co. is a demo store by [Megal-D -prints our response headers andgrepkeeps the ones that matter. The page goes topage.mdas Markdown; its first image came along as a Markdown image.$MEG_API_KEYcomes from your shell, so the key never sits in your history or in the URL.
The same call on the uploads folder that answers 403 to everyone:
curl -sS https://get.megalania.io/v1 \
-H "Authorization: Bearer $MEG_API_KEY" \
-H "Content-Type: application/json" \
-D - -o page.md \
-d '{"url": "https://lab.megalania.io/wp-content/uploads/", "response_types": "markdown", "max_credits": "64"}' \
| grep -iE '^(HTTP|x-(target-status|route|credits-charged|billing|hops):)'
head -c 160 page.mdHTTP/2 422
x-billing: blocked
x-hops: [{"rung":"h:dc","identity":"firefox","ms":3940,"outcome":"blocked"},{"rung":"h:res","identity":"firefox","ms":5332,"outcome":"blocked"},{"rung":"c:dc","identity":"chrome","ms":1256,"outcome":"blocked"},{"rung":"c:res","identity":"chrome","ms":1507,"outcome":"blocked"}]
x-route: browser/residential
x-target-status: 403
{"error":{"code":"blocked","message":"The site blocked us and there is nothing to show. You are not charged. X-Untried lists the routes left to try; no X-Untriepage.mdnow holds our error JSON, not a page. Check the status line first, or add--fail-with-bodyso curl exits 22 and still writes the JSON.x-billing: blockedand nox-credits-chargedline: four routes tried, 0 credits.
One honest note: our test store blocks nobody, so these runs show the calls, not a bot filter being beaten. On a site that blocks, the call is the same; the price depends on the route that worked, and our MCP tool check_site tells you that price for free.
What it costs
| What we ran | Route | Credits |
|---|---|---|
| The about page as Markdown | h:dc | 1 |
| The uploads folder: 403 on every route | h:dc h:res c:dc c:res | 0 |
| Plain curl, with or without the browser flags | none | 0 |
Every rule is in Credits. Plans are on the pricing page.
How we measured: curl 8.14.1 with OpenSSL 3.5 and nghttp2, one Linux server, 16 Nov 2026. nc -l 8799 (OpenBSD netcat) for what curl sends, with and without the browser flags. Our test store for the 403 and for the browser-flags command. Our API on our dev cluster for the last two commands, against the same store on our dev domain. The outputs are copied from those runs.
Limits
- Public pages only. We do not log in, and we do not solve CAPTCHAs. When a page needs either, we stop and say so, for free.
- A page built by JavaScript needs
engine=browser. It costs more. See How to scrape JavaScript-rendered pages. - Read the site's terms. A 403 is sometimes the site saying no. Fetch public pages only, and follow the site's terms and robots.txt.
Why sites send 403 at all, and the other layers they check, are in Why your scraper gets 403 Forbidden.