MegalaniaBlog
  1. Blog
  2. How to fix
  3. curl 403

curl 403 Forbidden: why, and how to fix it

curl sends three headers and exits 0 on a 403. What it sends, how to make it fail, which flags fix it, and a call that climbs. Tested with curl 8.14.1.

Jordan HayesDeveloper Relations
3 min read
How to fix
A padlock cut open. Inside, four pins named UA, headers, cookies and TLS. Three sit on the line, one sits low.

The short answer: curl gets a 403 Forbidden when the site's bot filter decides it is not a browser, and curl's request is the shortest there is: Host, User-Agent: curl/8.14.1 and Accept: */*. -A with a browser User-Agent, Accept-Language and --compressed fix sites that read only the headers. curl already speaks HTTP/2, but its TLS hello is OpenSSL's, not Chrome's, and no flag changes that. And by default curl exits 0 on a 403, so scripts miss it. When headers are not enough, send the URL to our API: same curl, a different URL. A block we detect costs nothing.

What curl sends

Start a listener in one terminal with nc -l 8799. It prints every byte a client sends and never answers, so the client times out. Run this in a second terminal:

Terminal 2
curl -m 3 http://127.0.0.1:8799/
Output
GET / HTTP/1.1
Host: 127.0.0.1:8799
User-Agent: curl/8.14.1
Accept: */*
  • Three headers. That is the whole request: no Accept-Language, no Sec-Fetch-*, not even a Connection header.
  • No Accept-Encoding. curl asks for compression only with --compressed. Every browser asks for it, so its absence is a tell.
  • HTTP/2 over HTTPS. Built with nghttp2, curl 8.14.1 got our test store over HTTP/2, as a browser does. The TLS hello is OpenSSL's, which a bot filter can tell from Chrome's.

How curl reports a 403

Our test store answers 403 on its uploads folder, because folder listing is off. That is a real "no", not a bot filter, and a handy way to see what your code gets back:

Shell
curl -s https://lab.megalania.io/wp-content/uploads/ | head -c 120; echo; echo "exit code: ${PIPESTATUS[0]}"
curl -sS --fail https://lab.megalania.io/wp-content/uploads/; echo "exit code: $?"
Output
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html><head>
<title>403 Forbi
exit code: 0
curl: (22) The requested URL returned error: 403
exit code: 22
  • Without --fail, curl exits 0 on a 403 and prints the error page as if it were the page. A shell script that checks $? goes on with an error page saved as data.
  • With --fail (-f), curl prints curl: (22) The requested URL returned error: 403 and exits 22. -sS keeps the progress meter off but the error line on.

The free fix: browser headers

Send the headers a browser sends. This fixes sites that read only the headers:

Shell
curl -sS -o /dev/null -w '%{http_code} HTTP version: %{http_version}\n' --compressed \
  -A 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/154.0 Safari/537.36' \
  -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' \
  -H 'Accept-Language: en-US,en;q=0.9' \
  https://lab.megalania.io/about/
Output
200 HTTP version: 2
  • -A sets the User-Agent. --compressed asks for compression and unpacks the answer.
  • We sent the same flags to nc: curl 8.14.1 wrote Accept-Encoding: deflate, gzip, br, zstd. Browsers write gzip, deflate, br, zstd, so even the fixed request has curl's order.
  • In your browser's developer tools, Copy as cURL on a request gives every header the browser sent, in one paste.
Free beta, by invite76,000 credits for 30 days, no card. Apply, and we email your invite.Apply for the beta

The fix that climbs: our API

Send the page to our API instead. We fetch it like a real browser, move to a residential IP or a real Chrome when the site blocks us, and a block we detect costs nothing:

Shell
curl -sS https://get.megalania.io/v1 \
  -H "Authorization: Bearer $MEG_API_KEY" \
  -H "Content-Type: application/json" \
  -D - -o page.md \
  -d '{"url": "https://lab.megalania.io/about/", "response_types": "markdown", "max_credits": "64"}' \
  | grep -iE '^(HTTP|x-(target-status|route|credits-charged|billing|hops):)'
head -c 160 page.md
Output
HTTP/2 200 
x-credits-charged: 1
x-hops: [{"rung":"h:dc","identity":"chrome","ms":1896,"outcome":"ok"}]
x-route: http/datacenter
x-target-status: 200
![Wooden shelves with jars in a small shop](https://lab.megalania.io/wp-content/uploads/gibber/about-shop.webp)

# About

Gibber & Co. is a demo store by [Megal
  • -D - prints our response headers and grep keeps the ones that matter. The page goes to page.md as Markdown; its first image came along as a Markdown image.
  • $MEG_API_KEY comes from your shell, so the key never sits in your history or in the URL.

The same call on the uploads folder that answers 403 to everyone:

Shell
curl -sS https://get.megalania.io/v1 \
  -H "Authorization: Bearer $MEG_API_KEY" \
  -H "Content-Type: application/json" \
  -D - -o page.md \
  -d '{"url": "https://lab.megalania.io/wp-content/uploads/", "response_types": "markdown", "max_credits": "64"}' \
  | grep -iE '^(HTTP|x-(target-status|route|credits-charged|billing|hops):)'
head -c 160 page.md
Output
HTTP/2 422
x-billing: blocked
x-hops: [{"rung":"h:dc","identity":"firefox","ms":3940,"outcome":"blocked"},{"rung":"h:res","identity":"firefox","ms":5332,"outcome":"blocked"},{"rung":"c:dc","identity":"chrome","ms":1256,"outcome":"blocked"},{"rung":"c:res","identity":"chrome","ms":1507,"outcome":"blocked"}]
x-route: browser/residential
x-target-status: 403
{"error":{"code":"blocked","message":"The site blocked us and there is nothing to show. You are not charged. X-Untried lists the routes left to try; no X-Untrie
  • page.md now holds our error JSON, not a page. Check the status line first, or add --fail-with-body so curl exits 22 and still writes the JSON.
  • x-billing: blocked and no x-credits-charged line: four routes tried, 0 credits.

One honest note: our test store blocks nobody, so these runs show the calls, not a bot filter being beaten. On a site that blocks, the call is the same; the price depends on the route that worked, and our MCP tool check_site tells you that price for free.

What it costs

What we ranRouteCredits
The about page as Markdownh:dc1
The uploads folder: 403 on every routeh:dc h:res c:dc c:res0
Plain curl, with or without the browser flagsnone0

Every rule is in Credits. Plans are on the pricing page.

How we measured: curl 8.14.1 with OpenSSL 3.5 and nghttp2, one Linux server, 16 Nov 2026. nc -l 8799 (OpenBSD netcat) for what curl sends, with and without the browser flags. Our test store for the 403 and for the browser-flags command. Our API on our dev cluster for the last two commands, against the same store on our dev domain. The outputs are copied from those runs.

Limits

  • Public pages only. We do not log in, and we do not solve CAPTCHAs. When a page needs either, we stop and say so, for free.
  • A page built by JavaScript needs engine=browser. It costs more. See How to scrape JavaScript-rendered pages.
  • Read the site's terms. A 403 is sometimes the site saying no. Fetch public pages only, and follow the site's terms and robots.txt.

Why sites send 403 at all, and the other layers they check, are in Why your scraper gets 403 Forbidden.