The short answer: a 429 Too Many Requests when scraping means you sent more requests than the site allows in some window, and it wants you to slow down. It is not a ban yet. Read the Retry-After header, wait that long, and send the same request again. With no header, wait 1 second, then 2, then 4, with a little random jitter. Then fix the cause: fewer requests at once and a pause between pages. On our test store, 8 pages at 1 request a second never got a 429. When we sent the same 8 pages to our API all at once, above our limit of 5 at a time, we got 6 429s, waited 1 second each time, and all 8 pages came back, for 8 credits. The 429s cost nothing.
What 429 Too Many Requests means when scraping
429 is the HTTP status for “slow down”. A site or the service in front of it counts your requests: per IP address, per cookie, per API key, or per account, over a window of seconds or minutes. Above the limit, it answers 429 instead of the page.
Two things to know:
Retry-Aftertells you how long to wait. It is a number of seconds (Retry-After: 30) or a date (Retry-After: Wed, 21 Oct 2026 07:28:00 GMT). Not every site sends it.- A
429you ignore can become a403. Many sites count the requests you send after a429too. Keep going at the same speed, and the next step is a block.
A 429 can also come from a proxy or an API between you and the site. Then the limit is theirs, not the site’s. The body or the docs say which.
Back off in Python, for free
This helper waits for Retry-After when there is one, and backs off with jitter when there is not. It treats a 503 the same way, because many sites send a 503 with Retry-After when they are busy.
import random
import sys
import time
from email.utils import parsedate_to_datetime
import requests
def retry_after_seconds(value):
if not value:
return None
if value.strip().isdigit():
return int(value)
try:
return max(0, parsedate_to_datetime(value).timestamp() - time.time())
except (TypeError, ValueError):
return None
def polite_get(session, url, tries=5, base=1.0, cap=60.0):
for attempt in range(tries):
resp = session.get(url, timeout=30)
if resp.status_code not in (429, 503):
return resp
wait = retry_after_seconds(resp.headers.get("Retry-After"))
if wait is None:
wait = random.uniform(0, min(cap, base * 2 ** attempt))
print(f" {resp.status_code} on {url}, waiting {wait:.1f} s")
time.sleep(min(wait, cap))
return resp
session = requests.Session()
session.headers["User-Agent"] = "polite-scraper/1.0"
site = sys.argv[1] if len(sys.argv) > 1 else "https://lab.megalania.io"
for n in range(1, 9):
url = f"{site}/shop/page/{n}/" if n > 1 else f"{site}/shop/"
resp = polite_get(session, url)
print(resp.status_code, url, len(resp.content))
time.sleep(1)pip install requests python polite_get.py
What it printed:
200 https://lab.megalania.io/shop/ 75690 200 https://lab.megalania.io/shop/page/2/ 76272 200 https://lab.megalania.io/shop/page/3/ 75831 200 https://lab.megalania.io/shop/page/4/ 75859 200 https://lab.megalania.io/shop/page/5/ 76517 200 https://lab.megalania.io/shop/page/6/ 76971 200 https://lab.megalania.io/shop/page/7/ 76597 200 https://lab.megalania.io/shop/page/8/ 76261
No 429: one page a second is slow enough for the Lab. To see the retry work, we pointed polite_get at a small local server that answers 429 with Retry-After: 1 twice, then 200. It waited 1 second twice and got the page on the third try.
Why each part is there:
- One
Sessionreuses the connection and keeps cookies, so the site sees one visitor, not eight. time.sleep(1)between pages is the real fix. Backing off only treats the symptom.- Jitter (
random.uniform) stops many workers from waking up at the same moment and hitting the limit again together. - A cap on the wait stops a
Retry-After: 86400from freezing your script for a day. Log it and move on instead.
How fast is too fast?
The site does not tell you its limit, so start slow and measure:
- Start at one request at a time, one a second.
- Raise it a step only after a long run with no
429. - At the first
429, go back one step and stay there.
Run one worker per site. Ten workers on ten sites are fine; ten workers on one site look like a flood. Read the site’s robots.txt too: a Crawl-delay line, when there is one, is the pause the owner asked for.
Our 429s, and the site’s
When you send pages through us, a 429 comes from one of two places, and the error.code in the JSON body says which. Every one of them is free.
error.code |
Who said slow down | What to do |
|---|---|---|
over_concurrency |
Us: you have more requests running at once than your plan allows (5 in the beta). | Wait the seconds in Retry-After, then send it again. |
rate_limited |
Us: too many calls a second to batches, monitors or the account API. | Wait the seconds in Retry-After. |
target_rate_limited |
The site. | We already waited for you: see below. Wait the seconds in Retry-After. |
daily_cost_cap, period_cost_cap |
Us: your usage limit for the day or the grant is used up. | Do not retry. The message says when it resets. |
When the site answers 429 to a datacenter exit, we try one residential exit, at its normal price. If the site still says slow down, you get target_rate_limited, and we hold your requests to that host until Retry-After ends, so you do not pay for more 429s. We did not see this one on the Lab.
A 429 releases the Idempotency-Key. Send the same request with the same key, and it is charged once. Every code is in Error codes.
Retry our 429 in Python
This sends the 8 shop pages all at once, above the beta limit of 5:
import os
import time
import uuid
from concurrent.futures import ThreadPoolExecutor
import requests
API = "https://get.megalania.io/v1"
HEADERS = {"Authorization": f"Bearer {os.environ['MEG_API_KEY']}"}
URLS = ["https://lab.megalania.io/shop/"] + [f"https://lab.megalania.io/shop/page/{n}/" for n in range(2, 9)]
def fetch(url, tries=4):
key = str(uuid.uuid4())
for _ in range(tries):
r = requests.get(API, params={"url": url}, headers={**HEADERS, "Idempotency-Key": key}, timeout=90)
if r.status_code != 429:
return r
err = r.json()["error"]["code"]
wait = int(r.headers.get("Retry-After", "1"))
print(f" 429 {err} on {url}, Retry-After {wait} s")
if err not in ("over_concurrency", "target_rate_limited", "rate_limited"):
return r
time.sleep(wait)
return r
start = time.time()
with ThreadPoolExecutor(max_workers=8) as pool:
results = list(pool.map(fetch, URLS))
for url, r in zip(URLS, results):
print(r.status_code, r.headers.get("X-Cache"), r.headers.get("X-Credits-Charged", "0"), url)
print(f"{sum(int(r.headers.get('X-Credits-Charged', 0)) for r in results)} credits, {time.time() - start:.1f} s")export MEG_API_KEY='meg_sk_...' python burst.py
What it printed:
429 over_concurrency on https://lab.megalania.io/shop/page/2/, Retry-After 1 s 429 over_concurrency on https://lab.megalania.io/shop/page/6/, Retry-After 1 s 429 over_concurrency on https://lab.megalania.io/shop/page/8/, Retry-After 1 s 429 over_concurrency on https://lab.megalania.io/shop/page/2/, Retry-After 1 s 429 over_concurrency on https://lab.megalania.io/shop/page/6/, Retry-After 1 s 429 over_concurrency on https://lab.megalania.io/shop/page/8/, Retry-After 1 s 200 miss 1 https://lab.megalania.io/shop/ 200 miss 1 https://lab.megalania.io/shop/page/2/ 200 miss 1 https://lab.megalania.io/shop/page/3/ 200 miss 1 https://lab.megalania.io/shop/page/4/ 200 miss 1 https://lab.megalania.io/shop/page/5/ 200 miss 1 https://lab.megalania.io/shop/page/6/ 200 miss 1 https://lab.megalania.io/shop/page/7/ 200 miss 1 https://lab.megalania.io/shop/page/8/ 8 credits, 4.3 s
Five pages started at once. The other three got 429 over_concurrency with Retry-After: 1, twice each, while the first five were still running. Each one waited a second and went again with the same key. All 8 pages came back, 1 credit each, in 4.3 seconds. X-Cache: miss says each page was fetched fresh, not served from a recent copy.
Only the codes that end on their own are retried. A daily_cost_cap returns at once: waiting a second will not help. The Python SDK does the same for over_concurrency: it retries once, after Retry-After, with the same key.
Let a batch pace it
Simpler still: do not run the workers yourself. Send the list as one batch. We run a few requests per site at a time, and you poll for the end instead of handling 429s.
import os
import time
import uuid
import requests
API = "https://api.megalania.io/v1/batches"
HEADERS = {"Authorization": f"Bearer {os.environ['MEG_API_KEY']}"}
URLS = ["https://lab.megalania.io/shop/"] + [f"https://lab.megalania.io/shop/page/{n}/" for n in range(2, 9)]
start = time.time()
r = requests.post(API, json={"urls": URLS}, headers={**HEADERS, "Idempotency-Key": str(uuid.uuid4())}, timeout=30)
r.raise_for_status()
batch_id = r.json()["id"]
while True:
r = requests.get(f"{API}/{batch_id}", params={"wait_seconds": 25}, headers=HEADERS, timeout=40)
if r.status_code == 429:
time.sleep(int(r.headers.get("Retry-After", "1")))
continue
r.raise_for_status()
batch = r.json()
if batch["status"] in ("done", "failed", "canceled"):
break
c = batch["counts"]
print(f"{batch['status']}: {batch['done_count']} of {batch['item_count']} pages, {c['charged_credits']} credits, {time.time() - start:.1f} s")What it printed:
done: 8 of 8 pages, 8 credits, 6.2 s
No 429 to handle, and no worker pool to tune. wait_seconds=25 holds each poll until the batch ends, so the loop does not spin. For a list you check every day, see How to track competitor prices and stock every day. For batch options and results, see Batches.
What it costs
| Step | Pages | Credits |
|---|---|---|
| Your own script, one page a second | 8 | 0 |
8 pages at once through /v1, with 6 429s on the way |
8 | 8 |
| The same 8 pages as one batch | 8 | 8 |
Any of our 429s, blocked or our errors |
— | 0 |
These prices are for this site on the datacenter route. A site that blocks it costs more. Check yours first with GET /v1/tier?host=... or the MCP tool check_site; both are free. Plans are on the pricing page.
How we measured: our dev cluster on 16 Nov 2026, from 06:23 to 06:38 UTC, the Lab store on our dev domain, one run of each script. Our balance went down by exactly 8 credits for the burst, so the 429s cost nothing.
Limits
- We do not raise a site’s limit. A site that answers
429to every route stays slow.target_rate_limitedsaves you the cost of hitting it again, not the wait. - A
403after many429s is a block. Slow down first; then read Why your scraper gets 403 Forbidden. - Five requests at once in the beta. More is a free
429 over_concurrency.GET /v1/accountshows yourplan.concurrency.
Fetch public pages only. Read the site’s terms and robots.txt, and follow them.