The short answer: Scrapy raises DownloadTimeoutError when a response does not arrive within DOWNLOAD_TIMEOUT: 180 seconds by default. Its retry middleware then tries the request again, twice by default, so one hung page can hold a slot for 9 minutes before Scrapy logs Gave up retrying. Set DOWNLOAD_TIMEOUT to what a page really needs, raise RETRY_TIMES for flaky servers, turn on AutoThrottle, and add an errback so you see each failure. When a page is slow because it needs a browser, send it to our API: one request from your spider, and an answer that runs out of time costs nothing.
What Scrapy does on a slow server
Start a listener in one terminal with nc -lk 8799. It takes each connection and never answers, like a server that hangs. Run this in a second terminal:
scrapy runspider slow.py
# slow.py
import scrapy
class Slow(scrapy.Spider):
name = "slow"
start_urls = ["http://127.0.0.1:8799/"]
custom_settings = {"DOWNLOAD_TIMEOUT": 5}
def parse(self, response):
print(response.status)Retrying <GET http://127.0.0.1:8799/> (failed 1 times): Getting http://127.0.0.1:8799/ took longer than 5.0 seconds. Retrying <GET http://127.0.0.1:8799/> (failed 2 times): Getting http://127.0.0.1:8799/ took longer than 5.0 seconds. Gave up retrying <GET http://127.0.0.1:8799/> (failed 3 times): Getting http://127.0.0.1:8799/ took longer than 5.0 seconds. Error downloading <GET http://127.0.0.1:8799/> 'downloader/exception_type_count/scrapy.exceptions.DownloadTimeoutError': 3, 'elapsed_time_seconds': 15.06128920300398, 'retry/count': 2,
- Three tries of 5 seconds, then it gives up: 15 seconds in all. We set
DOWNLOAD_TIMEOUTto 5 to keep the run short. With the default of 180, the same page takes 9 minutes. ncgot three requests, each withUser-Agent: Scrapy/2.18.0andAccept-Language: en. A retry is a new request on a new connection.- The stats say
DownloadTimeoutError3 times andretry/count2: the first try plusRETRY_TIMES, which is 2 by default.
How Scrapy reports a timeout
Our test store answers fast, so give it a limit too short to meet. The error is the same one a slow site gives you:
import scrapy
class Seen(scrapy.Spider):
name = "seen"
custom_settings = {"DOWNLOAD_TIMEOUT": 0.05, "LOG_LEVEL": "ERROR"}
async def start(self):
yield scrapy.Request("https://lab.megalania.io/returns/", errback=self.failed)
def failed(self, failure):
print(type(failure.value).__name__, "|", failure.value)ERROR: Gave up retrying <GET https://lab.megalania.io/returns/> (failed 3 times): Getting https://lab.megalania.io/returns/ took longer than 0.05 seconds. DownloadTimeoutError | Getting https://lab.megalania.io/returns/ took longer than 0.05 seconds.
- The
errbackgets the failure after the last retry, withDownloadTimeoutErrorasfailure.value. Without anerrback, the spider only logs it. - The retries ran before the
errback, so one call tofailed()stands for three tries.
The free fix: timeouts and retries
Set a limit on every request, and retry a few times with a pause. This fixes servers that are slow now and then:
import scrapy
class Fixed(scrapy.Spider):
name = "fixed"
start_urls = ["https://lab.megalania.io/returns/"]
custom_settings = {"DOWNLOAD_TIMEOUT": 20, "RETRY_TIMES": 3, "AUTOTHROTTLE_ENABLED": True, "LOG_LEVEL": "ERROR"}
def parse(self, response):
print(response.status, response.css("h1::text").get(), len(response.text), "characters")200 Returns 55477 characters
DOWNLOAD_TIMEOUT: 20fails a hung page after 20 seconds, not 180. Pick it from your slowest good page, not from the default.RETRY_TIMES: 3gives a flaky server four tries. AutoThrottle slows down when the server's answers slow down, which often stops the timeouts in the first place.
The fix that climbs: our API
Send the page to our API instead. We fetch it, run a real Chrome when you ask, and an answer that runs out of time costs nothing:
# Run: MEG_API_KEY=YOUR_KEY scrapy runspider api.py -a page=https://lab.megalania.io/returns/
import os
import scrapy
class ViaApi(scrapy.Spider):
name = "via_api"
custom_settings = {"DOWNLOAD_TIMEOUT": 100, "LOG_LEVEL": "ERROR"}
async def start(self):
body = {"url": self.page, "response_types": "markdown", "max_credits": "64"}
body.update({k: v for k, v in vars(self).items() if k in ("engine", "actions", "timeout")})
yield scrapy.http.JsonRequest(
"https://get.megalania.io/v1",
data=body,
headers={"Authorization": f"Bearer {os.environ['MEG_API_KEY']}"},
meta={"handle_httpstatus_list": [422, 503]},
)
def parse(self, response):
h = lambda k: (response.headers.get(k) or b"-").decode()
print("status:", response.status, "route:", h("X-Route"), "credits:", h("X-Credits-Charged"))
print(response.text[:150])status: 200 route: http/datacenter credits: 1 # Returns Changed your mind? Send it back within 60 days for a full refund, as long as it is unused and has its tags. ## How to return 1. Write to
JsonRequestsends the body as JSON with the right header.DOWNLOAD_TIMEOUT: 100is longer than our own limit of 90 seconds a page.handle_httpstatus_listlets our422and503answers reachparse. Without it, Scrapy drops them before your code sees our error JSON.
The same call on a page whose list never finishes loading, with a 20-second limit:
MEG_API_KEY=YOUR_KEY scrapy runspider api.py -a page=https://lab.megalania.io/new-arrivals/ \ -a engine=browser -a actions=wait_for:#item-99 -a timeout=20
status: 422 route: - credits: -
{"error":{"code":"action_failed","field":"actions","message":"actions entry 0 could not run: no visible element matched its selector in time, or the t#item-99never appears on this page, so the browser waited until the time for actions ran out, and we answered422action_failed.- No
X-Credits-Chargedheader, so the spider printed-: nothing was charged.-apasses each setting to the spider as an attribute.
One honest note: our test store is fast, so these runs show a short limit or a page that never gets ready, not a slow site. On a slow site, the call is the same; the price depends on the route that worked, and our MCP tool check_site tells you that price for free.
What it costs
| What we ran | Route | Credits |
|---|---|---|
| The returns page as Markdown | h:dc | 1 |
| A page whose list never finished, in the browser | c:dc | 0 |
slow.py, seen.py and fix.py, straight from Scrapy | none | 0 |
Every rule is in Credits. Plans are on the pricing page.
How we measured: Scrapy 2.18.0 with Twisted 26.10 on Python 3.13.5, one Linux server, 16 Nov 2026. nc -lk 8799 (OpenBSD netcat) as a server that never answers. Our test store for the short limit and for fix.py. Our API on our dev cluster for api.py, against the same store on our dev domain. The outputs are copied from those runs; the log lines are cut to the message.
Limits
- Public pages only. We do not log in, and we do not solve CAPTCHAs. When a page needs either, we stop and say so, for free.
- A page built by JavaScript needs
engine=browser. It costs more. See How to scrape JavaScript-rendered pages. - Slow can mean slow down. A site that answers slowly may be under load. Keep your rate low, fetch public pages only, and follow the site's terms and robots.txt.
What to do when a site answers 429 instead is in 429 Too Many Requests when scraping.