MegalaniaBlog
  1. Blog
  2. How to fix
  3. Scrapy timeout

Scrapy timeout: DownloadTimeoutError, retries, and the fix

Scrapy waits 180 seconds a try and retries twice. What it does on a slow server, how it reports a timeout, and the settings that fix it. Scrapy 2.18.

Jordan HayesDeveloper Relations
3 min read
How to fix
A request arrow leaves a terminal card and stops at a stopwatch. Behind it a server card sits with a spinning wait icon. A second arrow goes around, through a small retry loop, and reaches a page that reads 200.

The short answer: Scrapy raises DownloadTimeoutError when a response does not arrive within DOWNLOAD_TIMEOUT: 180 seconds by default. Its retry middleware then tries the request again, twice by default, so one hung page can hold a slot for 9 minutes before Scrapy logs Gave up retrying. Set DOWNLOAD_TIMEOUT to what a page really needs, raise RETRY_TIMES for flaky servers, turn on AutoThrottle, and add an errback so you see each failure. When a page is slow because it needs a browser, send it to our API: one request from your spider, and an answer that runs out of time costs nothing.

What Scrapy does on a slow server

Start a listener in one terminal with nc -lk 8799. It takes each connection and never answers, like a server that hangs. Run this in a second terminal:

Terminal 2
scrapy runspider slow.py

# slow.py
import scrapy


class Slow(scrapy.Spider):
    name = "slow"
    start_urls = ["http://127.0.0.1:8799/"]
    custom_settings = {"DOWNLOAD_TIMEOUT": 5}

    def parse(self, response):
        print(response.status)
Output
Retrying <GET http://127.0.0.1:8799/> (failed 1 times): Getting http://127.0.0.1:8799/ took longer than 5.0 seconds.
Retrying <GET http://127.0.0.1:8799/> (failed 2 times): Getting http://127.0.0.1:8799/ took longer than 5.0 seconds.
Gave up retrying <GET http://127.0.0.1:8799/> (failed 3 times): Getting http://127.0.0.1:8799/ took longer than 5.0 seconds.
Error downloading <GET http://127.0.0.1:8799/>
'downloader/exception_type_count/scrapy.exceptions.DownloadTimeoutError': 3,
'elapsed_time_seconds': 15.06128920300398,
'retry/count': 2,
  • Three tries of 5 seconds, then it gives up: 15 seconds in all. We set DOWNLOAD_TIMEOUT to 5 to keep the run short. With the default of 180, the same page takes 9 minutes.
  • nc got three requests, each with User-Agent: Scrapy/2.18.0 and Accept-Language: en. A retry is a new request on a new connection.
  • The stats say DownloadTimeoutError 3 times and retry/count 2: the first try plus RETRY_TIMES, which is 2 by default.

How Scrapy reports a timeout

Our test store answers fast, so give it a limit too short to meet. The error is the same one a slow site gives you:

seen.py
import scrapy


class Seen(scrapy.Spider):
    name = "seen"
    custom_settings = {"DOWNLOAD_TIMEOUT": 0.05, "LOG_LEVEL": "ERROR"}

    async def start(self):
        yield scrapy.Request("https://lab.megalania.io/returns/", errback=self.failed)

    def failed(self, failure):
        print(type(failure.value).__name__, "|", failure.value)
Output
ERROR: Gave up retrying <GET https://lab.megalania.io/returns/> (failed 3 times): Getting https://lab.megalania.io/returns/ took longer than 0.05 seconds.
DownloadTimeoutError | Getting https://lab.megalania.io/returns/ took longer than 0.05 seconds.
  • The errback gets the failure after the last retry, with DownloadTimeoutError as failure.value. Without an errback, the spider only logs it.
  • The retries ran before the errback, so one call to failed() stands for three tries.

The free fix: timeouts and retries

Set a limit on every request, and retry a few times with a pause. This fixes servers that are slow now and then:

fix.py
import scrapy


class Fixed(scrapy.Spider):
    name = "fixed"
    start_urls = ["https://lab.megalania.io/returns/"]
    custom_settings = {"DOWNLOAD_TIMEOUT": 20, "RETRY_TIMES": 3, "AUTOTHROTTLE_ENABLED": True, "LOG_LEVEL": "ERROR"}

    def parse(self, response):
        print(response.status, response.css("h1::text").get(), len(response.text), "characters")
Output
200 Returns 55477 characters
  • DOWNLOAD_TIMEOUT: 20 fails a hung page after 20 seconds, not 180. Pick it from your slowest good page, not from the default.
  • RETRY_TIMES: 3 gives a flaky server four tries. AutoThrottle slows down when the server's answers slow down, which often stops the timeouts in the first place.
Free beta, by invite76,000 credits for 30 days, no card. Apply, and we email your invite.Apply for the beta

The fix that climbs: our API

Send the page to our API instead. We fetch it, run a real Chrome when you ask, and an answer that runs out of time costs nothing:

api.py
# Run: MEG_API_KEY=YOUR_KEY scrapy runspider api.py -a page=https://lab.megalania.io/returns/
import os

import scrapy


class ViaApi(scrapy.Spider):
    name = "via_api"
    custom_settings = {"DOWNLOAD_TIMEOUT": 100, "LOG_LEVEL": "ERROR"}

    async def start(self):
        body = {"url": self.page, "response_types": "markdown", "max_credits": "64"}
        body.update({k: v for k, v in vars(self).items() if k in ("engine", "actions", "timeout")})
        yield scrapy.http.JsonRequest(
            "https://get.megalania.io/v1",
            data=body,
            headers={"Authorization": f"Bearer {os.environ['MEG_API_KEY']}"},
            meta={"handle_httpstatus_list": [422, 503]},
        )

    def parse(self, response):
        h = lambda k: (response.headers.get(k) or b"-").decode()
        print("status:", response.status, "route:", h("X-Route"), "credits:", h("X-Credits-Charged"))
        print(response.text[:150])
Output
status: 200 route: http/datacenter credits: 1
# Returns

Changed your mind? Send it back within 60 days for a full refund, as long as it is unused and has its tags.

## How to return

1. Write to
  • JsonRequest sends the body as JSON with the right header. DOWNLOAD_TIMEOUT: 100 is longer than our own limit of 90 seconds a page.
  • handle_httpstatus_list lets our 422 and 503 answers reach parse. Without it, Scrapy drops them before your code sees our error JSON.

The same call on a page whose list never finishes loading, with a 20-second limit:

Terminal
MEG_API_KEY=YOUR_KEY scrapy runspider api.py -a page=https://lab.megalania.io/new-arrivals/ \
  -a engine=browser -a actions=wait_for:#item-99 -a timeout=20
Output
status: 422 route: - credits: -
{"error":{"code":"action_failed","field":"actions","message":"actions entry 0 could not run: no visible element matched its selector in time, or the t
  • #item-99 never appears on this page, so the browser waited until the time for actions ran out, and we answered 422 action_failed.
  • No X-Credits-Charged header, so the spider printed -: nothing was charged. -a passes each setting to the spider as an attribute.

One honest note: our test store is fast, so these runs show a short limit or a page that never gets ready, not a slow site. On a slow site, the call is the same; the price depends on the route that worked, and our MCP tool check_site tells you that price for free.

What it costs

What we ranRouteCredits
The returns page as Markdownh:dc1
A page whose list never finished, in the browserc:dc0
slow.py, seen.py and fix.py, straight from Scrapynone0

Every rule is in Credits. Plans are on the pricing page.

How we measured: Scrapy 2.18.0 with Twisted 26.10 on Python 3.13.5, one Linux server, 16 Nov 2026. nc -lk 8799 (OpenBSD netcat) as a server that never answers. Our test store for the short limit and for fix.py. Our API on our dev cluster for api.py, against the same store on our dev domain. The outputs are copied from those runs; the log lines are cut to the message.

Limits

  • Public pages only. We do not log in, and we do not solve CAPTCHAs. When a page needs either, we stop and say so, for free.
  • A page built by JavaScript needs engine=browser. It costs more. See How to scrape JavaScript-rendered pages.
  • Slow can mean slow down. A site that answers slowly may be under load. Keep your rate low, fetch public pages only, and follow the site's terms and robots.txt.

What to do when a site answers 429 instead is in 429 Too Many Requests when scraping.