The short answer: Selenium raises TimeoutException in two places. driver.get() waits up to 300 seconds by default for the page to load, then fails with timeout: Timed out receiving message from renderer. A WebDriverWait that never sees its condition fails too, with an empty message, so the error does not say which element was missing. Set driver.set_page_load_timeout() to a limit you can live with, wait for the element you need with expected_conditions, and retry get() a few times. When you would rather not run Chrome yourself, our API runs the same wait in our browser, and an answer that runs out of time costs nothing.
What Selenium does on a slow server
Start a listener in one terminal with nc -lk 8799. It takes each connection and never answers, like a server that hangs. Run this in a second terminal:
import time
from selenium import webdriver
from selenium.common.exceptions import TimeoutException
opts = webdriver.ChromeOptions()
opts.add_argument("--headless=new")
driver = webdriver.Chrome(options=opts)
print("default page load timeout:", driver.timeouts.page_load, "s")
driver.set_page_load_timeout(5)
start = time.monotonic()
try:
driver.get("http://127.0.0.1:8799/")
except TimeoutException as err:
print(type(err).__name__, "|", err.msg.splitlines()[0], f"| after {time.monotonic() - start:.1f} s")
driver.quit()default page load timeout: 300.0 s TimeoutException | timeout: Timed out receiving message from renderer: -0.003 | after 5.0 s
- Without
set_page_load_timeout,get()would have waited 300 seconds, the defaultdriver.timeouts.page_load. Timed out receiving message from rendereris ChromeDriver's message for a page load that ran out of time. The number after it is not your limit: here it was-0.003.
How Selenium reports a timeout
Our test store answers fast, so give it a limit too short to meet. The error is the same one a slow site gives you:
from selenium import webdriver
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
opts = webdriver.ChromeOptions()
opts.add_argument("--headless=new")
driver = webdriver.Chrome(options=opts)
driver.get("https://lab.megalania.io/new-arrivals/")
try:
WebDriverWait(driver, 3).until(EC.presence_of_element_located((By.CSS_SELECTOR, "#item-99")))
except TimeoutException as err:
print(type(err).__name__, "| message:", repr(err.msg))
print("cards on the page:", len(driver.find_elements(By.CSS_SELECTOR, "#arrivals li")))
driver.quit()TimeoutException | message: '' cards on the page: 12
- The message is empty.
WebDriverWait.until()raisesTimeoutExceptionwith no text unless you pass one:until(condition, message="..."). - The page was fine: 12 cards, and no
#item-99. A wrong selector and a slow page look the same from here.
The free fix: timeouts and retries
Set a limit on every request, and retry a few times with a pause. This fixes servers that are slow now and then:
import time
from selenium import webdriver
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
opts = webdriver.ChromeOptions()
opts.add_argument("--headless=new")
driver = webdriver.Chrome(options=opts)
driver.set_page_load_timeout(30)
start = time.monotonic()
for attempt in range(3):
try:
driver.get("https://lab.megalania.io/new-arrivals/")
WebDriverWait(driver, 20).until(EC.presence_of_element_located((By.CSS_SELECTOR, "#item-12")))
break
except TimeoutException:
time.sleep(2 ** attempt)
cards = driver.find_elements(By.CSS_SELECTOR, "#arrivals li")
print(len(cards), f"cards in {time.monotonic() - start:.1f} s | last:", cards[-1].find_element(By.CSS_SELECTOR, "h2").text)
driver.quit()12 cards in 0.5 s | last: Cable Knit Cotton Throw
presence_of_element_locatedon#item-12waits for the 12th card, sofind_elementsruns after the list is drawn.- Each failed try sleeps 1, 2, then 4 seconds. Catch
TimeoutExceptiononly, so a wrong selector or a closed browser still stops the script.
The fix that climbs: our API
Send the page to our API instead. We fetch it, run a real Chrome when you ask, and an answer that runs out of time costs nothing:
# Run: MEG_API_KEY=YOUR_KEY python fetch.py '#item-12'
import os
import sys
import requests
r = requests.post(
"https://get.megalania.io/v1",
headers={"Authorization": f"Bearer {os.environ['MEG_API_KEY']}"},
json={
"url": "https://lab.megalania.io/new-arrivals/",
"engine": "browser",
"actions": f"wait_for:{sys.argv[1]}",
"response_types": "json",
"extract": "items:#arrivals li{name:h2|price:.price}*",
"timeout": 20,
},
timeout=100,
)
print(r.status_code, r.headers.get("X-Route", "-"), "credits", r.headers.get("X-Credits-Charged", 0))
print(r.json().get("items", [])[:2] or r.text[:130])200 browser/datacenter credits 19
[{'name': '20 cm Chef’s Knife', 'price': '$112.00'}, {'name': 'Alpine Down Parka', 'price': '$349.00'}]wait_for:#item-12is the same wait asfix.py, run in our Chrome, and theextractrule returned name and price from each card.- 19 credits: 15 for the browser, 1 for JSON, 3 for the wait. The
’is the store's own escaping.
The same call on a page whose list never finishes loading, with a 20-second limit:
MEG_API_KEY=YOUR_KEY python fetch.py '#item-99'
422 - credits 0
{"error":{"code":"action_failed","field":"actions","message":"actions entry 0 could not run: no visible element matched its select- Our Chrome met the same missing element as
seen.py, and we answered422action_failedwhen the time for actions ran out. - Unlike Selenium's empty message, ours names the step that failed and what to change.
One honest note: our test store is fast, so these runs show a short limit or a page that never gets ready, not a slow site. On a slow site, the call is the same; the price depends on the route that worked, and our MCP tool check_site tells you that price for free.
What it costs
| What we ran | Route | Credits |
|---|---|---|
| The new arrivals list as JSON, in the browser | c:dc | 19 |
| The same page waiting for an element that never appears | c:dc | 0 |
sends.py, seen.py and fix.py, in your own Chrome | none | 0 |
Every rule is in Credits. Plans are on the pricing page.
How we measured: Selenium 4.52.0 for Python with Chrome 156 headless, Python 3.13.5, one Linux server, 17 Nov 2026. nc -lk 8799 (OpenBSD netcat) as a server that never answers. Our test store for the missing element and for fix.py. Our API on our dev cluster for fetch.py, against the same store on our dev domain. The outputs are copied from those runs.
Limits
- Public pages only. We do not log in, and we do not solve CAPTCHAs. When a page needs either, we stop and say so, for free.
- A page built by JavaScript needs
engine=browser. It costs more. See How to scrape JavaScript-rendered pages. - Slow can mean slow down. A site that answers slowly may be under load. Keep your rate low, fetch public pages only, and follow the site's terms and robots.txt.
What to do when a site answers 429 instead is in 429 Too Many Requests when scraping.