MegalaniaBlog
Blog · page 2 of 7

How to scrape it. With code that runs.

Each guide answers one question, shows the request, and says what a page costs.

· How to fixaiohttp timeout: the 5-minute default, and how to set one

aiohttp waits up to 300 seconds a request and raises a TimeoutError with no message. What ClientTimeout sets, SocketTimeoutError, and a retry.

· How to fixaxios timeout: no default, ECONNABORTED, and the fix

axios waits forever unless you set timeout. What a hung server does, ECONNABORTED versus ETIMEDOUT, a retry loop, and a call that climbs. axios 1.21.

· How to fixcurl timeout: error 28, and the options that fix it

curl waits forever by default and exits 28 when a limit runs out. What --max-time and --connect-timeout do, how --retry works, and our API.

· How to fixn8n HTTP Request timeout: the 5-minute default, and the fix

The n8n HTTP Request node waits 5 minutes and blames a server that is offline. What the error means, Timeout and Retry on Fail, and our API.

· How to fixNode fetch timeout: 300 seconds, then fetch failed

Node's fetch has no timeout option. It waited 301 seconds, then failed with UND_ERR_HEADERS_TIMEOUT. AbortSignal.timeout, a retry loop, and our API.

· How to fixPlaywright timeout: Timeout 30000ms exceeded, and the fix

Playwright waits 30 seconds for a page and for each element. What a hung server and a missing element look like, the timeout settings, and our API.

· How to fixPython requests timeout: no default, and how to set one

requests waits forever unless you pass timeout. What a hung server does, ReadTimeout and ConnectTimeout, a Retry setup, and a call that climbs.

· How to fixSelenium TimeoutException: page loads and waits, and the fix

Selenium waits 300 seconds for a page and raises TimeoutException with an empty message on a wait. What each one means, the settings, and our API.

· How to fix429 Too Many Requests when scraping: what to do

What a 429 means when you scrape, how to read Retry-After and back off in Python, and how to pace a run under the limit. Measured on our test store.

· AI agentsScrape product prices with an AI agent

Ask Claude, Cursor or ChatGPT for every price on a store page. One MCP server, a free site check, then one extract call returns old and new prices.

· AI agentsClaude Code WebFetch gets a 403 or an empty page: the fix

When WebFetch in Claude Code hits a 403, a bot check or a JavaScript page, add one MCP server and one rule. The agent reads the page instead of guessing.

· AI agentsCrewAI web scraping: give a crew tools that read pages

Connect a CrewAI crew to our MCP server with one config, keep two tools, and lock the cost of each page. Python code, measured on our test store.