MegalaniaBlog
Blog · page 6 of 7

How to scrape it. With code that runs.

Each guide answers one question, shows the request, and says what a page costs.

· How to fixnode-fetch 403 Forbidden: why it happens, and the fix

node-fetch names itself and sorts its headers. What it and Node's own fetch send, how a 403 comes back, and the fix. node-fetch 3.3.2, Node 24.

· How to fixPython requests 403 Forbidden: why, and the fix

Python requests tells every site it is a script. What it sends, how it reports a 403, what headers fix, and a call that climbs. With requests 2.32.3.

· How to fixaxios empty response: the page loads, the list does not

axios gets the HTML, never the list a page's script draws. Why the response is empty, how to notice, and two fixes: the page's own API, or a browser.

· How to fixhttpx timeout: ReadTimeout after 5 seconds, and the fix

httpx gives up after 5 seconds by default, and its retries skip read timeouts. What it does, how it reports a timeout, and the fix. httpx 0.28.1.

· How to fixPuppeteer timeout: navigation and selector timeouts, fixed

Puppeteer stops after 30 seconds by default. What a hung server and a missing selector look like, how to catch TimeoutError, and the fix. Puppeteer 25.

· How to fixScrapy timeout: DownloadTimeoutError, retries, and the fix

Scrapy waits 180 seconds a try and retries twice. What it does on a slow server, how it reports a timeout, and the settings that fix it. Scrapy 2.18.

· AI agentsHow to crawl a website to Markdown for an LLM

Two calls turn a whole site into Markdown files for an LLM. Map the site to list its URLs, then send them as one batch. Code and costs inside.

· AI agentsExtract structured data from any web page with a JSON schema

Send a URL and a JSON Schema, get back JSON that matches it. When to use a schema, when CSS rules or embedded data are cheaper, and what each costs.

· AI agentsGive Cursor and Claude Code web access to read docs mid-task

Your coding agent guesses when it cannot read the docs. Add one MCP server, a short project rule, and it reads the current page before it writes code.

· How to scrapeHow to monitor a web page for changes with an API

Watch a page, or a list of up to 100, on a schedule. Get a signed webhook with only what changed, as a text diff or old and new values per field.

· How to scrapeHow to scrape infinite scroll and load-more pages

Infinite scroll loads items as you scroll. First find the JSON it calls. If you cannot, scroll in a real browser with page actions. Code and costs inside.

· How to fixWhy your scraper gets 403 Forbidden, and how to fix it

A 403 when scraping rarely means no access. It means the site thinks you are a bot. Four free fixes, a check for each, and when to stop.