How to scrape it. With code that runs.
Each guide answers one question, shows the request, and says what a page costs.
node-fetch names itself and sorts its headers. What it and Node's own fetch send, how a 403 comes back, and the fix. node-fetch 3.3.2, Node 24.
Python requests tells every site it is a script. What it sends, how it reports a 403, what headers fix, and a call that climbs. With requests 2.32.3.
axios gets the HTML, never the list a page's script draws. Why the response is empty, how to notice, and two fixes: the page's own API, or a browser.
httpx gives up after 5 seconds by default, and its retries skip read timeouts. What it does, how it reports a timeout, and the fix. httpx 0.28.1.
Puppeteer stops after 30 seconds by default. What a hung server and a missing selector look like, how to catch TimeoutError, and the fix. Puppeteer 25.
Scrapy waits 180 seconds a try and retries twice. What it does on a slow server, how it reports a timeout, and the settings that fix it. Scrapy 2.18.
Two calls turn a whole site into Markdown files for an LLM. Map the site to list its URLs, then send them as one batch. Code and costs inside.
Send a URL and a JSON Schema, get back JSON that matches it. When to use a schema, when CSS rules or embedded data are cheaper, and what each costs.
Your coding agent guesses when it cannot read the docs. Add one MCP server, a short project rule, and it reads the current page before it writes code.
Watch a page, or a list of up to 100, on a schedule. Get a signed webhook with only what changed, as a text diff or old and new values per field.
Infinite scroll loads items as you scroll. First find the JSON it calls. If you cannot, scroll in a real browser with page actions. Code and costs inside.
A 403 when scraping rarely means no access. It means the site thinks you are a bot. Four free fixes, a check for each, and when to stop.