How to scrape it. With code that runs.
Each guide answers one question, shows the request, and says what a page costs.
Read web pages in an n8n workflow as clean Markdown, with no browser to host. The Megalania node or one HTTP Request node, tested in n8n 2.47.
Connect an OpenAI Agents SDK agent to our MCP server, keep two tools, and lock what one page may cost. Python code, measured on our test store.
Give a Pydantic AI agent web pages with MCPToolset, only the tools it needs, and a per-page credit cap the model cannot change. Tested on Pydantic AI 2.68.
The same page as HTML, compact HTML, Markdown, text and CSS fields, counted with one public tokenizer. Markdown was 22 to 59 times fewer tokens.
When rotating proxies fix a block and when they do not. What a proxy pool costs to run, and one request that climbs from datacenter to residential.
What Cloudflare checks, what Python requests and curl_cffi look like on the wire, and when an API is worth it. Measured fingerprints, no promises.
Read any HTML table into CSV rows with two CSS rules, then turn product spec tables from many pages into one wide CSV. Python code and real costs.
Read every image URL and its alt text with two CSS rules, then download the files yourself, for free. Python code and real costs on our test store.
Get every page of a paginated list. Follow the next link page by page, or read the last page number and send the rest as one batch. Code, times, costs.
Read every review on a product page, its rating, date and text, with one CSS row rule. Then a whole store in one batch. Python code and real costs.
Each size and color has its own price, SKU and stock. Read every variant with one CSS rule, then a whole store in one batch. Python code and real costs.
Read sitemap.xml and sitemap indexes in Python with the standard library, for free. Or one map call when a site blocks you. Real counts on our test store.