MegalaniaBlog
Blog · page 4 of 7

How to scrape it. With code that runs.

Each guide answers one question, shows the request, and says what a page costs.

· How to scrapeWeb scraping in n8n without running a browser

Read web pages in an n8n workflow as clean Markdown, with no browser to host. The Megalania node or one HTTP Request node, tested in n8n 2.47.

· AI agentsOpenAI Agents SDK web scraping: an MCP server with a fixed cost

Connect an OpenAI Agents SDK agent to our MCP server, keep two tools, and lock what one page may cost. Python code, measured on our test store.

· AI agentsPydantic AI web scraping: MCP tools with a cap the model cannot raise

Give a Pydantic AI agent web pages with MCPToolset, only the tools it needs, and a per-page credit cap the model cannot change. Tested on Pydantic AI 2.68.

· AI agentsFeed web pages to an LLM with fewer tokens

The same page as HTML, compact HTML, Markdown, text and CSS fields, counted with one public tokenizer. Markdown was 22 to 59 times fewer tokens.

· How to fixRotating proxies for web scraping: when you need them

When rotating proxies fix a block and when they do not. What a proxy pool costs to run, and one request that climbs from datacenter to residential.

· How to scrapeScrape a site behind Cloudflare in Python: what works, what does not

What Cloudflare checks, what Python requests and curl_cffi look like on the wire, and when an API is worth it. Measured fingerprints, no promises.

· How to scrapeHow to scrape an HTML table into CSV

Read any HTML table into CSV rows with two CSS rules, then turn product spec tables from many pages into one wide CSV. Python code and real costs.

· How to scrapeHow to scrape images from a website

Read every image URL and its alt text with two CSS rules, then download the files yourself, for free. Python code and real costs on our test store.

· How to scrapeScrape multiple pages in Python, two ways

Get every page of a paginated list. Follow the next link page by page, or read the last page number and send the rest as one batch. Code, times, costs.

· How to scrapeHow to scrape product reviews into JSON

Read every review on a product page, its rating, date and text, with one CSS row rule. Then a whole store in one batch. Python code and real costs.

· How to scrapeHow to scrape product variants: price and stock for every size

Each size and color has its own price, SKU and stock. Read every variant with one CSS rule, then a whole store in one batch. Python code and real costs.

· How to scrapeHow to scrape every URL from a sitemap

Read sitemap.xml and sitemap indexes in Python with the standard library, for free. Or one map call when a site blocks you. Real counts on our test store.