How to scrape it. With code that runs.
Each guide answers one question, shows the request, and says what a page costs.
· AI agentsHow to crawl a website to Markdown for an LLMTwo calls turn a whole site into Markdown files for an LLM. Map the site to list its URLs, then send them as one batch. Code and costs inside.
· AI agentsExtract structured data from any web page with a JSON schemaSend a URL and a JSON Schema, get back JSON that matches it. When to use a schema, when CSS rules or embedded data are cheaper, and what each costs.
· AI agentsGive Cursor and Claude Code web access to read docs mid-taskYour coding agent guesses when it cannot read the docs. Add one MCP server, a short project rule, and it reads the current page before it writes code.
· How to scrapeHow to monitor a web page for changes with an APIWatch a page, or a list of up to 100, on a schedule. Get a signed webhook with only what changed, as a text diff or old and new values per field.
· How to scrapeHow to scrape infinite scroll and load-more pagesInfinite scroll loads items as you scroll. First find the JSON it calls. If you cannot, scroll in a real browser with page actions. Code and costs inside.
· How to fixWhy your scraper gets 403 Forbidden, and how to fix itA 403 when scraping rarely means no access. It means the site thinks you are a bot. Four free fixes, a check for each, and when to stop.
· AI agentsCrawl a docs site into Markdown for RAGTurn a documentation site into clean Markdown files for RAG. Check llms.txt, read the sitemap, fetch each page as Markdown, and keep code blocks intact.
· How to scrapeHow to scrape any Shopify storeGet title, price, currency, stock and SKU from any Shopify product page as JSON. The free products.json route first, then one request in cURL or Python.
· How to scrapeHow to scrape JavaScript-rendered pagesFirst check if you need a browser at all. A three-request test, then one render with engine browser when you must. Real page sizes and costs inside.
· How to scrapeHow to scrape job postings from company career pagesMost career pages carry a JobPosting JSON-LD block. Read title, location, type, date and company in one plain HTTP request, no browser. Real costs inside.
· AI agentsGive your AI agent web access with an MCP serverAdd one MCP server and your agent can check a site for free, then fetch any public page as clean Markdown. One setup command, real page sizes and costs.
· How to scrapeHow to scrape product data from any storeMost stores publish name, SKU, price and stock as JSON-LD for search engines. Read it on any platform with one request and no selectors, in cURL or Python.
· How to scrapeHow to track competitor prices and stock every dayA daily price and stock tracker in one Python file. Fetch each product page as JSON, save a snapshot, diff it with yesterday's. Real page costs inside.
· AI agentsTurn any web page into Markdown for an LLMSend one request with response_types markdown and get the page's main content back as clean Markdown, menus and footers gone. Real token counts inside.
· How to scrapeHow to scrape any WooCommerce storeGet name, price, stock and SKU from WooCommerce product pages. Try the free Store API first, then one request in cURL or Python, with real page costs.