https://megalania.io/blog/How to scrape the web, with code that runs and what each page costs.enhttps://megalania.io/blog/crawl-website-to-markdown/https://megalania.io/blog/crawl-website-to-markdown/Wed, 21 Oct 2026 00:00:00 GMTTwo calls turn a whole site into Markdown files for an LLM. Map the site to list its URLs, then send them as one batch. Code and costs inside.support@megalania.io (Jordan Hayes)https://megalania.io/blog/extract-structured-data-llm/https://megalania.io/blog/extract-structured-data-llm/Wed, 21 Oct 2026 00:00:00 GMTSend a URL and a JSON Schema, get back JSON that matches it. When to use a schema, when CSS rules or embedded data are cheaper, and what each costs.support@megalania.io (Jordan Hayes)https://megalania.io/blog/mcp-web-access-coding-agents/https://megalania.io/blog/mcp-web-access-coding-agents/Wed, 21 Oct 2026 00:00:00 GMTYour coding agent guesses when it cannot read the docs. Add one MCP server, a short project rule, and it reads the current page before it writes code.support@megalania.io (Jordan Hayes)https://megalania.io/blog/monitor-website-changes/https://megalania.io/blog/monitor-website-changes/Wed, 21 Oct 2026 00:00:00 GMTWatch a page, or a list of up to 100, on a schedule. Get a signed webhook with only what changed, as a text diff or old and new values per field.support@megalania.io (Jordan Hayes)https://megalania.io/blog/scrape-infinite-scroll/https://megalania.io/blog/scrape-infinite-scroll/Wed, 21 Oct 2026 00:00:00 GMTInfinite scroll loads items as you scroll. First find the JSON it calls. If you cannot, scroll in a real browser with page actions. Code and costs inside.support@megalania.io (Jordan Hayes)https://megalania.io/blog/scraper-403-forbidden/https://megalania.io/blog/scraper-403-forbidden/Wed, 21 Oct 2026 00:00:00 GMTA 403 when scraping rarely means no access. It means the site thinks you are a bot. Four free fixes, a check for each, and when to stop.support@megalania.io (Jordan Hayes)https://megalania.io/blog/docs-site-to-rag/https://megalania.io/blog/docs-site-to-rag/Sun, 18 Oct 2026 00:00:00 GMTTurn a documentation site into clean Markdown files for RAG. Check llms.txt, read the sitemap, fetch each page as Markdown, and keep code blocks intact.support@megalania.io (Jordan Hayes)https://megalania.io/blog/how-to-scrape-shopify-store/https://megalania.io/blog/how-to-scrape-shopify-store/Sun, 18 Oct 2026 00:00:00 GMTGet title, price, currency, stock and SKU from any Shopify product page as JSON. The free products.json route first, then one request in cURL or Python.support@megalania.io (Jordan Hayes)https://megalania.io/blog/javascript-pages/https://megalania.io/blog/javascript-pages/Sun, 18 Oct 2026 00:00:00 GMTFirst check if you need a browser at all. A three-request test, then one render with engine browser when you must. Real page sizes and costs inside.support@megalania.io (Jordan Hayes)https://megalania.io/blog/job-postings/https://megalania.io/blog/job-postings/Sun, 18 Oct 2026 00:00:00 GMTMost career pages carry a JobPosting JSON-LD block. Read title, location, type, date and company in one plain HTTP request, no browser. Real costs inside.support@megalania.io (Jordan Hayes)https://megalania.io/blog/mcp-server-ai-agents/https://megalania.io/blog/mcp-server-ai-agents/Sun, 18 Oct 2026 00:00:00 GMTAdd one MCP server and your agent can check a site for free, then fetch any public page as clean Markdown. One setup command, real page sizes and costs.support@megalania.io (Jordan Hayes)https://megalania.io/blog/product-data-json-ld/https://megalania.io/blog/product-data-json-ld/Sun, 18 Oct 2026 00:00:00 GMTMost stores publish name, SKU, price and stock as JSON-LD for search engines. Read it on any platform with one request and no selectors, in cURL or Python.support@megalania.io (Jordan Hayes)https://megalania.io/blog/track-prices-stock/https://megalania.io/blog/track-prices-stock/Sun, 18 Oct 2026 00:00:00 GMTA daily price and stock tracker in one Python file. Fetch each product page as JSON, save a snapshot, diff it with yesterday's. Real page costs inside.support@megalania.io (Jordan Hayes)https://megalania.io/blog/web-page-to-markdown/https://megalania.io/blog/web-page-to-markdown/Sun, 18 Oct 2026 00:00:00 GMTSend one request with response_types markdown and get the page's main content back as clean Markdown, menus and footers gone. Real token counts inside.support@megalania.io (Jordan Hayes)https://megalania.io/blog/woocommerce/https://megalania.io/blog/woocommerce/Sun, 18 Oct 2026 00:00:00 GMTGet name, price, stock and SKU from WooCommerce product pages. Try the free Store API first, then one request in cURL or Python, with real page costs.support@megalania.io (Jordan Hayes)