MegalaniaBlog
Blog · page 5 of 7

How to scrape it. With code that runs.

Each guide answers one question, shows the request, and says what a page costs.

· How to scrapeScrape a website from another country: country_code and X-Geo

Make a request leave from Germany, Japan or Brazil with one parameter, check where it left from, and see what each costs. Measured on our test store.

· How to scrapeHow to scrape a website to CSV

Turn a product list into a CSV file. One request per page with CSS rules returns JSON, a short Python loop follows the next link and writes the rows.

· How to scrapeScrapy JavaScript rendering without a browser on your side

Keep your Scrapy spider and its selectors. Send the pages that need JavaScript through one API call and get rendered HTML back. Measured on our test store.

· AI agentsGive a Vercel AI SDK agent web access

Give a Vercel AI SDK agent tools that read web pages as Markdown or JSON, with a credit cap the model cannot raise. Measured on our test store.

· How to scrapeWeb scraping from the command line: curl, jq and one CLI

Scrape pages, list a site's URLs and run a batch to CSV from a shell, with curl or one small CLI and jq. No script to write. Measured on our test store.

· How to scrapeWeb scraping in Node.js with plain fetch, no headless browser

Read pages from a Node.js or TypeScript service with built-in fetch. Markdown, JSON fields and a batch of 8 pages, with no browser to run.

· How to scrapeWeb scraping with Python requests: when a plain GET is not enough

Keep the requests library you know. Send the same call to one API and get Markdown, JSON fields and JavaScript pages. Measured on our test store.

· How to scrapeWeb scraping webhooks: get told when a batch is done

Stop polling. Send a batch with a webhook URL, check the signed call in 40 lines of Python, and read the results when it arrives. Retries measured on dev.

· How to scrapeTurn a website into a JSON API

A small Python server of your own that answers with a site's products as JSON. Rules from inspect, fields from extract, and max_age for a cheap cache.

· How to scrapeWooCommerce products as JSON: the whole catalog from the Store API

Get every product of a WooCommerce store as JSON from its public Store API, page by page. Free first, then through our API for 1 credit a call.

· How to fixaxios 403 Forbidden: why it happens in Node, and the fix

axios in Node sends an API client's headers, one of them a format no browser offers. What it sends, how it throws on a 403, and the fix. axios 1.21.1.

· How to fixcurl 403 Forbidden: why, and how to fix it

curl sends three headers and exits 0 on a 403. What it sends, how to make it fail, which flags fix it, and a call that climbs. Tested with curl 8.14.1.