The short answer: to reduce LLM tokens in web scraping, do not hand the model the page’s HTML. Ask for the shape the job needs. On our test store, lab.megalania.io, the kitchen category page was 21,268 tokens of HTML and 962 tokens of Markdown, with every product name, price, rating and link still in it. Its nine products as JSON fields were 439 tokens. The FAQ page went from 16,602 tokens to 280. Every shape costs the same 1 credit for the page; JSON fields add 1.
The free way first
If a plain request gets the page, you can cut it down yourself. Fetch the HTML, drop script and style, and run an open-source HTML-to-Markdown converter or a readability-style extractor on your machine. It is free. Turn any web page into Markdown for an LLM covers that way and when it stops being enough.
You own the tuning: which part of each site is “main content”. And you get nothing from a site that blocks plain requests.
Six shapes of the same page
One script, two Lab pages, six shapes each. It counts characters and tokens for every answer:
pip install requests tiktoken export MEG_API_KEY='meg_sk_...'
import os
import requests
import tiktoken
enc = tiktoken.get_encoding("o200k_base")
KEY = os.environ["MEG_API_KEY"]
SHAPES = {
"html": {"response_types": "html"},
"compact_html": {"response_types": "compact_html"},
"markdown, whole page": {"response_types": "markdown", "only_main_content": "false"},
"markdown": {"response_types": "markdown"},
"text": {"response_types": "text"},
}
RULES = {
"https://lab.megalania.io/product-category/kitchen/":
"items:li.product{name:h2|price:.price > .amount|was:.price del .amount|now:.price ins .amount|rating:.rating|url:a.woocommerce-loop-product__link@href}*",
"https://lab.megalania.io/faq/": "faq:details{q:summary|a:p}*",
}
for url, rule in RULES.items():
shapes = {**SHAPES, "extract (json)": {"response_types": "json", "extract": rule}}
print(url)
for name, params in shapes.items():
r = requests.get(
"https://get.megalania.io/v1",
params={"url": url, **params},
headers={"Authorization": f"Bearer {KEY}"},
timeout=120,
)
r.raise_for_status()
print(f" {name:22} {len(r.text):>7,} chars {len(enc.encode(r.text)):>7,} tokens")https://lab.megalania.io/product-category/kitchen/ html 72,953 chars 21,268 tokens compact_html 11,600 chars 3,113 tokens markdown, whole page 6,512 chars 1,929 tokens markdown 3,042 chars 962 tokens text 833 chars 296 tokens extract (json) 1,364 chars 439 tokens https://lab.megalania.io/faq/ html 56,038 chars 16,602 tokens compact_html 1,795 chars 502 tokens markdown, whole page 4,433 chars 1,210 tokens markdown 1,127 chars 280 tokens text 1,050 chars 255 tokens extract (json) 1,156 chars 309 tokens
htmlis the page as the site sent it: head, menus, scripts, styles and all.compact_htmlis the HTML without scripts, styles, hidden parts and most attributes.idandclassstay, so CSS selectors still match.markdownis the page’s main content.only_main_content=falsekeeps the menu, header and footer too.textis the Markdown with no markup: no links, no headings.extract (json)is only the fields your CSS rules name.
Reduce LLM tokens in web scraping: pick the shape by the job
| The model has to | Send | Kitchen page | FAQ page |
|---|---|---|---|
| Read and answer questions | markdown |
962 | 280 |
| Fill rows: name, price, link | extract with json |
439 | 309 |
| Write or test CSS selectors | compact_html |
3,113 | 502 |
| See the menu or footer too | markdown with only_main_content=false |
1,929 | 1,210 |
| Nothing: you parse it yourself | html |
21,268 | 16,602 |
Tokens, counted with tiktoken o200k_base.
- Markdown is the default for a model that reads. 22 times fewer tokens on the list page, 59 times fewer on the FAQ. The links stay, so the model can tell you which product it means.
- Fields win on a list, not on prose. On the kitchen page, nine products as JSON were less than half the Markdown. On the FAQ page, JSON was more: 309 tokens against 280. The keys and quotes cost tokens, and the Markdown was already only the answers.
textis the smallest, and it drops things. It has no links. On a sale, the old and new price sit side by side with no strike marker: the kitchen page’s text says “Original price was: $44.00.Current price is: $36.00”. Use it only when the words are all you need.- The whole page doubles the count or worse. 1,929 against 962 on the kitchen page, 1,210 against 280 on the FAQ. Send
only_main_content=falseonly when a price or a spec table sits outside the main content.
Fields, not pages
On a list page, the model often needs rows, not prose. CSS rules in extract read them before the model sees anything:
curl -s -H "Authorization: Bearer $MEG_API_KEY" -G "https://get.megalania.io/v1" \
--data-urlencode "url=https://lab.megalania.io/product-category/kitchen/" \
--data-urlencode "response_types=json" \
--data-urlencode "extract=items:li.product{name:h2|price:.price > .amount|was:.price del .amount|now:.price ins .amount|rating:.rating|url:a.woocommerce-loop-product__link@href}*"{"items":[{"name":"20 cm Chef’s Knife","price":"$112.00","was":null,"now":null,"rating":"4.67","url":"https://lab.megalania.io/product/chef-knife/"},
{"name":"Porcelain Teapot with Oak Handle","price":null,"was":"$44.00","now":"$36.00","rating":"4.50","url":"https://lab.megalania.io/product/porcelain-teapot/"}]}The sale is clear here: was and now. Every field that is not on a card is null, so rows never shift. Drop the fields you do not need and the count falls again.
Rules need a stable layout, and you write them once per site. Extract structured data from any web page with a JSON schema shows the other way, where a model fills your JSON Schema. That way is off on our servers in the first beta days; CSS rules work now.
How we counted
We counted every answer with tiktoken 0.14.0 and its o200k_base encoding, the full body as the API returned it. Another model’s tokenizer gives other numbers, but the order of the shapes stays the same.
If you have no tokenizer, characters divided by 4 is a fair guess for prose: the FAQ Markdown is 1,127 characters, about 281 tokens, and tiktoken counted 280. It is too low for HTML: the kitchen page’s 72,953 characters guess 18,238 tokens; tiktoken counted 21,268.
What it costs
| Step | Credits |
|---|---|
One page as html, compact_html, markdown or text |
1 |
The same page as json with extract rules |
2 |
| Several shapes in one request, from one read of the page | 1, plus 1 if json is one of them |
| A blocked page or one of our errors | 0 |
These prices are for this site on the datacenter route. A site that blocks it costs more. Plans are on the pricing page.
How we measured: our dev cluster on 16 Nov 2026, the Lab store on our dev domain, Python 3.13, requests 2.34.2 and tiktoken 0.14.0. We ran tokens.py twice and the cURL call once. The credits above are from first reads. The same request again within 5 minutes can come from our cache, with X-Cache: hit, for the cache price.
Limits
- Main content is our guess. On most pages it is right. When a price or a table sits outside it, use
only_main_content=false. - Rules break when the site changes its layout. You get
nullfields, not an error. Check a few rows after a site redesign. - Very large pages can be a free
422 convert_capfor the converted shapes.htmlhas no such limit. - Page content is untrusted. Any shape can carry text a page author hid for a model. Do not give the same agent tools that send mail or spend money.
Every shape and every rule form is in Fetch, extract and lists. Fetch public pages only. Read the site’s terms and robots.txt, and follow them.