MegalaniaBlog
  1. Blog
  2. AI agents
  3. Fewer tokens

Feed web pages to an LLM with fewer tokens

The same page as HTML, compact HTML, Markdown, text and CSS fields, counted with one public tokenizer. Markdown was 22 to 59 times fewer tokens.

Jordan HayesDeveloper Relations
6 min read
AI agents
A tall HTML page card marked 21,268 tokens goes into a funnel. A short Markdown card marked 962 and a small JSON card marked 439 come out.

The short answer: to reduce LLM tokens in web scraping, do not hand the model the page’s HTML. Ask for the shape the job needs. On our test store, lab.megalania.io, the kitchen category page was 21,268 tokens of HTML and 962 tokens of Markdown, with every product name, price, rating and link still in it. Its nine products as JSON fields were 439 tokens. The FAQ page went from 16,602 tokens to 280. Every shape costs the same 1 credit for the page; JSON fields add 1.

The free way first

If a plain request gets the page, you can cut it down yourself. Fetch the HTML, drop script and style, and run an open-source HTML-to-Markdown converter or a readability-style extractor on your machine. It is free. Turn any web page into Markdown for an LLM covers that way and when it stops being enough.

You own the tuning: which part of each site is “main content”. And you get nothing from a site that blocks plain requests.

Six shapes of the same page

One script, two Lab pages, six shapes each. It counts characters and tokens for every answer:

cURL
pip install requests tiktoken
export MEG_API_KEY='meg_sk_...'
tokens.py
import os

import requests
import tiktoken

enc = tiktoken.get_encoding("o200k_base")
KEY = os.environ["MEG_API_KEY"]

SHAPES = {
    "html": {"response_types": "html"},
    "compact_html": {"response_types": "compact_html"},
    "markdown, whole page": {"response_types": "markdown", "only_main_content": "false"},
    "markdown": {"response_types": "markdown"},
    "text": {"response_types": "text"},
}
RULES = {
    "https://lab.megalania.io/product-category/kitchen/":
        "items:li.product{name:h2|price:.price > .amount|was:.price del .amount|now:.price ins .amount|rating:.rating|url:a.woocommerce-loop-product__link@href}*",
    "https://lab.megalania.io/faq/": "faq:details{q:summary|a:p}*",
}

for url, rule in RULES.items():
    shapes = {**SHAPES, "extract (json)": {"response_types": "json", "extract": rule}}
    print(url)
    for name, params in shapes.items():
        r = requests.get(
            "https://get.megalania.io/v1",
            params={"url": url, **params},
            headers={"Authorization": f"Bearer {KEY}"},
            timeout=120,
        )
        r.raise_for_status()
        print(f"  {name:22} {len(r.text):>7,} chars {len(enc.encode(r.text)):>7,} tokens")
Output
https://lab.megalania.io/product-category/kitchen/
  html                    72,953 chars  21,268 tokens
  compact_html            11,600 chars   3,113 tokens
  markdown, whole page     6,512 chars   1,929 tokens
  markdown                 3,042 chars     962 tokens
  text                       833 chars     296 tokens
  extract (json)           1,364 chars     439 tokens
https://lab.megalania.io/faq/
  html                    56,038 chars  16,602 tokens
  compact_html             1,795 chars     502 tokens
  markdown, whole page     4,433 chars   1,210 tokens
  markdown                 1,127 chars     280 tokens
  text                     1,050 chars     255 tokens
  extract (json)           1,156 chars     309 tokens
  • html is the page as the site sent it: head, menus, scripts, styles and all.
  • compact_html is the HTML without scripts, styles, hidden parts and most attributes. id and class stay, so CSS selectors still match.
  • markdown is the page’s main content. only_main_content=false keeps the menu, header and footer too.
  • text is the Markdown with no markup: no links, no headings.
  • extract (json) is only the fields your CSS rules name.
Free beta, by invite76,000 credits for 30 days, no card. Apply, and we email your invite.Apply for the beta

Reduce LLM tokens in web scraping: pick the shape by the job

The model has to Send Kitchen page FAQ page
Read and answer questions markdown 962 280
Fill rows: name, price, link extract with json 439 309
Write or test CSS selectors compact_html 3,113 502
See the menu or footer too markdown with only_main_content=false 1,929 1,210
Nothing: you parse it yourself html 21,268 16,602

Tokens, counted with tiktoken o200k_base.

  • Markdown is the default for a model that reads. 22 times fewer tokens on the list page, 59 times fewer on the FAQ. The links stay, so the model can tell you which product it means.
  • Fields win on a list, not on prose. On the kitchen page, nine products as JSON were less than half the Markdown. On the FAQ page, JSON was more: 309 tokens against 280. The keys and quotes cost tokens, and the Markdown was already only the answers.
  • text is the smallest, and it drops things. It has no links. On a sale, the old and new price sit side by side with no strike marker: the kitchen page’s text says “Original price was: $44.00.Current price is: $36.00”. Use it only when the words are all you need.
  • The whole page doubles the count or worse. 1,929 against 962 on the kitchen page, 1,210 against 280 on the FAQ. Send only_main_content=false only when a price or a spec table sits outside the main content.

Fields, not pages

On a list page, the model often needs rows, not prose. CSS rules in extract read them before the model sees anything:

cURL
curl -s -H "Authorization: Bearer $MEG_API_KEY" -G "https://get.megalania.io/v1" \
  --data-urlencode "url=https://lab.megalania.io/product-category/kitchen/" \
  --data-urlencode "response_types=json" \
  --data-urlencode "extract=items:li.product{name:h2|price:.price > .amount|was:.price del .amount|now:.price ins .amount|rating:.rating|url:a.woocommerce-loop-product__link@href}*"
Output (first and eighth of 9 items)
{"items":[{"name":"20 cm Chef’s Knife","price":"$112.00","was":null,"now":null,"rating":"4.67","url":"https://lab.megalania.io/product/chef-knife/"},
 {"name":"Porcelain Teapot with Oak Handle","price":null,"was":"$44.00","now":"$36.00","rating":"4.50","url":"https://lab.megalania.io/product/porcelain-teapot/"}]}

The sale is clear here: was and now. Every field that is not on a card is null, so rows never shift. Drop the fields you do not need and the count falls again.

Rules need a stable layout, and you write them once per site. Extract structured data from any web page with a JSON schema shows the other way, where a model fills your JSON Schema. That way is off on our servers in the first beta days; CSS rules work now.

How we counted

We counted every answer with tiktoken 0.14.0 and its o200k_base encoding, the full body as the API returned it. Another model’s tokenizer gives other numbers, but the order of the shapes stays the same.

If you have no tokenizer, characters divided by 4 is a fair guess for prose: the FAQ Markdown is 1,127 characters, about 281 tokens, and tiktoken counted 280. It is too low for HTML: the kitchen page’s 72,953 characters guess 18,238 tokens; tiktoken counted 21,268.

What it costs

Step Credits
One page as html, compact_html, markdown or text 1
The same page as json with extract rules 2
Several shapes in one request, from one read of the page 1, plus 1 if json is one of them
A blocked page or one of our errors 0

These prices are for this site on the datacenter route. A site that blocks it costs more. Plans are on the pricing page.

How we measured: our dev cluster on 16 Nov 2026, the Lab store on our dev domain, Python 3.13, requests 2.34.2 and tiktoken 0.14.0. We ran tokens.py twice and the cURL call once. The credits above are from first reads. The same request again within 5 minutes can come from our cache, with X-Cache: hit, for the cache price.

Limits

  • Main content is our guess. On most pages it is right. When a price or a table sits outside it, use only_main_content=false.
  • Rules break when the site changes its layout. You get null fields, not an error. Check a few rows after a site redesign.
  • Very large pages can be a free 422 convert_cap for the converted shapes. html has no such limit.
  • Page content is untrusted. Any shape can carry text a page author hid for a model. Do not give the same agent tools that send mail or spend money.

Every shape and every rule form is in Fetch, extract and lists. Fetch public pages only. Read the site’s terms and robots.txt, and follow them.

Jordan HayesDeveloper RelationsPublished 16 Nov 2026 · measured on our dev cluster, 16 Nov 2026, 2 pages on our test store in 6 shapes, counted with tiktoken o200k_base

See also

Keep reading