MegalaniaBlog
  1. Blog
  2. AI agents
  3. Fresh RAG index

Keep a LlamaIndex RAG index fresh when the website changes

A daily monitor finds the pages that changed, and LlamaIndex re-embeds only those. Measured on our test store, with local embeddings and no model key.

Jordan HayesDeveloper Relations
6 min read
AI agents
Four page cards feed a vector index box. A clock tag marks a daily check. One page card, deals, glows with a changed tag, and only its arrow into the index is lit.

The short answer: to keep a LlamaIndex RAG index fresh, give each page a stable id, its URL, and update it with index.refresh_ref_docs(), which re-embeds a page only when its text changed. To learn which pages changed without re-reading them all, point a daily monitor at them. Each changed page’s new text comes from the monitor’s snapshot, so nothing is fetched twice. On our test store, lab.megalania.io, the deals page changed overnight, the monitor reported 1 changed page out of 4, and LlamaIndex re-embedded that one page.

The script

cURL
pip install llama-index-core llama-index-readers-megalania llama-index-embeddings-huggingface httpx
export MEG_API_KEY='meg_sk_...'

The embeddings run on your own machine (BAAI/bge-small-en-v1.5), so you need no model key to try this. Swap in any embedding model you use.

fresh_index.py
import json
import os
import sys

import httpx
from llama_index.core import Document, Settings, StorageContext, VectorStoreIndex, load_index_from_storage
from llama_index.embeddings.huggingface import HuggingFaceEmbedding
from llama_index.readers.megalania import MegalaniaReader
from megalania import Megalania

URLS = [
    "https://lab.megalania.io/shipping/",
    "https://lab.megalania.io/returns/",
    "https://lab.megalania.io/faq/",
    "https://lab.megalania.io/deals/",
]
API = "https://api.megalania.io"
KEY = os.environ["MEG_API_KEY"]
Settings.embed_model = HuggingFaceEmbedding(model_name="BAAI/bge-small-en-v1.5")


def read_all():
    docs = MegalaniaReader().load_data(URLS)
    print("read", len(docs), "pages, credits:", sum(d.metadata["credits_charged"] for d in docs))
    return [Document(text=d.text, id_=d.metadata["source"], metadata={"source": d.metadata["source"]}) for d in docs]


def build():
    VectorStoreIndex.from_documents(read_all()).storage_context.persist("storage")
    with Megalania(KEY) as meg:
        monitor = meg.create_monitor(name="RAG pages", urls=URLS, every="1d")
    state = {"monitor": monitor["id"], "seen": []}
    json.dump(state, open("state.json", "w"))
    print("monitor", monitor["id"])


def refresh():
    state = json.load(open("state.json"))
    mid, h = state["monitor"], {"Authorization": f"Bearer {KEY}"}
    with Megalania(KEY) as meg:
        checks = meg.list_monitor_checks(mid, outcome="changed")["checks"]
    new = [c for c in checks if c["id"] not in state["seen"]]
    positions = set()
    for c in new:
        detail = httpx.get(f"{API}/v1/monitors/{mid}/checks/{c['id']}", headers=h).json()
        positions |= {(p["position"], p["url"]) for p in detail["diff"]["pages"] if p["outcome"] == "changed"}
    docs = []
    for pos, url in sorted(positions):
        link = httpx.get(f"{API}/v1/monitors/{mid}/pages/{pos}/snapshot", headers=h).json()["download_url"]
        docs.append(Document(text=httpx.get(link).text, id_=url, metadata={"source": url}))
    state["seen"] += [c["id"] for c in new]
    json.dump(state, open("state.json", "w"))
    print(len(new), "changed checks:", [u for _, u in sorted(positions)])
    update(docs)


def update(docs):
    index = load_index_from_storage(StorageContext.from_defaults(persist_dir="storage"))
    updated = index.refresh_ref_docs(docs)
    index.storage_context.persist("storage")
    print("re-embedded:", [d.id_ for d, u in zip(docs, updated) if u])


def ask(question):
    index = load_index_from_storage(StorageContext.from_defaults(persist_dir="storage"))
    for hit in index.as_retriever(similarity_top_k=1).retrieve(question):
        prices = [l.strip() for l in hit.text.splitlines() if "Current price" in l]
        print(round(hit.score, 3), hit.node.ref_doc_id, "|", prices[:2])


{"build": build, "refresh": refresh, "reread": lambda: update(read_all()), "ask": lambda: ask(sys.argv[2])}[sys.argv[1]]()
  • build reads the 4 pages as Markdown, embeds them, saves the index in ./storage, and creates a monitor that checks the same pages once a day. Its first check runs at once and becomes the baseline.
  • refresh asks the monitor for checks with outcome=changed that it has not seen yet, takes the pages that changed, downloads each one’s latest snapshot, and hands them to refresh_ref_docs.
  • reread is the other way, with no monitor: read every page again and let refresh_ref_docs skip the ones that did not change.
  • ask runs a retriever over the index, with no model, and prints the top page with its first two prices.

Day one: build, and nothing to refresh

Output, 16 Nov 2026
$ python fresh_index.py build
read 4 pages, credits: 4
monitor 01a1e5e9-811b-7b01-a497-78003c23e75d
$ python fresh_index.py reread
read 4 pages, credits: 4
re-embedded: []
$ python fresh_index.py refresh
0 changed checks: []
re-embedded: []

The same 4 pages, read again a minute later, re-embedded nothing: the text had not changed, so refresh_ref_docs kept the stored vectors.

Free beta, by invite76,000 credits for 30 days, no card. Apply, and we email your invite.Apply for the beta

Day two: one page changed

Our test store’s deals page gets new prices every night. The next morning, the monitor’s check said changed for 1 page out of 4, for 8 credits. Then:

Output, 17 Nov 2026
$ python fresh_index.py refresh
1 changed checks: ['https://lab.megalania.io/deals/']
re-embedded: ['https://lab.megalania.io/deals/']
$ python fresh_index.py ask "Which products are on sale today?"
0.595 https://lab.megalania.io/deals/ | ['Sale! Original price was: $34.00.Current price is: $26.95.', 'Sale! Original price was: $58.00.Current price is: $48.95.']
$ python fresh_index.py reread
read 4 pages, credits: 4
re-embedded: []
$ python fresh_index.py refresh
0 changed checks: []
re-embedded: []
  • Only the deals page was re-embedded. Shipping, returns and the FAQ kept their vectors.
  • The answer has today’s prices. The retriever found the deals page and its new prices, which were not on the page the day before.
  • The snapshot was free. Downloading it three times left our balance where it was: the snapshot is the text the monitor already fetched and checked. It is the same Markdown our reader returns.
  • reread agreed. Reading all 4 pages again after the refresh re-embedded nothing, because the index already had the new deals text.

The one thing that breaks refresh_ref_docs

LlamaIndex decides “changed” from a hash of the document’s text and its metadata. Our reader puts request_id and credits_charged in the metadata, and request_id is new on every read. So with the reader’s documents as they come, every refresh re-embeds every page. We measured it: the same 4 unchanged pages, twice, re-embedded all 4 both times.

The fix is in read_all(): build each Document with id_ set to the URL and only source in its metadata. Then the hash covers the text, and unchanged pages are skipped. Give the snapshot documents in refresh() the same shape, so both ways agree.

Monitor or re-read: what each costs

Step Credits
build: 4 plain pages as Markdown 4
A monitor check of 4 pages: each page’s price, plus 1 per page 8
Downloading a changed page’s snapshot 0
reread: all 4 pages again 4
A failed or blocked check 0

So for these 4 pages, a daily monitor costs about 240 credits a month and a daily reread from your own cron about 120. Re-reading is cheaper. The monitor costs 1 credit more per page, and for that:

  • We run the schedule. No cron, no server awake at 3 a.m.; you call refresh when you need the index.
  • Ignore rules. A page with a date or a visitor counter changes every day. A monitor’s ignore regex drops that text before it compares, so the page does not count as changed. Re-reading would re-embed it daily.
  • A webhook. Give the monitor a webhook_url, and it calls you with the changed pages, instead of you asking.
  • A diff per page, kept 30 days, to see what changed before you trust the new text.

For a few pages that change often, re-read. For many pages that rarely change, or pages with noise to ignore, use a monitor. All the rules are in Credits and Watch pages with monitors. Plans are on the pricing page.

How we measured: our dev cluster, the Lab store on our dev domain, Python 3.13, llama-index-core 0.14.26, llama-index-readers-megalania 0.1.1, llama-index-embeddings-huggingface 0.9.0 with BAAI/bge-small-en-v1.5 on CPU, megalania 0.3.0 and httpx 0.28.1. We ran build on 16 Nov 2026, then refresh the next morning after the deals page changed; we started that day’s check with the monitor’s run-now call instead of waiting for its time. The outputs are copied from those runs.

Limits

  • Monitors run at most once an hour on the beta plan, and a list holds up to 100 pages. For a whole site, make several monitors, or re-read on your own schedule.
  • Removed pages stay in the index. This script adds and updates; when you drop a URL, call index.delete_ref_doc(url) too.
  • Our Python SDK has no call for a check’s detail or a page snapshot yet, so the script uses two plain GETs with httpx.
  • Public pages only. We do not log in, and we do not solve CAPTCHAs. Read each site’s terms and robots.txt, and follow them.

More on loading pages into LlamaIndex, and on agent tools, is in LlamaIndex web scraping.

Jordan HayesDeveloper RelationsPublished 17 Nov 2026 · measured on our dev cluster, 16 and 17 Nov 2026, llama-index-core 0.14.26, llama-index-readers-megalania 0.1.1, local bge-small embeddings, 4 pages of our test store

See also

Keep reading