The short answer: to keep a LlamaIndex RAG index fresh, give each page a stable id, its URL, and update it with index.refresh_ref_docs(), which re-embeds a page only when its text changed. To learn which pages changed without re-reading them all, point a daily monitor at them. Each changed page’s new text comes from the monitor’s snapshot, so nothing is fetched twice. On our test store, lab.megalania.io, the deals page changed overnight, the monitor reported 1 changed page out of 4, and LlamaIndex re-embedded that one page.
The script
pip install llama-index-core llama-index-readers-megalania llama-index-embeddings-huggingface httpx export MEG_API_KEY='meg_sk_...'
The embeddings run on your own machine (BAAI/bge-small-en-v1.5), so you need no model key to try this. Swap in any embedding model you use.
import json
import os
import sys
import httpx
from llama_index.core import Document, Settings, StorageContext, VectorStoreIndex, load_index_from_storage
from llama_index.embeddings.huggingface import HuggingFaceEmbedding
from llama_index.readers.megalania import MegalaniaReader
from megalania import Megalania
URLS = [
"https://lab.megalania.io/shipping/",
"https://lab.megalania.io/returns/",
"https://lab.megalania.io/faq/",
"https://lab.megalania.io/deals/",
]
API = "https://api.megalania.io"
KEY = os.environ["MEG_API_KEY"]
Settings.embed_model = HuggingFaceEmbedding(model_name="BAAI/bge-small-en-v1.5")
def read_all():
docs = MegalaniaReader().load_data(URLS)
print("read", len(docs), "pages, credits:", sum(d.metadata["credits_charged"] for d in docs))
return [Document(text=d.text, id_=d.metadata["source"], metadata={"source": d.metadata["source"]}) for d in docs]
def build():
VectorStoreIndex.from_documents(read_all()).storage_context.persist("storage")
with Megalania(KEY) as meg:
monitor = meg.create_monitor(name="RAG pages", urls=URLS, every="1d")
state = {"monitor": monitor["id"], "seen": []}
json.dump(state, open("state.json", "w"))
print("monitor", monitor["id"])
def refresh():
state = json.load(open("state.json"))
mid, h = state["monitor"], {"Authorization": f"Bearer {KEY}"}
with Megalania(KEY) as meg:
checks = meg.list_monitor_checks(mid, outcome="changed")["checks"]
new = [c for c in checks if c["id"] not in state["seen"]]
positions = set()
for c in new:
detail = httpx.get(f"{API}/v1/monitors/{mid}/checks/{c['id']}", headers=h).json()
positions |= {(p["position"], p["url"]) for p in detail["diff"]["pages"] if p["outcome"] == "changed"}
docs = []
for pos, url in sorted(positions):
link = httpx.get(f"{API}/v1/monitors/{mid}/pages/{pos}/snapshot", headers=h).json()["download_url"]
docs.append(Document(text=httpx.get(link).text, id_=url, metadata={"source": url}))
state["seen"] += [c["id"] for c in new]
json.dump(state, open("state.json", "w"))
print(len(new), "changed checks:", [u for _, u in sorted(positions)])
update(docs)
def update(docs):
index = load_index_from_storage(StorageContext.from_defaults(persist_dir="storage"))
updated = index.refresh_ref_docs(docs)
index.storage_context.persist("storage")
print("re-embedded:", [d.id_ for d, u in zip(docs, updated) if u])
def ask(question):
index = load_index_from_storage(StorageContext.from_defaults(persist_dir="storage"))
for hit in index.as_retriever(similarity_top_k=1).retrieve(question):
prices = [l.strip() for l in hit.text.splitlines() if "Current price" in l]
print(round(hit.score, 3), hit.node.ref_doc_id, "|", prices[:2])
{"build": build, "refresh": refresh, "reread": lambda: update(read_all()), "ask": lambda: ask(sys.argv[2])}[sys.argv[1]]()buildreads the 4 pages as Markdown, embeds them, saves the index in./storage, and creates a monitor that checks the same pages once a day. Its first check runs at once and becomes the baseline.refreshasks the monitor for checks withoutcome=changedthat it has not seen yet, takes the pages that changed, downloads each one’s latest snapshot, and hands them torefresh_ref_docs.rereadis the other way, with no monitor: read every page again and letrefresh_ref_docsskip the ones that did not change.askruns a retriever over the index, with no model, and prints the top page with its first two prices.
Day one: build, and nothing to refresh
$ python fresh_index.py build read 4 pages, credits: 4 monitor 01a1e5e9-811b-7b01-a497-78003c23e75d $ python fresh_index.py reread read 4 pages, credits: 4 re-embedded: [] $ python fresh_index.py refresh 0 changed checks: [] re-embedded: []
The same 4 pages, read again a minute later, re-embedded nothing: the text had not changed, so refresh_ref_docs kept the stored vectors.
Day two: one page changed
Our test store’s deals page gets new prices every night. The next morning, the monitor’s check said changed for 1 page out of 4, for 8 credits. Then:
$ python fresh_index.py refresh 1 changed checks: ['https://lab.megalania.io/deals/'] re-embedded: ['https://lab.megalania.io/deals/'] $ python fresh_index.py ask "Which products are on sale today?" 0.595 https://lab.megalania.io/deals/ | ['Sale! Original price was: $34.00.Current price is: $26.95.', 'Sale! Original price was: $58.00.Current price is: $48.95.'] $ python fresh_index.py reread read 4 pages, credits: 4 re-embedded: [] $ python fresh_index.py refresh 0 changed checks: [] re-embedded: []
- Only the deals page was re-embedded. Shipping, returns and the FAQ kept their vectors.
- The answer has today’s prices. The retriever found the deals page and its new prices, which were not on the page the day before.
- The snapshot was free. Downloading it three times left our balance where it was: the snapshot is the text the monitor already fetched and checked. It is the same Markdown our reader returns.
rereadagreed. Reading all 4 pages again after the refresh re-embedded nothing, because the index already had the new deals text.
The one thing that breaks refresh_ref_docs
LlamaIndex decides “changed” from a hash of the document’s text and its metadata. Our reader puts request_id and credits_charged in the metadata, and request_id is new on every read. So with the reader’s documents as they come, every refresh re-embeds every page. We measured it: the same 4 unchanged pages, twice, re-embedded all 4 both times.
The fix is in read_all(): build each Document with id_ set to the URL and only source in its metadata. Then the hash covers the text, and unchanged pages are skipped. Give the snapshot documents in refresh() the same shape, so both ways agree.
Monitor or re-read: what each costs
| Step | Credits |
|---|---|
build: 4 plain pages as Markdown |
4 |
| A monitor check of 4 pages: each page’s price, plus 1 per page | 8 |
| Downloading a changed page’s snapshot | 0 |
reread: all 4 pages again |
4 |
| A failed or blocked check | 0 |
So for these 4 pages, a daily monitor costs about 240 credits a month and a daily reread from your own cron about 120. Re-reading is cheaper. The monitor costs 1 credit more per page, and for that:
- We run the schedule. No cron, no server awake at 3 a.m.; you call
refreshwhen you need the index. - Ignore rules. A page with a date or a visitor counter changes every day. A monitor’s
ignoreregex drops that text before it compares, so the page does not count as changed. Re-reading would re-embed it daily. - A webhook. Give the monitor a
webhook_url, and it calls you with the changed pages, instead of you asking. - A diff per page, kept 30 days, to see what changed before you trust the new text.
For a few pages that change often, re-read. For many pages that rarely change, or pages with noise to ignore, use a monitor. All the rules are in Credits and Watch pages with monitors. Plans are on the pricing page.
How we measured: our dev cluster, the Lab store on our dev domain, Python 3.13, llama-index-core 0.14.26, llama-index-readers-megalania 0.1.1, llama-index-embeddings-huggingface 0.9.0 with BAAI/bge-small-en-v1.5 on CPU, megalania 0.3.0 and httpx 0.28.1. We ran build on 16 Nov 2026, then refresh the next morning after the deals page changed; we started that day’s check with the monitor’s run-now call instead of waiting for its time. The outputs are copied from those runs.
Limits
- Monitors run at most once an hour on the beta plan, and a list holds up to 100 pages. For a whole site, make several monitors, or re-read on your own schedule.
- Removed pages stay in the index. This script adds and updates; when you drop a URL, call
index.delete_ref_doc(url)too. - Our Python SDK has no call for a check’s detail or a page snapshot yet, so the script uses two plain GETs with
httpx. - Public pages only. We do not log in, and we do not solve CAPTCHAs. Read each site’s terms and robots.txt, and follow them.
More on loading pages into LlamaIndex, and on agent tools, is in LlamaIndex web scraping.