Keeping a RAG index current

A support team changes its returns policy from thirty days to fourteen. The document is saved, the indexing job runs, and the assistant still answers thirty. Its citation opens the current policy, which makes the answer look even stranger.

There are several ways to get here. The connector may have missed the edit. The index may contain chunks from both versions. An answer cache may still hold yesterday’s response. A useful freshness check has to follow the change all the way from the source document to the passages used in an answer.

Keep a record of each source

Give every document a stable identity that includes its source system and tenant. Keep its source revision, content hash, access rules and last successful ingestion time in a small manifest. Store the chunk IDs belonging to each indexed revision. A title or file path alone makes a poor identity because people rename and move documents.

The manifest lets a worker answer a specific question: which revision of this document is currently searchable? Put that revision on every chunk and in retrieval logs. A timestamp saying that the nightly job completed cannot tell you whether this particular policy reached the index.

Decide how the connector handles overlapping jobs. If an older fetch finishes after a newer one, it must not publish over the newer revision. Use the source’s ordering guarantee where it has one. For opaque revision tokens, serialise updates per document or check the current source revision again before publishing.

Replace the whole document’s chunk set

Suppose a policy shrinks from five chunks to three. Updating chunks one, two and three in place leaves chunks four and five behind. Those leftovers may contain exactly the rule that was removed.

Build the new chunks under a separate revision. Check that parsing succeeded, embeddings are complete and the chunks are queryable before marking that revision active. Every retrieval path must exclude inactive revisions. In a design with an external manifest, validate candidate passages against it before sending them to the model, and fetch more candidates if rejected chunks leave too few results.

Keep the previous revision available until the switch completes, then clean it up. Record failed builds so an empty parser result cannot silently replace a working document. Some sources allow genuinely empty documents; handle that as an explicit case.

For a rebuild of the entire index, a search service may provide a native switch. Elasticsearch supports changing an index alias through a single request. Its alias documentation also explains action-level errors and the must_exist option. Check the result before treating the new index as live.

Give deletion its own path

A file disappearing from one connector response is ambiguous. The file may have been deleted, the account may have lost access, or the listing may be incomplete. Deleting every unseen document after a failed crawl can empty a healthy index. Keeping everything forever serves withdrawn material.

Use explicit deletion events where available. Otherwise, reconcile against a complete, successful source listing. Record a deletion marker before removing chunks, so retrieval can stop using the document while physical cleanup catches up. Ensure an old queued update cannot revive it.

Source change	Immediate handling	Background work
Text edited	Prepare a new revision	Publish the complete chunk set, then remove the old one
Document withdrawn	Mark the source unavailable for retrieval	Remove chunks and invalidate dependent answers
Access revoked	Reject retrieval for the affected user or group	Refresh indexed access metadata and caches
Connector unavailable	Record the failed sync and age of the last success	Retry; retain the last confirmed source inventory

Check the connector’s deletion contract. For example, Azure Blob indexers need a deletion-detection strategy. With native soft delete, the indexer must see the blob during its retention window. Microsoft’s deletion guidance recommends allowing enough retention time for indexing delays. A schedule that normally runs every hour can still miss a deletion during a prolonged outage.

Separate access changes from embedding work

Revoking access should not wait behind a large embedding batch. Keep a current authorisation check at the retrieval boundary, and define how quickly a source permission change must reach it. If that check relies on a synchronised copy, its delay is part of the access guarantee. When permission cannot be established, withhold the passage.

Apply the same rule to answer caches. A cached answer may contain text from several documents; record those dependencies and recheck access before reuse. Previously retrieved passages in a long conversation also need attention. Revocation cannot erase text already shown to a user, but the application can prevent those passages from being sent into a new model call.

For ordinary edits, choose an acceptable delay with the people who own the content. A staff biography and an emergency operating instruction may need different treatment. If a document becomes too old to trust, exclude it or show its age and route the question to a person, according to the workflow.

Measure a change reaching search

Track the time between a source revision becoming available and that revision passing a retrieval check. Break the delay into discovery, extraction, embedding and publication so a slow stage has an owner. Keep failed documents visible even when the rest of a batch succeeds.

Add a controlled test document with a unique value. Change the value, query through the same retrieval path as the assistant, and assert that the new revision appears and the old one is excluded. Test the retrieved passages directly before checking the generated answer; a model can happen to answer correctly despite stale evidence.

Extend the evaluation set with a document that shrinks, a deletion, a permission revocation and two updates delivered out of order. Run each through the real connector and publication code in a test source. For every case, record the last moment the obsolete passage can reach a model call. That gives the team a freshness limit it can actually verify.