How to Keep a RAG Corpus Up to Date and Verify That Old Documents No Longer Appear

Maintaining the corpus is not the same as evaluating answers

A RAG corpus is maintained when the content available for retrieval continues to match the sources the team considers valid. The work includes identifying documents and versions, processing changes, updating indexed units, and verifying the result. This is distinct from evaluating whether the model writes a correct answer or grounds it well in the retrieved passages. The two kinds of evaluation can be related, but they are not interchangeable: a retrieval test may reveal that a withdrawn document still appears, even if the generated answer seems reasonable.

The supplied Microsoft documentation describes creating an unstructured data pipeline for AI applications. The Google Cloud documentation covers managing a RAG corpus and, according to the information available, includes creating a corpus and importing or uploading documents. These references provide context about processing and ingestion, but the available information does not confirm a universal procedure for updating or deleting documents. The steps below are therefore an adaptable operational design, not a claim that a particular feature exists in every product.

The practical goal is to be able to answer questions about content state: which source a passage came from, which version was processed, whether a partial failure occurred, and which test demonstrates that a removal took effect. If the team cannot answer these questions, it should not treat the corpus as up to date merely because a processing run finished.

Inventory sources before automating

Start by listing the sources that may feed the corpus and assigning an owner to each one. A source might be an internal documentation repository, a catalog, a set of regulations, or a collection of files. The inventory does not need to depend on a specific product, but it must let you distinguish each origin and identify who decides whether its content is added, changed, or withdrawn.

For each document, retain a stable identifier if the source system provides one. Also record the source location or collection, the owner, a verifiable date or version, and the document's state—for example, active, awaiting processing, or withdrawn. If there is no formal version, you can record a change signal available in the source, provided the team documents what it means and its limitations. Do not assume that a modification date is enough to distinguish identical content from two different revisions.

Link these properties to every derived unit that is indexed, such as a passage or record. The aim is to be able to locate all units derived from a source when it changes or is withdrawn. If the indexing mechanism does not allow metadata to be stored, or does not document how to filter and remove records, note that as a limitation to test; do not assume that deleting the source file automatically removes its derived content.

Also define the rules for adding and withdrawing content. For example, specify who authorizes a document's transition to active, what counts as a replacement, and how conflicting sources are resolved. These are team governance decisions, not properties inherent to RAG. Writing them down prevents a technical update from accidentally deciding which version should be considered valid.

Decide which changes require reprocessing

Not every change has to trigger the same amount of work. A localized correction might require processing one document; a change to extraction rules or text-splitting behavior might affect an entire collection. A configuration change that transforms the stored representation could warrant a broader rebuild. These are decision categories: before applying them, check which components the stack uses and which operations it supports.

Write a policy that maps change signals to reprocessing scope. For example, a new version identified at the source triggers reprocessing of that document; a global extraction change triggers a review of the affected collection; uncertainty about the consistency of the set may trigger a test rebuild. Do not turn these examples into automatic actions until you have first validated document identity, deletion capability, and tool behavior.

The pipeline described by Microsoft can serve as a general reference for thinking about unstructured data processing, but the available information does not specify which incremental update, synchronization, or deletion operations it implements. Likewise, the Google Cloud reference on corpus management confirms the topic of corpus creation and import, but does not support conclusions about the conditions for modifying or deleting records. Actual behavior must be checked in the documentation for the chosen stack and through controlled tests.

Design the additions and changes lifecycle

An update lifecycle can be organized into explicit stages. First, detect a new source or a change using a signal available in the source system. Then record the observed version and validate that the document is eligible under the team's rules. Next, extract the content, process it with the agreed configuration, and retain metadata that makes it possible to trace it back to its source.

Before publishing the result, decide how to handle the transition between versions. One possible strategy is to prepare and check the new version before making it available, if the platform allows it to be kept separate. Another possibility is to replace the content directly and verify it afterward. Neither strategy should be presented as universally superior: the right choice depends on documented capabilities, the system's tolerance for interruption, and how partial state is handled.

At a minimum, the run record should show the target source and version, detection date, planned scope, the state of each stage, and verification results. If processing is interrupted, this information helps distinguish a source that was not detected from a failed extraction or an update that was only partly applied. Do not mark a version as current merely because its processing started.

Include an observable completion criterion. For example, the operation does not move to completed until you verify that the expected units are retrievable and that the previous version no longer appears when it should have been replaced. The criterion must be feasible with the specific tool. If the tool offers no way to check this, record the limitation and design an external retrieval test.

Manage removals and replacements without leaving residue

Removing a file from the source and removing its representations from the index are different actions, unless the specific integration documents and demonstrates otherwise. A removal can leave passages, derived copies, or duplicate records if everything originating from the source cannot be identified. For that reason, the relationship between source ID and indexed units should be designed before the first removal occurs.

For a replacement, record which version is no longer valid and which one should take its place. Check that the new version is processed correctly and that the old one no longer appears in searches where it was previously relevant. If both versions must coexist for a legitimate reason, define how to distinguish them and the state of each. Do not present them as equivalent if the source policy considers them incompatible.

A removal should leave evidence: the withdrawn source, request time, owner, affected units or operation scope, outcome, and test queries. If the operation fails, keep a pending or failed state and do not present the corpus as fully up to date. Recovery might involve retrying, rebuilding part of the corpus, or escalating the issue; the choice depends on available capabilities and should be recorded.

The supplied sources do not provide enough confirmation to claim that a specific platform automatically deletes every passage derived from a document when that document is deleted. The Reddit discussion included as a source is anecdotal and cannot establish guaranteed behavior. Check deletion features in the official documentation for the specific product and, above all, verify the result with a reproducible test.

Control partial updates and failures

An update process can stop after detecting a change, during extraction, or after modifying only part of the index. The operational design should make that distinction visible. Define simple states such as pending, processing, completed, and failed, and specify what each one means. If the team needs to record more detailed stages, it can add them as long as operators understand how they relate to one another.

Decide what content is considered available while an update is incomplete. In some systems it may be possible to keep the previous version until the new one is validated; in others, the update may create an outage window or mixed results. Do not promise atomicity or interruption-free replacement unless the platform documentation confirms it and a test verifies it.

Set a retry policy that does not create duplicates. Before rerunning a process, check whether the previous operation left partial units and how to identify them. If you cannot determine that, treat the outcome as uncertain and avoid a blind second write. Record the scope, decision, and subsequent test so another operator can reconstruct which version was available.

Choose between incremental updates and rebuilding

An incremental update limits the work to affected documents or collections; it may be appropriate when the system can precisely identify each source and remove or replace its derivatives. A rebuild processes a broader set and may make an end-to-end check easier, but it can also require more time and resources. These are general operational considerations, not performance guarantees or descriptions of a particular tool.

Compare the two options using data from your environment: source volume, change frequency, observed processing time, processing cost, how easily affected items can be identified, and the consequences of an incomplete update. Also consider the stack's documented ability to update, delete, query states, and restore a previous version. If an essential capability has not been confirmed, record it as an uncertainty to resolve before automating.

Do not choose incremental updates just because they seem cheaper, and do not rebuild automatically after every change. The decision depends on whether the impact can be bounded and the consistency of the result demonstrated. A test rebuild in a controlled environment can help estimate time and reveal differences; it does not, by itself, demonstrate that a production rebuild is safe.

Test that the corpus reflects the expected state

Acceptance tests should check corpus behavior, not just whether a technical task finished without an error. The supplied information about creating corpora for RAG systems and using them in evaluation supports the relevance of designing checks for RAG systems, but it does not prescribe a particular test suite or a universal threshold here. Queries and criteria should be derived from the sources and risks of each application.

Build a small, stable set of queries. Include queries that should retrieve a current version, distinctive queries for a withdrawn source, and queries about unrelated documents that should remain available. Record the expected source and version, the observed result, and when the test ran. If retrieval depends on ranking, thresholds, or other settings, note the relevant configuration so changes between runs can be interpreted.

Add tests for duplicates and conflicts. A search may return the old and new versions, two copies of the same document, or passages that do not retain enough provenance to distinguish them. Define acceptable outcomes: for example, one active version when policy requires exclusivity, or both versions clearly identified when coexistence is authorized.

A negative test—not retrieving a withdrawn source—requires care: the fact that it does not appear in one query does not prove it has disappeared from the entire index. Use several representative queries and, if the tool permits it, inspect records associated with the source ID. State the actual scope of the check and avoid turning a sample into an absolute guarantee.

Retrieval evaluation also does not by itself determine whether the model writes a correct answer. If the team also needs to assess the final answer, it should maintain a separate test set with criteria appropriate to that task. This makes it possible to distinguish a corpus-currency problem from a generation or context-use problem.

Periodic routine and limits of the evidence

A periodic routine should review pending changes, failed runs, sources without owners, versions whose currency has not been confirmed, and acceptance tests that have not been run. The frequency depends on the rate of change and the risk: a regulatory collection may require more frequent checks than an occasional reference archive. The supplied sources do not establish a universal schedule; each team should justify its own.

Also review whether extraction or processing rules have changed since the last update and whether that change requires earlier documents to be reprocessed. Check that IDs and metadata still flow through to retrieved units. If it is impossible to reconstruct which source and version contributed to a test, prioritize fixing that traceability gap before expanding automation.

The available sources have limited scope. The Microsoft documentation confirms the topic of unstructured data pipelines for AI; the Google Cloud documentation covers managing and importing documents into a RAG corpus; and the Institute for Knowledge Engineering article covers creating corpora for RAG and using them in evaluation. None of the supplied excerpts confirms the exact behavior of a platform for removals, replacements, transactions, rebuilds, or incremental updates. Those capabilities must be verified in the product's current documentation and through tests in the real environment.

Keep exploring

Corrections and transparency

If you spot incorrect or outdated information, send us a correction with the page and source we should review.