AI Agent Knowledge Base: Which Policy Version Should It Answer From?
Start with the short answer ↓
Updating a document is not proof that an AI agent uses the new policy. Define source ownership, effective dates, retrieval eligibility and a small regression set before changing the answers your business relies on.
Prepared with AI assistance. These are practical scoping recommendations; examples are illustrative, not client results.
Start with the rule that may change, not the upload button
A service team changes the minimum notice for a new booking from 48 hours to 72 hours. Someone replaces the procedure in a shared folder. The agent still has yesterday's answer in an open conversation, while an older PDF remains searchable under the same title. Which rule should it explain now? This is an illustrative planning scenario, not a client incident or a delivered AgenticShip feature.
Retrieval-augmented generation, often called RAG, brings selected source material into an AI response. It does not by itself decide which business document is authoritative today. The implementation needs a source lifecycle: who approves a version, when it becomes effective, where it may be used and how its replacement is verified. A recent upload time is not the same as a business effective date.
Give each policy a stable identity and an accountable owner
Record a policy ID that survives file renaming, a version ID, an owner, an approval state, effective-from and effective-until boundaries, the relevant timezone, audience, language and source location. Agree what event determines applicability: the time of the question, the time a booking was created or another business event. Avoid letting two independently uploaded copies silently become competing authorities.
The following is our proposed source-state worksheet, not a vendor standard. The retrieval layer must enforce eligibility using fields the selected system actually supports, with application checks where needed. A prompt asking the model to use the latest file is not an equivalent control.
| Source state | Use in a current answer | Evidence needed |
|---|---|---|
| Draft | Exclude from customer-facing policy answers | Named owner and outstanding approval |
| Approved, future-effective | Do not present as today's rule; explain a future change only when appropriate | Effective boundary, timezone and transition rule |
| Active and applicable | Eligible for the permitted audience and question | Approved version, applicability fields and verified indexed copy |
| Retired | Exclude from current guidance; retain separately for authorized historical questions if required | Replacement ID and retention/access decision |
| Conflicted or withdrawn | Stop the affected answer and route the gap to its owner | Conflicting IDs or withdrawal record; no invented winner |
Check the searchable copy, not only the source folder
A document and the searchable representation used by the agent are separate checkpoints. Verify that the intended version reached the index, that the old version is no longer eligible for current answers and that the relevant passages carry the correct metadata. Record the ingestion result and inspect an actual retrieval result. A successful upload alone does not demonstrate that the answer path changed.
For example, Amazon Bedrock documents a data-source sync after files are added, changed or removed, with incremental processing of those changes. That describes this Bedrock workflow, not every AI platform. Confirm the update and removal behavior of the chosen connector instead of assuming every edit propagates immediately.
Keep the authoritative source and indexed copy consistent. If an implementation has both direct index edits and a source-sync process, check whether a later sync can restore unwanted material. Test removal in the actual retrieval path. Do not delete required business history merely to hide it from current-policy answers.
Prefer recent content only after deciding what is eligible
Recency ranking and policy eligibility solve different problems. A newly uploaded draft should not outrank an approved rule into authority. An old but still applicable contract may matter more than a recent general FAQ. Filter by the agreed status, scope and effective boundary before asking relevance ranking to select useful passages.
Microsoft's documentation for Azure AI Search freshness-aware retrieval, a preview feature, explicitly distinguishes a preference for newer material from a hard filter: older content can still appear. This is useful background for the distinction, not a recommendation to adopt a preview service. Our implementation recommendation is to test eligibility separately from ranking quality.
Keep access controls independent of freshness. A correct current document for one account, staff role or language audience is not automatically authorized for another. If the system cannot establish an eligible source, the useful result is a visible gap for review, not a plausible answer assembled from excluded documents.
Worked example: a 48-hour rule becomes a 72-hour rule
Suppose the procedure owner approves version B of BOOKING-NOTICE, effective November 1, 2026 at 09:00 in the business's agreed timezone. Version A requires 48 hours of advance notice; B requires 72. For this example only, the owner decides that B applies to bookings created at or after that boundary and that existing bookings keep their original terms. These are invented inputs, not legal advice or a universal booking policy.
Before the boundary, an answer about a new booking uses A. After it, the same question uses B. A question about a booking created earlier must use that booking's applicable rule, not automatically today's rule. If creation time or policy assignment is missing, ask for the missing fact or hand the case to the owner. Do not infer it from the conversation's latest message.
Prepare B and its metadata before the cutover, then verify how the chosen system activates it. At the boundary, check a new retrieval and the generated answer, including the source/version reference. Re-evaluate an answer drafted in an older session before it is used after the change. If a workflow caches retrieved passages or answer drafts, explicitly test its invalidation or revalidation path; changing a source file is not proof that an existing conversation has forgotten it.
Run a small change-specific regression set
In an isolated test environment, write the expected eligible version and the expected answer or escalation before each test. Cover seven cases: a new booking just before the boundary; one exactly at it; an earlier booking discussed later; two documents with the same title; an active document that is withdrawn; a previously opened conversation containing A; and a question for which no permitted, effective source exists.
Add a conflict case where two approved sources disagree and an access case where the right answer exists only in another account's material. Test Hebrew and English when both belong to the operating scope. Record which source passages were retrieved, not only whether the final wording sounds correct. A correct answer drawn from an ineligible source is still a boundary failure.
Release only against the workflow owner's agreed criteria. A forbidden source, an invented policy or a confident answer to an unresolved conflict should block the affected scope under the proposed contract. This focused set supplements the wider agent evaluation; it is not a statistical guarantee or a replacement for ongoing monitoring.
Keep an update receipt and a usable stop path
The update receipt should identify the policy and old/new versions, approving owner, effective boundary, source changes, ingestion evidence, retrieval tests and activation time. Keep references to the evidence with appropriate access and retention; do not copy unnecessary customer data into a general knowledge log. Name the person who resolves a conflict and the limited manual path available while the affected answers are paused.
Restoring an old index is not automatically a safe rollback if its policy is no longer valid. A rollback may mean disabling that answer path and routing questions to a person until an eligible version is available. Preserve unaffected work only when its source boundary is understood. After release, review stale-source incidents, unresolved questions and correction effort separately from answer volume.
For AI agent implementation, the practical scope is one owned source set, an explicit change process and evidence that the intended answer path follows it. Expand after those checks work. The deliverable is not a promise that the agent always knows the latest rule; it is a controlled way to update, verify and stop the specific knowledge workflow.
Primary references
The guidance above is AgenticShip's proposed approach. These references provide technical background for the relevant recommendations.