A customer writes: “You said cancellation is free up to 48 hours before the appointment. My quote says something different.”
Support opens the conversation. The bot’s answer is fluent and specific. Unfortunately, the trace does not show whether it used the public policy page, an old PDF, a sales template or a customer-specific document.
This is not necessarily a freshness problem. Both documents might be current. They may simply apply to different customer groups.
That is why a production RAG assistant needs provenance: enough information to answer the operational question, “What evidence produced this answer?”
Freshness tells you when; provenance tells you where and how
An expiry date on company knowledge is useful. It tells you whether a service fact is still meant to be trusted. It does not prove that the fact applies to this specific question.
A policy updated yesterday can still be the wrong policy for an existing contract. An older agreement can remain the legally or commercially relevant source for one customer. “Newest” and “applicable” are different attributes.
The W3C PROV family describes provenance as information about entities, activities and people involved in producing a piece of data or thing, allowing assessments about quality, reliability or trustworthiness. A Swiss SME does not need to implement the full semantic standard to benefit from the idea.
For a chatbot, the minimum useful record is usually:
- source identifier and document title;
- version or last approved date;
- section or chunk that supported the answer;
- scope, such as customer type, region or contract;
- retrieval timestamp and permission context;
- conflicting sources, if any.
Confirmed fact: W3C PROV provides a standard model for representing and exchanging provenance information.
Author recommendation: consequential customer answers should carry source ID, version and relevant passage in an internal trace even when the customer-facing UI only shows a short reference.
Do not turn every answer into a research paper
A user asking “Are you open on Saturday?” does not need five footnotes. A customer saying “thanks” needs none.
The value appears when an answer can create a commitment or dispute: price, cancellation, warranty, availability, refund, delivery timing, eligibility, contract conditions or service scope.
Those answers can switch into an evidence mode.
Evidence mode is not about making the chatbot verbose. The visible response can remain short: “According to our cancellation policy, version 12 August 2026, this booking can be cancelled without charge up to X.” A small source link or label is often enough.
Behind the scenes, store the richer trace: document ID, section, retrieval time, other high-ranking candidates and the customer/account context used to select the source.
For internal documents, permission still matters. Do not expose a confidential source merely because the RAG layer retrieved it. Provenance does not override access control.
The real failure mode is often two credible sources that disagree
Business knowledge is rarely a single clean manual.
The public website says one cancellation window. A PDF in Sales says another. An old template adds a group exception. The CRM contains a negotiated condition for this account.
A weak assistant picks the most similar passage and writes a confident synthesis.
A better system treats disagreement as information. If two sources conflict and the answer can affect money or a customer promise, the system should not hide the conflict behind a fluent paragraph.
A useful rule is: material consequence + unresolved source conflict = abstain or escalate.
The chatbot can say: “I found two different conditions that may apply here. I’m passing this to the team so we don’t give you the wrong commitment.” That is better service than a polished guess.
There is another exception: the source itself can be ambiguous. Provenance lets you identify the document, but it does not magically make a poorly written contract clause clear. It improves diagnosis of the knowledge problem; it does not eliminate the need for human judgement.
A practical provenance layer for a small-business RAG system
You can implement the operating pattern without rebuilding your entire stack.
- Give important documents a stable ID, owner and explicit version or approval date.
- Preserve source metadata when content is chunked and indexed; do not keep only anonymous text fragments.
- Define answer classes: routine fact, customer commitment, sensitive information and conflict case.
- Show customer-visible sources where they help verification; keep richer internal evidence for support and audit.
- Make the assistant refuse a definitive answer when evidence is conflicting, weak or outside the user’s permissions.
- Test what happens when a document is replaced, archived or deleted.
Deletion deserves special attention. If a source disappears next month, yesterday’s conversation should not falsely look fully reproducible. Keep enough metadata to understand what source was used, subject to your retention and privacy requirements.
Human handoff should include the evidence packet
A transfer that sends only “customer has a cancellation question” wastes the provenance work.
Send the customer’s question, the tentative answer, source names and versions, the conflict that was found and why the assistant stopped. For a price case, the note might say: “Public price list v4 says X; account contract contains a special rate; no final price promised.”
Now the employee can decide rather than repeat retrieval from scratch.
The human still owns the final judgement. A beautifully traceable source can be wrong, outdated in practice, mis-scoped or legally inapplicable. Provenance gives you accountability and debugging; it is not a guarantee of truth.
Minimum pilot: choose the knowledge that actually causes disputes
Do not begin with your easiest FAQs. Pick one area where customers and staff genuinely need evidence: cancellations, service coverage, delivery conditions or warranty.
Create 30–50 test questions. Include outdated-but-still-applicable contracts, conflicting public/internal wording, missing sources and a user who lacks permission to see one document.
Track answers with a complete source trace, consequential answers backed by the correct applicable source, source conflicts detected, handoffs containing usable evidence, and definitive claims produced without adequate support. A sixth useful metric is employee verification time after handoff.
The conclusion is simple: an AI chatbot should not only know an answer. For important answers, your business should be able to reconstruct why the system believed it. If you cannot do that, the model’s confidence is not a reliable business control.
Our article on keeping service knowledge current covers ownership. For internal material, permission-aware RAG prevents the citation itself becoming a leak.
Sources
Before implementation, choose the necessary approach. The comparison RAG or fine-tuning separates missing facts, retrieval failures and output problems while exposing ongoing maintenance.
FAQ
Does every chatbot reply need a visible citation?
No. Keep the experience light for routine interactions. But consequential claims should remain traceable internally and expose a source when that helps the customer verify the answer.
Is the document title enough?
Usually not. Version, section and retrieval time become important when documents are updated or multiple policies have similar names.
What should happen when two sources conflict?
Do not force a best guess for a material commitment. Flag the conflict and hand it to a person with the evidence attached.
Is provenance the same as RAG?
No. RAG retrieves information for generation. Provenance records where the information came from and how it contributed to the final answer.
What if the source is later deleted?
Your retention strategy should preserve enough metadata for audit while respecting privacy and data-minimisation requirements.