RAG or fine-tuning? Understand your company chatbot’s failure first

“We need to train AI on our company data.” That often starts a chatbot project. First ask what is actually missing. Does the system lack current contract terms? Does it fail to follow the required output format? Are the documents themselves contradictory? Those problems call for different remedies. Training is not automatically the right response to missing company knowledge.

For many small businesses, the useful starting point is a clear task with a few selected documents. Regular search across a larger collection may justify Retrieval Augmented Generation, or RAG. Fine-tuning instead adapts model behaviour using training examples. The approaches can be combined, but they do not perform the same job. Choose according to the observed failure rather than the more impressive name.

Three approaches and their actual work

The AWS overview of document-query options distinguishes directly supplied context, RAG and model adaptation. Instructions define the task, limits and required output. A limited workflow can receive its relevant information directly with the request. RAG searches a collection for material relevant to the question and supplies it for the answer. Fine-tuning adjusts model behaviour through examples. An adapted model still needs a route to facts that change after training.

The AWS comparison of RAG and fine-tuning recommends starting with RAG for questions about your own documents. That is useful guidance rather than a guarantee for your project. Test whether retrieval finds the right passage and whether the answer uses it correctly in your actual collection.

Start with a failure list

Collect representative questions and record the evidence needed for each expected answer. Include questions with no available answer and at least one involving conflicting sources. Use anonymised examples. The goal is not to upload as many conversation logs as possible. It is to describe the behaviour the business requires in a form you can check.

Classify every failure. Was the current source not found? Was it found but misunderstood? Is the substance correct but the format wrong? Did the assistant promise an unauthorised action? A stronger language model cannot reliably repair a poor search collection. A larger collection cannot replace an unclear approval rule. Classification helps avoid buying a sophisticated solution to the wrong problem.

When instructions may be enough

With a few stable rules and a small clear context, begin simply. An assistant might classify enquiries into three defined categories and identify missing information. Supply the category definitions, examples and boundaries. Establish which cases still fail after that improvement.

The simple approach has limits too. Frequently changing documents or different sources for each question make manual context selection awkward. An overly long instruction can conceal conflicting rules. Do not keep a simple setup merely on principle. Use its results to justify the next necessary step, and keep the successful tests as a baseline for comparison.

When to investigate RAG

RAG becomes interesting when answers depend on a larger or changing document collection. The work shifts towards source maintenance, retrieval, metadata and permissions. A new price sheet should replace an earlier version visibly. An internal personnel record should not enter a customer answer just because its vocabulary resembles the question.

Review retrieval separately from writing. First show the reviewer only the retrieved passages. Are they enough to answer the question? Then review the answer and its relationship to the passage. Elegant wording contributes little if retrieval already selected the wrong material. Missing evidence requires an identifiable limit and a next step rather than invented certainty.

When to investigate fine-tuning

Consider fine-tuning when a recurring behaviour problem remains after good instructions and examples. A particular classification task or consistent output structure may be worth investigating. You need suitable training data and a separate evaluation set. Improvement must appear outside the examples used to adapt the model.

Do not assume opening hours, live orders or revised conditions become permanently current knowledge inside the model. Define how those facts are supplied at runtime. Fine-tuning also does not automatically add citations or permission checks. Those functions still need an explicit workflow and verifiable interfaces.

Compare the continuing work

Approach	Ongoing work	Important test
Instructions	Maintain rules and supplied information.	An ambiguous case outside the examples.
RAG	Maintain sources, index, permissions and search.	A new source and a prohibited source.
Fine-tuning	Maintain dataset, model version and evaluation.	An unseen case in the separate test set.

Compare model usage alongside search operation, maintenance, review and human corrections. A lower cost per answer can disappear into frequent manual repair. A small pilot should expose the time needed to keep the system usable. Identify who approves a new source and how a failed answer leads to a specific improvement.

Use a changed document to test the choice

A revealing experiment is a deliberately revised rule. Use a fictional internal document and change the acceptance deadline for a test service. Ask the same question before and after. The system should use the current rule rather than keep presenting the old one as valid. Check whether the change has an owner and whether it can be reversed.

The experiment shows where facts live and what maintenance takes. It does not prove general accuracy. Add other questions and permitted user roles before making a production decision. Record the result, version and remaining failures so the eventual selection can be explained later.

Also include a question requiring more than one source. A correctly retrieved paragraph may still be insufficient when the answer depends on both a service policy and an exception list. Review whether the system recognises the missing part. That can reveal a retrieval design problem which training the answer model would not directly solve.

Keep the final test independent

Decide which questions support development and which remain unseen until the final check. Otherwise a better score may only reflect improved handling of familiar examples. Include changed wording and situations requiring a different answer. Try a short conversation in which the person corrects a detail or changes subject. Information carried over incorrectly can fail even when retrieval works. Record those failures separately. Before another investment, show the operating owner concrete remaining errors and their expected source rather than a single overall score.

Our articles about answer sources and provenance cover later traceability. This article concerns the investment decision before that. For a company chatbot, bring questions, current sources and observed failures. They allow us to assess which additional technology the workflow actually needs.