Recently we received a call from a customer needing a chatbot developed that was safe, accurate, and capable of near real-time responses. In simple terms, they needed an AI system that could answer support queries flawlessly, without exposing sensitive data or drifting away from the company’s voice. Requests like this are now more frequent as data security, latency, and response quality directly impact a company’s bottom line. AI systems that get these requirements wrong lose trust quickly and cost the business money. IBM’s 2025 report estimates the average global cost of a data breach at $4.44 million.
But generic chatbots and off-the-shelf LLMs often fail to meet enterprise expectations. There’s no doubt that LLMs are powerful. But these models face real constraints around token limits, context utilization, and hallucinations. These limitations are even more evident with the increasing demand for domain-specific knowledge and strict response formats. When all these aspects are at work, how do you build an AI that knows how to answer like an expert, knows what to answer based on real data, and still remains fast, safe, and controllable?
From my experience in developing models, I can say that the answer is not a single model or technique. It requires a broader architectural approach that separates what the model knows from how it responds, while combining learning with retrieval.
The Core Challenges
Early in the design phase, four fundamental challenges became clear.
1. Effective Context Limits (Beyond Token Counts)
Modern LLMs advertise context windows of 16K, 32K, or even 128K tokens. But in real-world use, anyone who works closely with these models knows their attention starts to break down much sooner. When large volumes of text are passed as context, models often underutilize information in the middle of the prompt, which is known as primacy–recency bias.
Increasing context size does not guarantee better answers. For enterprise environments, where knowledge bases can span millions of tokens, this is not a solution.
2. Weak Utilization of Long-Tail Information
LLMs may ignore it, misinterpret it, or overweight irrelevant sections even when relevant information is present in the prompt. Works, such as Lost in the Middle, support this theory. It highlights how long-context inputs often lead to incomplete reasoning if not carefully controlled. This makes naïve “dump everything into the prompt” strategies unreliable for complex, domain-heavy support systems.
3. Precision vs. Performance Trade-offs in Retrieval
Retrieval introduces real-world latency and compute costs. If you retrieve too much, then context increases response time and dilutes model attention. If you aim for too little, then context increases the risk of hallucinations. The real challenge is not retrieval itself, but precision retrieval. This guarantees minimum sufficient context required for correctness without overwhelming the system or the model.
4. Hallucinations Under Missing Context
LLMs rarely abstain from generating responses even when relevant information is missing. Instead, they respond confidently with generic or fabricated answers. In a support setting, this behaviour is unacceptable. It directly impacts trust, correctness, and compliance.
A close study of these constraints made one thing clear: passing more context was not the solution. We needed a smarter architecture.
The Answer: A Hybrid Architecture
Our work consistently pointed toward a hybrid approach combining retrieval augmented generation (RAG) with fine-tuned language models. The key insight was that fine-tuning and retrieval solve different problems. Fine-tuning teaches the model how to answer and retrieval supplies what to answer. We found that forcing one type of method to do both can lead to eithe inefficiency, instability, and higher costs. Therefore, we created a system to allow both components to use their strengths.
RAG: Precision through Retrieval
We made an effort to avoid “flooding” the model with large amounts of raw documents. In contrast, we created a searchable, curated knowledge base from our internal Q&A, product manuals, technical documentation, and policy and configuration reference materials. At inference time, the retriever selects only the most relevant content chunks and inserts them into the prompt. This enables answers to be based upon “real” data which is verified.
We found that this approach significantly decreased hallucination rates, improved factual accuracy, and increased the rate at which the models could respond to queries due to using smaller, contextual windows for each query. However, reliance solely upon RAG was insufficient. Even when the model’s retrieval accuracy was very high, the outputs exhibited large variability with respect to tone, structure, formatting, and the amount of procedural detail included. These outputs indicated that while factually accurate, the outputs were neither consistently structured nor formatted.
In one chatbot use case designed to improve engagement, evaluation showed that even with nearly 100% correct context, output correctness was only about 70%. The model was unable to extract long-context and failed to maintain the desired conversational tone needed to guide users to engage in deeper technical conversations or attend follow up meetings.
Thus, a fundamental flaw in retrieval has been identified: retrieval will provide information, but it will never provide instruction as to how to reason about or to communicate within a specific domain.
Fine-Tuning Qwen: Teaching the Model How to Answer
To improve consistency, tone, and reasoning, we fine-tuned the Qwen model on roughly 1000 expert Q & A pairs that were carefully selected to best fit the Qwen model’s domain. We did not want the model to learn facts. We wanted the model to learn how to apply domain specific language, to use the same company voice and style of communication, to follow the format for every response, to follow the process of answering the question, and to provide answers for the ‘edge-cases’ that occur within support workflows.
Fine-tuning adjusts how a model works, not what the model knows. That is an important point. Fine-tuning can lead to catastrophic forgetting and excessive computing costs if the entire model is fine-tuned. In order to limit these problems, we used low-rank adaptation (LoRA) adapters. LoRA adapters allow you to fine-tune only the small adapter matrices of your model, while preserving the majority of the base models general knowledge. LoRA adapters also reduce the amount of GPU memory needed by your model for fine-tuning and produce performance that is nearly equal to fully fine-tuning the model.
Once this was done, the results were clear. The model became much more consistent and much more nuanced. For stable, procedural questions, the model produced answers correctly, many times without doing any retrievals. However, as we expected, the model failed when trying to answer new feature questions, updated policy questions, and long-tail factual questions.
In the same chatbot example, fine-tuning the model improved the tone alignment to about 90% and decreased the accuracy of the model to about 50%. Once again, the lesson was reinforced, fine-tuning does not replace retrieval.
Why Neither RAG Nor Fine-Tuning Alone was Enough
These experiments with small language models gave us a clarity about trade-offs. First were RAG-only systems that have superior factual grounding and recency capabilities, but poor tone consistency and high latency. Second were fine-tuned-only systems that have superior voice and structure consistency, but failed when knowledge changed or long-tail facts were needed.
We saw first-hand that choosing only one would mean accepting the weaknesses of the other. The combination of using a fine-tuned model in conjunction with a RAG model produced better results than each used separately. Tone accuracy improved to roughly 75%, where as RAG had no reliable tone control and the fine-tuned model was accurate at roughly 90%. Factual correctness improved to approximately 73%, which exceeded both the fine-tuned model (approximately 50%), and RAG alone (approximately 70%). The fine-tuned model’s understanding of domain information and expected output format allowed it to better utilize and apply the context extracted from the retrieval process than a base model.
