A finance team we spoke with last year had a neat AI feature: ask a question in plain language, get the number back. "What was gross margin in the Eastern Cape last quarter?" The number came back instantly, phrased in a confident sentence. It was wrong by four percentage points. Not wildly wrong: plausibly wrong, the kind of wrong that gets quoted in a board pack before anyone checks. The team's first instinct was to try a better model. Their second was to rewrite the prompt. Both instincts were aimed at the wrong part of the system.
The mistake was not the model. The mistake was letting the model be the thing that produced the number.
This post is about a principle that sounds obvious once stated, is violated constantly in practice, and sits at the centre of good AI governance: a language model should never be the source of truth for anything that has to be correct. Not the number, not the rule, not the decision. The model is allowed to read, retrieve, phrase, summarise, and explain. It is not allowed to be the system of record. The teams that trust AI in production are not the ones with the most accurate model. They are the ones who drew that line deliberately and built the architecture to enforce it.
Hallucination is structural, not a tuning problem
The reason the line matters is that hallucination is not a defect you can engineer away with a bigger model or a better prompt. It is a property of what a language model is.
The clearest statement of this came from OpenAI in September 2025, in a paper by Adam Kalai, Ofir Nachum, Santosh Vempala and Edwin Zhang titled Why Language Models Hallucinate. Their argument, grounded in computational learning theory, is that hallucinations are ordinary statistical errors (the same kind of error any classifier makes) and that they persist because the way models are trained and evaluated rewards confident guessing over admitting uncertainty. A model optimised to be a good test-taker will guess when unsure, because guessing scores better than abstaining. OpenAI's own framing is blunt: hallucinations remain a fundamental challenge for all large language models, and even their latest, lower-hallucination models still produce them.
Santosh Vempala, one of the paper's authors, put the deeper point plainly in a later discussion: a language model is built to reproduce the distribution of its training data, not to validate whether a statement is true. If it has not seen every true statement, and it cannot have, it will sometimes generate false ones. There is no internal mechanism inside the model that checks its own output against reality. The fluency and the factuality are produced by the same machinery, which is exactly why a wrong answer arrives with the same confidence as a right one.
This has a hard consequence that most AI roadmaps quietly ignore. If you cannot remove hallucination by improving the model, then "make the model more accurate" is not a strategy for correctness. It reduces the rate of error; it does not change the kind of system you have. A system whose correctness depends on a model never producing a confident falsehood is a system that will, eventually and unpredictably, produce a confident falsehood. The only question is whether it does so in a low-stakes place or in a board pack.
The fix is architectural, not statistical
If the model cannot be trusted to be the source of truth, the answer is not to keep trying to make it trustworthy. The answer is to stop asking it to be.
This is the shift that the most reliable production AI of 2026 has already made. Researchers at Berkeley's AI Research lab named it in early 2024: the move from monolithic models to compound AI systems: architectures that orchestrate a language model alongside retrieval, tools, deterministic code, and sometimes smaller specialised models, rather than asking one model to do everything. Databricks, summarising the same shift, makes the practical case directly: orchestrating a model with external tools and data sources makes the overall system more trustworthy, because the facts come from a source you control and the output format is enforced by code rather than hoped for in a prompt. By their count, the majority of enterprise LLM applications now use retrieval for grounding and a large share use multi-step chains. The single-model-does-everything pattern is already the minority.
The principle underneath the architecture is a division of labour. Separate the parts of your system that must be correct from the parts the model is genuinely good at, and let each be handled by the thing suited to it.
- The model handles language. Understanding a question phrased five different ways. Turning a structured result into a readable sentence. Summarising a long document. Extracting candidate fields from messy text. These are the things language models are extraordinarily good at, and none of them require the model to be the authority on a fact.
- Deterministic logic handles truth. The number is produced by a query or a calculation, not by the model. The decision is produced by a rule you wrote, not by the model's judgement. The classification that triggers an action runs through logic you can read. The model can phrase the result and explain it. It does not get to author it.
In the finance example, the correct architecture is not a cleverer prompt. It is: the model interprets the question and maps it to a defined metric; a query engine computes the number from the warehouse; the model phrases the answer around a number it did not invent. The model never touches the arithmetic. When it gets the question wrong, it returns the wrong metric, visibly and debuggably, rather than a plausible wrong number. That is a failure you can catch. The other kind is not.
Anything the model produced is a draft until something deterministic checks it
The same discipline applies to the increasingly common pattern of using a model to extract structured data from unstructured sources: pulling figures off an invoice, fields out of a contract, attributes from a product description. This is useful and it works. It is also a place where teams quietly let model output become source-of-truth without noticing.
The rule that holds up: anything a model extracted is labelled as model-generated and treated as a draft until a deterministic check or a human confirms it. It does not silently flow downstream into a calculation or a decision as if it were a verified fact. The label travels with the data. When an auditor (or a customer, or your own engineer at 2am) asks "where did this number come from," the honest answer is either "a system that computed it deterministically" or "a model suggested it and here is who confirmed it." The answer is never "the AI said so, and we wired it straight into the ledger."
There is research support for the strength of this boundary. A 2025 line of work examining whether hallucination can be eliminated at all found that it could be suppressed reliably only by adding an external validation mechanism that sits outside the model: a component the model cannot talk its way around, which checks claims and forces abstention when they cannot be supported. The finding is consistent with the architectural argument: the thing that makes a system trustworthy is not inside the model. It is the deterministic layer you put around it.
Why this is an AI governance problem, not just an engineering one
It is tempting to file all of this under engineering hygiene. It is more than that, because the line between "the model decides" and "the model explains" is exactly the line that audits, regulators, and serious buyers care about.
A deterministic decision layer is reproducible. Anyone can read the rule, run it by hand, and get the same answer. A model's judgement is not reproducible in that sense: run it again and you may get a different answer, and you cannot point to the line of logic that produced it. The first survives diligence. The second generates a very uncomfortable meeting.
The regulatory direction reinforces this. The EU AI Act's Article 14 requires that high-risk AI systems be designed so a human can understand their capabilities and limitations, detect when they are misbehaving, and correctly interpret their output: explainability built into the architecture, not bolted on afterwards. Article 26 requires deployers to keep the system's logs for at least six months. The Act's transparency obligations apply from August 2026, with the full high-risk obligations phased in further (the December 2025 to May 2026 "Digital Omnibus" process moved the Annex III high-risk rules to December 2027). A system where deterministic rules produce the decisions and the model only narrates them satisfies "comprehensible, interpretable, reproducible, logged" almost for free. A system where a model makes the call has to retrofit an explanation for a decision that, by construction, has no readable derivation. One of these is cheaper to certify than the other, and the gap widens the more regulated your customers are.
Even outside the regulated industries, the buyers worth having ask the same question in plainer words: can you show me why the system said that? "We can reproduce every figure and show you the rule behind every recommendation" is a sales advantage. "Our AI is very accurate, trust it" is the thing that loses the deal in the room.
The discipline, stated plainly
The principle is not anti-AI. It is the opposite: it is what lets you put AI in front of paying customers without flinching. The work is deciding, for every part of your system, which side of the line it sits on.
A short version of the discipline that has held up across the teams we work with:
- Decide which outputs must be correct. Numbers, decisions, anything that triggers money moving or an action being taken. Those are deterministic. Full stop.
- Let the model do what it is good at. Language, interpretation, retrieval, drafting, explanation. None of these require it to be the authority on a fact.
- Label model output as model output. Extracted or generated data is a draft until something deterministic or someone human confirms it. It never silently becomes source-of-truth.
- Make the truth reproducible. If you cannot read the rule and reproduce the result by hand, you do not yet have a system that will survive an audit.
- Keep the validator outside the model. The thing that checks correctness must be something the model cannot override.
The teams who get this right are not the ones with the most impressive demo. They are the ones who can answer "where did that number come from" without a pause. In 2026, with hallucination now understood as structural rather than incidental, that answer is the difference between a system you can build a business on and a system that works beautifully right up until the afternoon it doesn't. Getting that architecture right, the governance layer that keeps a model in its lane, is exactly the kind of problem good AI consulting exists to solve.
The model is a brilliant reader and a fluent writer. It is not your system of record. Build it so it never has to be. The companion argument, about what the data you own is and is not good for, is in your data moat is not your source of truth.
References
- OpenAI: Why Language Models Hallucinate (research summary, September 2025). https://openai.com/index/why-language-models-hallucinate/
- Adam Tauman Kalai, Ofir Nachum, Santosh Vempala, Edwin Zhang: Why Language Models Hallucinate (arXiv:2509.04664). https://arxiv.org/abs/2509.04664
- Arize AI: OpenAI's Santosh Vempala Explains Why Language Models Hallucinate. https://arize.com/blog/openais-santosh-vempala-explains-why-language-models-hallucinate/
- Incentives or Ontology? A Structural Rebuttal to OpenAI's Hallucination Thesis and Stemming Hallucination Using a Licensing Oracle (arXiv:2512.14801). https://arxiv.org/abs/2512.14801
- The shift from monolithic models to compound AI systems originates with Berkeley AI Research (Zaharia et al., 2024) and is documented in From Standalone LLMs to Integrated Intelligence: A Survey of Compound AI Systems (arXiv:2506.04565). https://arxiv.org/abs/2506.04565
- Databricks: What Are Compound AI Systems? https://www.databricks.com/blog/what-are-compound-ai-systems
- IBM: What Are Compound AI Systems? https://www.ibm.com/think/topics/compound-ai-systems
- EU Artificial Intelligence Act: Article 14: Human Oversight. https://artificialintelligenceact.eu/article/14/
- EU Artificial Intelligence Act: Article 26: Obligations of Deployers of High-Risk AI Systems. https://artificialintelligenceact.eu/article/26/
- European Commission: Regulatory Framework for AI (implementation timeline, including the 2026 Digital Omnibus). https://digital-strategy.ec.europa.eu/en/policies/regulatory-framework-ai
Written by JP, Sixees Labs. Last reviewed June 2026.