Hallucination

A language model confidently states that your firm's capital adequacy ratio is 14.2 per cent. The number is plausible, well-formatted, and completely fabricated. Hallucination is AI generating content that looks authoritative but has no basis in fact, and in financial services, a single hallucinated number in a regulatory filing or customer communication can trigger enforcement action.

What is hallucination?

Hallucination is the term for when a large language model generates content that is factually incorrect, unsupported by its input data, or entirely fabricated, while presenting it with the same confidence as accurate information. The model does not know it is wrong. It has no concept of truth. It generates the statistically most likely next token, and sometimes the most likely text is wrong.

The problem is structural, not incidental. Language models are trained to produce fluent, plausible text, not verified text. A model that generates "The FCA's 2024 enforcement actions totalled 312 million pounds" may be producing a plausible number that appears nowhere in any FCA publication. The output reads exactly like a factual statement. Without verification against a source, there is no way to distinguish it from one.

Hallucination rates vary by task and model. Factual question answering typically shows hallucination rates of 3 to 15 per cent depending on the domain and model. Summarisation is more reliable but can still introduce details not present in the source text. Code generation can produce syntactically valid code that calls functions that do not exist. The risk is proportional to how far the task strays from the model's training data.

The landscape

The EU AI Act requires that high-risk AI systems produce outputs that are "accurate, robust and cybersecure." Hallucination directly contradicts the accuracy requirement. For financial institutions using LLMs in high-risk contexts (credit decisions, insurance assessments, regulatory filings), demonstrating that hallucination risk is managed is a compliance obligation, not a quality preference.

The FCA's Consumer Duty requires that customer communications are fair, clear, and not misleading. A hallucinated statement in a customer communication is misleading by definition, even if no one intended it. The firm's responsibility extends to the outputs of its AI systems, regardless of whether a human or a model produced the text.

Model providers are improving but have not solved the problem. Newer models hallucinate less frequently than older ones. Techniques like reinforcement learning from human feedback (RLHF) and constitutional AI reduce but do not eliminate hallucination. No model provider guarantees factual accuracy. The responsibility for verification remains with the deploying institution.

How AI changes this

Grounding is the primary mitigation. By connecting the model to verified sources through retrieval-augmented generation, you constrain its outputs to information that exists in your knowledge base. The model is less likely to fabricate when it has relevant source material to draw from. Grounding reduces hallucination rates by 50 to 80 per cent in well-configured systems, but it does not eliminate them entirely.

Citation requirements force traceability. When the system must cite the specific document and section that supports each claim, hallucinated claims become visible: they either lack a citation or cite a source that does not support the claim. Building citation into the output format is the single most effective quality control for financial services applications.

Automated verification is emerging. A second model or a rule-based system checks the primary model's output against source data, flagging inconsistencies before the output reaches a human or a downstream system. For numerical claims (financial figures, regulatory thresholds, dates), automated verification against structured databases catches errors that a human reviewer might miss.

The practical approach is defence in depth. Grounding reduces hallucination. Citation makes it visible. Automated verification catches residual errors. Human review provides the final check. No single technique is sufficient. The combination produces outputs reliable enough for a regulated environment.

What to know before you start

Assume the model will hallucinate and design accordingly. Do not build systems that trust LLM output without verification. Every output that reaches a customer, a regulator, or a decision-maker must pass through at least one verification step. The specific verification depends on the use case: source citation for text, database cross-reference for numbers, human review for consequential communications.

Build evaluation datasets for your specific use cases. Generic hallucination benchmarks do not predict how the model will behave on your documents. Create test sets that include questions with known answers from your corpus, questions where the answer is not in the corpus (to test whether the model admits uncertainty), and adversarial questions designed to provoke hallucination. Run these tests regularly, not just at deployment.

Guardrails should enforce output constraints programmatically. If the model should only reference policy numbers that exist, validate the output against a list of valid policy numbers. If it should only cite figures from a specific report, cross-reference against that report's data. Programmatic checks are faster, cheaper, and more reliable than human review for structured verification.

Start by classifying your use cases by hallucination tolerance. Internal draft generation (high tolerance, human reviews everything) is different from customer-facing answers (low tolerance, every statement must be verifiable) which is different from risk calculations (zero tolerance, use deterministic systems instead). Match the verification investment to the risk. Not every use case needs the same level of protection.