AI Observability

A credit scoring model in production makes 50,000 decisions per day. Last month, its approval rate for applicants under 25 dropped by 12 percentage points. Nobody noticed for three weeks. The model had not been changed. The input data had shifted. By the time the drift was detected, thousands of applicants had received decisions based on a model that was no longer performing as validated. AI observability is the practice of instrumenting AI systems so that these failures surface in minutes, not weeks.

What is AI observability?

AI observability is the ability to understand the internal state and behaviour of an AI system from its external outputs. Whether you are monitoring a credit scoring model or a customer-facing chatbot, the principle is the same: instrument the system so that failures surface before they cause harm. It extends traditional software observability (monitoring logs, metrics, and traces) to AI-specific concerns: model accuracy over time, prediction drift, data distribution shift, fairness metrics across protected groups, latency, token usage, hallucination rates, and the relationship between model inputs, outputs, and the outcomes they produce.

In financial services, observability is a regulatory expectation, not a best practice. The PRA's model risk management framework (SS1/23) requires firms to monitor model performance on an ongoing basis, detect degradation, and take corrective action. A model that was validated at deployment and then not monitored fails this requirement. The validation is a point-in-time assessment. Observability is the continuous assurance that the model continues to perform as validated.

The challenge is that AI failures are often silent. A traditional software bug causes an error message or a crash. A model drift causes slightly worse decisions, gradually, across thousands of cases. No individual decision looks obviously wrong. The aggregate effect, higher default rates, more complaints, biased outcomes, appears in business metrics weeks or months later. Observability closes this feedback gap by detecting drift at the model level before it manifests at the business level.

The landscape

The EU AI Act mandates post-market monitoring for high-risk AI systems, including systems used for credit scoring, insurance pricing, and AML screening. Providers and deployers must establish monitoring systems proportionate to the nature of the AI system and its risks. This is not aspirational guidance. It is a legal requirement with enforcement mechanisms. Financial services firms deploying high-risk AI must have observability infrastructure in place before the compliance deadline.

The MLOps tooling ecosystem has matured rapidly. Platforms for model monitoring, experiment tracking, and deployment management have moved from startup novelty to enterprise readiness. Tools like MLflow, Weights & Biases, Evidently AI, and proprietary platforms from cloud providers offer observability capabilities out of the box. The challenge for financial services is not the availability of tooling but its integration with existing governance and risk management frameworks. A monitoring dashboard that the data science team watches but the model risk team does not is insufficient.

The scope of AI observability is expanding beyond traditional ML models to include LLM-based systems. Monitoring a credit scoring model means tracking prediction accuracy and distribution stability. Monitoring a customer-facing LLM means tracking response quality, hallucination frequency, refusal rates, latency, prompt injection attempts, and compliance with content guardrails. The observability requirements for generative AI are more complex and less standardised than for traditional ML.

How AI changes this

Automated drift detection monitors the statistical distribution of model inputs and outputs, alerting when they diverge from the baseline established during validation. If the distribution of applicant ages, incomes, or credit scores shifts, the model may be operating outside its validated range. The alert triggers investigation: is the shift a genuine change in the applicant population, or a data quality issue? The system detects. The human diagnoses.

Fairness monitoring tracks model outcomes across protected groups in real time. If the approval rate for a specific demographic segment drops disproportionately, the system flags it before the pattern becomes entrenched. This is essential for compliance with the Equality Act and for meeting the FCA's expectations on fair treatment. Quarterly fairness audits are not sufficient when the model makes thousands of decisions per day. Continuous monitoring catches bias as it emerges.

LLM-specific observability tracks the metrics that matter for generative AI: response relevance, factual grounding (whether the response is supported by retrieved documents), toxicity and safety scores, and user satisfaction signals. For financial services, the critical metric is accuracy: does the model's response align with the source material? A compliance copilot that confidently cites a regulation incorrectly is worse than one that says "I don't know." Observability catches the confident errors.

Root cause analysis traces failures back to their origin. When a model's performance degrades, was it a data issue (upstream system changed its output format), a model issue (concept drift in the underlying relationship), or an infrastructure issue (latency spike causing timeout and fallback behaviour)? Observability platforms that correlate model metrics with data pipeline health and infrastructure metrics enable faster diagnosis and resolution. The integration with data governance frameworks ensures data quality issues are surfaced through the same monitoring infrastructure.

What to know before you start

Define the metrics before deploying the model. What does "good performance" look like for this specific model? Accuracy is not always the right metric. For a fraud detection model, false negative rate matters more than overall accuracy. For a customer-facing LLM, hallucination rate and response latency matter more than perplexity. Define thresholds that trigger investigation and thresholds that trigger automatic rollback, and document these as part of the model's risk appetite.

Connect model monitoring to business outcomes. A model metric dashboard that shows "accuracy: 92 per cent" is meaningless without context. Connect it to the business outcome: "approval rate within target range," "default rate below threshold," "complaint rate stable." When model metrics and business metrics diverge (the model looks fine but defaults are rising), the observability system should surface the discrepancy.

Observability infrastructure must be independent of the model it monitors. If the monitoring system relies on the same infrastructure as the model, an infrastructure failure takes out both the model and the monitoring simultaneously. This is the equivalent of a fire alarm that fails in a fire. Deploy monitoring on separate infrastructure with independent alerting. For AI systems that access sensitive data, observability also serves a security function: detecting anomalous model behaviour that may indicate a prompt injection attack or data extraction attempt.

Start with your highest-risk model: the one making the most decisions, using the most sensitive data, or carrying the most regulatory scrutiny. Instrument it fully: input distributions, output distributions, fairness metrics, latency, and error rates. Set alerting thresholds based on your model validation baseline. Once you have established the observability pattern for one model, extending it to additional models is an engineering exercise, not a design exercise. The first model teaches you what to monitor. The rest is repetition.