Explainable AI (XAI)

A customer's mortgage application is declined. The letter says "your application did not meet our criteria." The customer asks why. Nobody can tell them, because the model that made the decision is a 200-tree gradient-boosted ensemble that no single person fully understands. This is the problem that explainable AI (XAI) exists to solve, and in financial services, the ability to explain is not a feature. It is a regulatory requirement.

What is explainable AI?

Explainable AI refers to techniques and methods that make the outputs of AI systems understandable to humans. This ranges from global explanations (what does this model do in general?) to local explanations (why did this model produce this specific output for this specific input?). In financial services, explainability serves three distinct audiences: the customer who receives a decision, the regulator who oversees the institution, and the internal teams who operate and validate the model.

The technical approaches fall into two categories. Intrinsically interpretable models, such as logistic regression, decision trees, and scorecards, are understandable by design. Their structure reveals the relationship between inputs and outputs directly. Post-hoc explanation methods, such as SHAP (SHapley Additive exPlanations), LIME (Local Interpretable Model-agnostic Explanations), and counterfactual explanations, generate explanations for models that are not inherently interpretable, including neural networks and ensemble methods.

The trade-off between accuracy and interpretability is real but frequently overstated. For many financial services applications, well-engineered interpretable models perform within 1 to 2 percentage points of complex black-box alternatives. The marginal accuracy gain from opacity is rarely worth the governance, compliance, and operational costs it introduces. The institutions achieving the best outcomes use interpretable models where possible and reserve complex models for problems where the accuracy gap genuinely justifies the explainability overhead.

The landscape

The EU AI Act requires that high-risk AI systems provide sufficient transparency for users to interpret and use outputs appropriately. For credit scoring, this means the applicant and the institution must be able to understand the basis for a decision. The Act does not prescribe specific technical methods for explainability, but it creates a legal standard that post-hoc explanations must meet.

The FCA's feedback statement on AI in financial services (2024) emphasises that firms must be able to explain AI-driven decisions to consumers in a meaningful way. "Meaningful" is the operative word: a list of feature importance scores is not a meaningful explanation to a consumer. The explanation must be actionable, telling the customer what they could change to achieve a different outcome, and accurate, reflecting the actual basis for the decision.

The PRA's SS1/23 requires that models used in business decisions be understood by their users and validators. For ML models, this means the institution must be able to articulate what the model has learned, how it behaves across different segments, and where its outputs are most uncertain. This understanding is tested during supervisory reviews and internal model validation.

How AI changes this

SHAP values have become the standard for local explanations in financial services. They decompose a model's output for a specific input into contributions from each feature, grounded in game theory. For a credit decision, SHAP can identify that the primary drivers of a decline were high utilisation, short credit history, and recent applications, providing the basis for a customer explanation. SHAP works with any model type, making it the default choice for institutions deploying diverse model architectures.

Counterfactual explanations tell the customer what would need to change for the decision to be different. "Your application would have been approved if your credit utilisation were below 40 per cent" is more actionable than "credit utilisation was the most important factor." Counterfactual methods are particularly valuable for consumer-facing explanations and are gaining traction as a complement to feature importance approaches.

Concept-based explanations are emerging for complex models. Rather than explaining at the feature level (income, age, postcode), these methods explain in terms of higher-level concepts (financial stability, payment reliability) that are more meaningful to both customers and business users. This requires mapping features to business concepts, which is a domain exercise, not a purely technical one.

Model validation teams use explainability tools to challenge model behaviour. A validator who can see that a model relies heavily on a feature that should not be predictive (a branch code, an application timestamp) can identify issues that accuracy metrics would not reveal. Explainability is not just a compliance tool; it is a quality assurance tool.

What to know before you start

Choose the explainability method based on the audience, not the model. A regulator reviewing model risk management needs global feature importance and segment-level analysis. A customer needs a plain-language reason for their decision. An internal operator needs alert-level explanations that support triage decisions. One method will not serve all three.

Post-hoc explanations can be unfaithful. SHAP and LIME approximate the model's reasoning; they do not reveal it directly. In edge cases, the explanation may not accurately represent why the model produced a specific output. For high-stakes decisions, test the fidelity of your explanations against known cases. If the explanation does not match what the model actually does, it is worse than no explanation because it creates false confidence.

Intrinsically interpretable models should be the default for customer-facing decisions. A well-engineered scorecard or logistic regression produces explanations that are inherently faithful, that require no approximation, and that can be communicated to customers without a technical intermediary. Reserve complex models and post-hoc explanations for internal use cases where the accuracy gain justifies the governance cost.

Start with customer-facing decisions where the explanation obligation is clearest: credit, insurance pricing, and claims. Build the explanation pipeline, from model output to customer communication, before deploying the model. If the explanation cannot be generated, formatted, and delivered to the customer within the decision workflow, the model is not ready for production.