Model Interpretability

Two models predict the same outcome with identical accuracy. One is a logistic regression with twelve features. The other is a neural network with three hidden layers. The regulator asks: how does the model make decisions? For the first, you can answer in minutes. For the second, it takes a team of specialists and a set of approximation tools. Model interpretability is the property that determines how easily a model's internal logic can be understood, and in regulated financial services, it is a design constraint, not an afterthought.

What is model interpretability?

Model interpretability is the degree to which a human can understand the cause of a model's decisions. An interpretable model reveals its reasoning transparently: the relationship between inputs and outputs is visible in the model's structure. This is distinct from explainability, which uses external tools to approximate the reasoning of models that are not inherently interpretable. The distinction matters because interpretable models provide faithful explanations by design, while post-hoc explanations of opaque models can be approximate or misleading.

In financial services, interpretability supports multiple objectives: regulatory compliance, model validation, operational trust, and customer communication. A credit risk officer who understands why a model flags a specific borrower can act on the signal with confidence. A validator who can trace the model's logic can identify flaws that performance metrics alone would miss. A regulator who can inspect the model's structure can assess whether it meets supervisory expectations.

The spectrum runs from fully transparent (a decision tree you can read) to fully opaque (a large neural network). Between these poles sit models with varying degrees of interpretability: generalised additive models, rule lists, Bayesian networks, and constrained gradient-boosted models. The choice of where to sit on this spectrum is a design decision that should be made at the start of a project, not retrofitted at deployment.

The landscape

The PRA's SS1/23 does not prohibit complex models, but it requires that models be understood by their users and validators. For models used in capital calculations or regulatory reporting, this understanding must be demonstrable to the supervisor. The practical effect is that interpretability is a factor in model approval, and models that cannot be adequately explained face higher governance costs and longer approval timescales.

The FCA's Consumer Duty creates a parallel obligation for consumer-facing models. If a model's decision cannot be explained to the affected customer in meaningful terms, the firm may not be delivering good outcomes. This does not require the model itself to be interpretable, but interpretable models make compliance simpler and less expensive than opaque models with post-hoc explanation layers.

Industry practice is converging on a pragmatic position: use interpretable models for consumer-facing decisions and reserve complex models for internal applications where accuracy gains justify the governance overhead. This approach is not universal, but it is the emerging consensus among institutions that have weighed the total cost of model ownership, including validation, monitoring, and regulatory engagement, not just development accuracy.

How AI changes this

Advances in inherently interpretable ML have narrowed the accuracy gap. Explainable Boosting Machines (EBMs), which are generalised additive models with pairwise interactions, achieve performance competitive with gradient-boosted trees while maintaining full transparency. Each feature's contribution is visible as a shape function, and interactions between features are explicitly modelled and inspectable. For tabular data common in financial services, EBMs represent a significant advance in the interpretability-accuracy trade-off.

Constrained learning allows complex models to incorporate domain knowledge directly. Monotonicity constraints ensure that a credit model assigns higher risk to higher debt levels, even if the training data contains noise. Interaction constraints limit which features can combine, preventing the model from learning spurious relationships. These constraints reduce the model's flexibility but improve its reliability and interpretability.

Automated model simplification distils a complex model into a simpler approximation that preserves most of the accuracy. A neural network trained on millions of records can be approximated by a scorecard that captures 95 per cent of the predictive power in a fully transparent format. This approach uses the complex model for learning and the simple model for deployment, combining the strengths of both.

Model risk management benefits directly from interpretability. Validators can audit an interpretable model in days rather than weeks. Monitoring is simpler because deviations in the model's logic are directly visible. The total cost of ownership for an interpretable model, across its full lifecycle, is substantially lower than for an opaque one of equivalent accuracy.

What to know before you start

Define the interpretability requirement before selecting the modelling approach. The requirement varies by use case: a customer-facing credit decision requires individual-level interpretability that can be communicated to consumers. A portfolio risk model requires global interpretability that can be explained to the board and the regulator. An internal routing model may require minimal interpretability. Let the requirement drive the method.

Test the accuracy gap on your own data before assuming that complex models are necessary. Many teams default to gradient-boosted trees or neural networks without benchmarking against interpretable alternatives. Run the comparison on your actual dataset, with your actual features, and measure the gap. In many financial services applications, it is smaller than expected.

Interpretability does not guarantee correctness. A transparent model can be transparently wrong. A decision tree that splits on postcode may be perfectly interpretable and perfectly discriminatory. Interpretability makes errors visible, but fairness testing and validation are still required to ensure the model's logic is appropriate.

Start with the models that face the highest regulatory and customer scrutiny: credit decisions, insurance pricing, and claims triage. Migrate these to interpretable architectures where the accuracy trade-off is acceptable. Use the governance savings, faster validation, simpler monitoring, easier regulatory engagement, to build the case for interpretability as a default design principle across the organisation.