Model Risk Management

When a pricing model underestimates risk by 3 per cent across a portfolio of 50,000 policies, the aggregate loss runs into tens of millions. The model passed validation. The data was clean. The error was in an assumption that nobody re-examined for two years. Model risk management exists because models are opinions expressed in code, and opinions can be wrong in ways that only become visible when losses accumulate.

What is model risk management?

Model risk management (MRM) is the discipline of identifying, measuring, monitoring, and controlling the risk that arises from using models in business decisions. A model, in regulatory terms, is any quantitative method that processes data to produce estimates, forecasts, or scores used in decision-making. This includes credit scoring models, pricing algorithms, risk calculators, stress testing tools, and increasingly, machine learning systems used for fraud detection, claims triage, and customer segmentation.

Model risk materialises in two ways. First, a model can produce inaccurate outputs because of flawed assumptions, coding errors, or data quality problems. Second, a model can be used inappropriately: applied to a population it was not designed for, used beyond its validated range, or relied upon without understanding its limitations. Both forms of model risk have caused significant losses in financial services. The 2012 "London Whale" incident at JPMorgan, where a flawed value-at-risk model obscured trading losses exceeding six billion dollars, remains the most cited example.

The MRM framework encompasses model inventory, development standards, independent validation, ongoing performance monitoring, and governance structures with clear accountability. It applies to every model in the institution, not just the most complex ones. A simple regression model used in regulatory reporting carries model risk just as a deep learning fraud detection system does.

The landscape

The PRA's Supervisory Statement SS1/23 on model risk management principles for banks took full effect from May 2024. It builds on the US Federal Reserve's SR 11-7 guidance but goes further, requiring firms to maintain a comprehensive model inventory, establish clear model ownership, conduct independent validation, and implement ongoing monitoring for all material models. The scope explicitly includes AI and machine learning models.

For insurers, the PRA applies equivalent expectations through Solvency II internal model requirements and broader supervisory expectations. The principles of SS1/23 are relevant to any model that informs a material business or regulatory decision, regardless of whether the firm is a bank or an insurer. The FCA has signalled alignment with these principles for conduct-related models, including those used in consumer outcomes assessment under the Consumer Duty.

The challenge for most institutions is not understanding the requirements but implementing them at scale. A large bank or insurer may have thousands of models in production. Maintaining an accurate inventory, scheduling validations, monitoring performance, and governing changes across this portfolio requires dedicated resource and infrastructure. Many firms are still building the foundational MRM capabilities that SS1/23 demands.

How AI changes this

Machine learning models introduce specific MRM challenges that traditional frameworks were not designed for. ML models can be opaque, making it harder to understand why they produce specific outputs. They can drift as the data distribution changes, degrading in ways that are not visible without active monitoring. And they can be sensitive to adversarial inputs, producing incorrect outputs when presented with deliberately manipulated data.

Automated model monitoring addresses the drift problem. Platforms that continuously track model performance metrics, input data distributions, and output distributions alert the model owner when behaviour changes. For an ML-based fraud detection model, this means detecting when a new fraud pattern emerges that the model has not seen, rather than waiting for losses to accumulate before retraining.

Explainability tools generate the interpretability that MRM frameworks require. Feature importance analysis, counterfactual explanations, and partial dependence plots help validators understand what a model has learned and whether it is learning the right things. These tools do not eliminate the complexity of ML model validation, but they make it tractable.

Automated documentation generation captures model metadata, validation results, and performance history in a structured format that supports regulatory review. A model risk report that would take a validator days to compile manually can be generated automatically from the monitoring platform, freeing specialist resource for the judgemental aspects of validation that require human expertise.

What to know before you start

The model inventory is the foundation. If you do not have a complete, accurate inventory of all models in production, including vendor-supplied models and spreadsheet-based models, every other MRM activity is compromised. Start here. The inventory exercise consistently reveals models that were unknown to the central risk function, and some of those unknown models carry material risk.

Model validation for ML models requires different skills than traditional model validation. Validators need to understand gradient boosting, neural networks, feature engineering, and the specific ways ML models can fail. Invest in training your validation team or supplement them with data science expertise. A validation function that cannot challenge ML models effectively provides false assurance.

Vendor models are in scope. A credit scoring model purchased from a bureau or a fraud detection model embedded in a vendor platform is still a model used in your business decisions. SS1/23 requires firms to validate vendor models to the same standard as internally developed ones, though the validation approach may differ. Do not assume that the vendor's validation is sufficient for your regulatory obligations.

Start with a gap assessment against SS1/23's five principles: model identification and inventory, model development, validation, model use, and governance. The gap assessment reveals where your current MRM framework falls short and provides a prioritised remediation plan. Engage your PRA supervisor on the timeline for closing gaps, demonstrating a credible plan rather than waiting for a supervisory finding.