Secure AI

Financial services firms are deploying AI into environments that contain the most sensitive data in the economy: transaction records, identity documents, health information for insurance, and regulatory filings. The security model that protects a traditional web application does not address the attack surface of an AI system. Secure AI is the discipline of designing, building, and operating AI systems that resist adversarial attack, protect the data they process, and maintain integrity under hostile conditions.

What is secure AI?

Secure AI encompasses the security practices, architectural patterns, and governance frameworks specifically designed for AI systems. It addresses risks that do not exist in traditional software: model extraction (an adversary reverse-engineers the model by querying it), training data poisoning (an adversary corrupts the data used to train the model), adversarial examples (inputs crafted to cause the model to produce incorrect outputs), and prompt injection (natural language inputs that manipulate the model's behaviour).

The challenge is that AI systems violate assumptions that traditional security relies on. A traditional application has a deterministic code path: given the same input, it produces the same output, and that behaviour can be tested exhaustively. An AI model's behaviour is probabilistic and dependent on training data. The same input can produce different outputs depending on model temperature, context window contents, and random seed. This non-determinism makes traditional testing approaches insufficient and requires new security methodologies.

In financial services, secure AI is not a feature; it is a condition of deployment. The PRA's expectations on operational resilience, the FCA's requirements on systems and controls, and the UK GDPR's requirements on data protection by design all apply to AI systems. A firm that cannot demonstrate the security of its AI deployments faces the same supervisory consequences as one that cannot demonstrate the security of its core banking platform.

The landscape

The OWASP Top 10 for LLM Applications and the MITRE ATLAS framework provide taxonomies of AI-specific threats. MITRE ATLAS catalogues adversarial techniques against machine learning systems in the same way that ATT&CK catalogues techniques against traditional IT systems. Financial services security teams should map their AI deployments against these frameworks to identify coverage gaps, just as they map their network defences against ATT&CK.

The supply chain risk for AI is distinct from traditional software supply chain risk. A financial services firm deploying an LLM depends on the model's training data (which the firm typically does not control), the model weights (which may contain embedded biases or memorised data), and the inference infrastructure (which may be shared with other tenants in a cloud environment). Each dependency is a potential attack vector. A compromised training dataset can produce a model that behaves correctly for most inputs but fails on specific, adversary-chosen scenarios.

The EU AI Act and the UK's evolving AI governance framework are creating legal requirements for AI security. The Act requires providers to implement risk management systems, ensure data quality, maintain technical documentation, and enable human oversight. For financial services firms deploying third-party models, the question of responsibility is critical: when the model provider's security fails, who bears the regulatory consequence?

How AI changes this

Secure-by-design architecture for AI systems places security controls at the architectural level, not the application level. The principle of least privilege limits the AI model's access to only the data and capabilities it needs for each specific task. A customer-facing chatbot that can answer balance queries should not have access to the full customer database, even if the underlying LLM could process it. Architectural separation between the AI model and the data layer enforces this boundary regardless of the model's behaviour.

Model hardening techniques reduce the model's vulnerability to specific attacks. Adversarial training exposes the model to adversarial examples during training, improving its robustness. Output sanitisation strips potentially sensitive information from model responses before they reach the user. Input validation rejects inputs that match known attack patterns. None of these techniques provides complete protection, but layered together they significantly reduce the attack surface.

Continuous security monitoring for AI systems tracks model behaviour for signs of compromise or drift. A model that suddenly produces outputs inconsistent with its expected behaviour may have been compromised (through a training data poisoning attack or a prompt injection) or may be experiencing a more mundane failure (data distribution shift or infrastructure issue). Either way, the detection and response must be automated and fast. This is where AI observability and security monitoring converge.

What to know before you start

Treat AI systems as critical infrastructure from the security perspective. Apply the same security governance, risk assessment, and incident response processes that you apply to core banking systems. The temptation to treat AI deployments as "just another application" ignores the novel attack surface and the sensitivity of the data they process. If the AI system accesses customer data, it is as critical as the system that stores it.

Security assessment must be AI-specific. A traditional penetration test will not find prompt injection vulnerabilities, training data poisoning, or model extraction attacks. Supplement your existing security testing programme with AI red teaming that specifically targets AI vulnerabilities. The expertise required is different from traditional application security testing, and the testing methodology is still evolving.

Vendor security due diligence for AI must cover model-specific risks. Ask the model provider: how was the model trained? What data was used? What adversarial testing was conducted? How are model updates secured? What happens if a vulnerability is discovered in the model? The traditional vendor security questionnaire does not cover these questions, and the answers matter for a firm deploying the model in a regulated environment.

Start with a threat model for each AI deployment. Enumerate the assets (data, model, outputs), the threats (injection, extraction, poisoning, evasion), and the existing controls. Identify the gaps and prioritise remediation based on impact and likelihood. A threat model that takes a day to produce is more valuable than a comprehensive security programme that takes a year to design and never ships.