Intelligent Document Processing (IDP)

A compliance team processes 50,000 documents per month. Ten per cent require re-keying because the first extraction was wrong. The re-keying costs more than the original processing because someone must find the error, trace it back to the source, and correct it across every downstream system it has touched. Intelligent document processing (IDP) aims to get the extraction right the first time, and in financial services, "right" means accurate enough to satisfy the regulator.

What is intelligent document processing?

Intelligent document processing is the end-to-end automation of document-centric workflows: ingestion, classification, extraction, validation, and integration with downstream systems. It combines OCR, machine learning, natural language processing, and workflow orchestration into a pipeline that processes documents from receipt to actionable data without manual intervention on the happy path. IDP is the broader operational discipline within which Document AI provides the intelligence layer.

The "intelligent" distinction matters. Traditional document processing uses templates and rules: if the document is a specific form, extract the field at position X on the page. IDP uses machine learning to understand document structure and content, handling format variations, layout changes, and document types it has not seen before. This adaptability is what makes IDP viable for financial services, where the variety of incoming documents is too great for template-based approaches to cover.

The business case is straightforward. Financial institutions process millions of documents annually across customer due diligence, claims, loan origination, trade finance, and regulatory compliance. Manual processing costs 5 to 15 pounds per document depending on complexity. IDP reduces this to 0.50 to 2 pounds per document for document types within its capability. The ROI calculation depends on volume, accuracy requirements, and the cost of exceptions that fall through to manual processing.

The landscape

The market has consolidated around platforms that combine pre-built models with customisation capabilities. Vendors like ABBYY, Kofax, and Hyperscience offer IDP platforms with pre-trained models for common financial document types. Cloud providers (AWS Textract, Azure Document Intelligence, Google Document AI) offer extraction services that can be assembled into IDP pipelines. The choice between platform and build-your-own depends on the institution's volume, variety of document types, and integration requirements.

Generative AI has disrupted the traditional IDP approach. Pre-2023, IDP required training custom models for each document type, a process that took weeks to months per type. LLMs can extract information from documents they have never been trained on by understanding the content semantically. This compresses the time-to-value for new document types from months to days. The trade-off is cost per document: LLM inference is more expensive than a trained extraction model for high-volume, standardised documents.

Regulatory expectations on automated document processing are implicit but clear. The FCA expects firms to maintain accurate records and to demonstrate that their processes are reliable. The EU AI Act classifies AI systems used in creditworthiness assessment and other financial decisions as high risk, which extends to the document processing systems that feed data into those decisions. An IDP system that introduces errors into customer records or regulatory filings creates compliance risk. Validation and quality monitoring are not optional components of an IDP deployment.

How AI changes this

Adaptive extraction learns from corrections. When a human reviewer corrects an extraction error, the feedback is captured and used to improve future extractions of the same document type. Over time, the system's accuracy on each document type improves, and the proportion of documents requiring human review decreases. This continuous learning loop is what distinguishes IDP from static template-based extraction.

Cross-document understanding processes document packages as a whole rather than individual documents in isolation. A mortgage application package, a CDD case file, or a trade finance document set contains documents that reference each other. The applicant's name on the identity document should match the name on the bank statement and the name on the employment letter. IDP systems that validate consistency across a package catch errors that document-by-document processing misses.

Automated exception handling reduces the proportion of documents that fall through to manual processing. When an extraction confidence score falls below the threshold, the system can attempt alternative extraction strategies: different model, different preprocessing, or targeted queries to a language model. Only documents that fail all automated strategies reach a human reviewer. This second-pass automation can resolve 30 to 50 per cent of first-pass exceptions.

Integration with workflow orchestration creates end-to-end automation. The extracted data flows directly into case management, screening, or decisioning systems via APIs. A CDD document triggers sanctions screening automatically. A claims document triggers reserve estimation. The IDP system is the sensor; the downstream process is the actuator. Neither delivers full value without the other.

What to know before you start

Measure straight-through processing rate, not extraction accuracy alone. A system with 97 per cent field-level accuracy may still require human review on 30 per cent of documents because the errors concentrate on specific fields or document types. The metric that matters is the proportion of documents that pass through the entire pipeline without human intervention and without introducing errors into downstream systems.

Document preparation affects extraction accuracy more than model sophistication. Poor-quality scans, photographed documents at angles, and faxed documents with degraded resolution all reduce accuracy. Investing in document capture quality (better scanners, clearer submission guidelines for customers, rejection of sub-standard images) is often more cost-effective than investing in more sophisticated extraction models.

Build for the exception, not just the happy path. The documents that flow through without issues are not where the cost lies. The cost lies in the 10 to 20 per cent that require manual intervention. Design the human review interface to be fast and ergonomic, presenting the source document alongside the extracted data, highlighting uncertain fields, and enabling one-click corrections. The efficiency of exception handling determines the overall economics of the IDP deployment.

Start with a single document type at high volume. Bank statements, invoices, or identity documents from major issuing countries are common starting points. Measure the end-to-end processing cost (including exceptions) against your current manual cost. Only expand to additional document types once the pipeline is stable and the economics are proven. Expanding too quickly across document types dilutes focus and delays the point where any single type reaches production-grade accuracy.