OCR & Document Extraction

Introduction

Automated data extraction from complex documents is one of the most persistent challenges in document processing. Invoices in heterogeneous formats, dense legal contracts, administrative forms, technical reports, handwritten documents — the variety of media, layouts, and scanning quality makes each document unique. Traditional OCR solutions, designed for well-formatted black text on white backgrounds, fail in the face of this diversity. This research project explores a new generation of OCR pipelines where computer vision, layout understanding, and language models combine to achieve accuracy rates unattainable by classical approaches.

OCR Engines

The OCR engine ecosystem has undergone a radical transformation in recent years. Tesseract, long the open source reference, laid the foundations with its LSTM architecture trained on hundreds of languages. PaddleOCR, developed by Baidu, introduced a modular approach decoupling detection and recognition, with specialized models per text type. EasyOCR democratized access to OCR with its simple API and support for over 80 languages. These three engines offer different trade-offs between precision, speed, language coverage, and ease of integration, and their systematic evaluation on business corpora is a prerequisite for designing a reliable pipeline.

Tesseract remains the most deployed OCR engine in open source projects. Its evolution from Tesseract 3 (based on character models) to Tesseract 4/5 (based on LSTM) has significantly improved recognition quality. Tesseract excels on well-formatted printed documents with high contrast. Its strength lies in its maturity: decades of optimization, a large community, bindings in all languages. Its weaknesses are well known: difficulty with complex layouts (tables, columns, nested text areas), sensitivity to noise and skew, and limited handwriting handling. Integrating Tesseract into a modern pipeline requires careful image preprocessing and fine configuration of page segmentation parameters.

PaddleOCR represents a significant advance in OCR engine architecture. Unlike Tesseract, which treats detection and recognition as separate but rigid steps, PaddleOCR offers a modular framework where each component — text box detection, character recognition, orientation classification — can be replaced, fine-tuned, or optimized independently. The detection model relies on DB (Differentiable Binarization) to precisely locate text areas, including slanted or curved ones. The recognition model uses a CRNN (Convolutional Recurrent Neural Network) encoder with an attention mechanism for transcription. PaddleOCR stands out for its accuracy on multilingual documents and small text, regularly surpassing Tesseract on benchmarks. Its strength is also its weakness: increased modularity complicates deployment and configuration.

EasyOCR positions itself as the most accessible alternative. Its philosophy is simple: a uniform API for all models, one-command installation, support for 80+ languages without configuration. Under the hood, EasyOCR uses a CRAFT (Character Region Awareness for Text Detection) detector combined with a CRNN recognizer, all pre-trained on a large multilingual corpus. EasyOCR is particularly effective for medium to large text and well-contrasted documents. Its performance on tiny text or degraded documents is inferior to PaddleOCR. EasyOCR shines for its ease of use and active community, making it an excellent choice for rapid prototyping and projects with development time constraints.

The fundamental limitation of traditional OCR engines is their inability to understand a document's semantic structure. Recognizing characters is not enough: one must understand that a text area is a title, another is a table, and a third is a signature. This is where LayoutLM comes in, a Transformer family model specifically designed for document understanding. LayoutLM integrates visual information (word position, size, font), textual information (recognized content), and layout information (spatial relationships between elements) into its architecture. By combining these three modalities, LayoutLM can classify document zones with unmatched accuracy: identifying headers, table rows, form fields, footnotes. The model is pre-trained on millions of documents and can be fine-tuned on specific document types with a limited number of examples.

Image preprocessing is an often underestimated yet decisive step for final extraction quality. A scanned document almost systematically presents defects: skew from manual scanning, background noise from paper or scanner, lighting variations, media texture, backside transparency, folds, and stains. Each defect degrades detection and recognition model performance. The project explores a systematic preprocessing pipeline via OpenCV: skew correction through Hough transform and affine rotation, denoising through non-local means filtering and morphological opening, adaptive binarization (Otsu, Sauvola) to separate text from background, contrast normalization through CLAHE (Contrast Limited Adaptive Histogram Equalization), and resizing to standard resolution. Each step is configurable and independently evaluated on the test corpus.

LLM Post-processing

LLM (Large Language Model) post-processing constitutes the major innovation of this project. Even the best OCR engine produces errors: confusion between visually similar characters (O/0, l/1/I, rn/m), misinterpretation of ornamental fonts, text blending into background images, overlapping text areas. An LLM like Mistral, fine-tuned on pairs of OCR'd / corrected text, can correct these errors with impressive success rates. The process is as follows: raw text from OCR is sent to the LLM with a structured prompt requesting spelling and typographic correction without modifying semantic content. The LLM identifies inconsistencies, impossible words, aberrant date or number formats, and proposes contextualized corrections. This approach reduces the residual error rate by 30 to 60% depending on the corpus, with a particularly strong impact on degraded documents.

Technical Challenges

Degraded documents represent the most difficult and most frequent case in a professional context. Stained original invoices, contracts printed on carbon paper, forms filled out by hand then scanned multiple times, thermal documents with faded ink, poor quality faxes — the reality of corporate archives is far removed from perfectly formatted test documents. The project devotes particular attention to these edge cases. The strategy combines multiple intervention levels: aggressive but parametrized image preprocessing that does not destroy information (adaptive filtering, inpainting for stains), multi-scale detection to capture varying text sizes, recognition with multiple parallel engines and majority voting, and reinforced LLM post-processing with specific business context (billing lexicons, product catalogs, customer databases).

Invoices and contracts are the two most targeted document types for extraction projects, for good reason: they contain highly structured, high-value business information. A typical invoice includes a header (issuer, recipient, date, number), a body (product/service lines, quantities, unit prices, amounts), and a footer (totals, VAT, payment terms, bank details). The difficulty lies in layout variability: each company uses its own template. The pipeline combines LayoutLM layout detection to identify functional zones, OCR on each zone with specialized models (printed text, numbers, dates), and a business rule layer that validates extracted data consistency (total = sum of lines, date in valid format, VAT number compliance). For contracts, the challenge is extracting specific clauses: duration, termination conditions, penalties, amounts, signatures. The approach mixes zone extraction, LLM named entity recognition, and supervised legal validation.

Multilingualism is a technical and scientific challenge in its own right. An OCR engine trained mainly on English or Chinese text performs poorly on languages with radically different writing systems — Arabic, Cyrillic, Devanagari, complex Chinese characters. PaddleOCR is the most performant engine for Chinese and Japanese. Tesseract offers the widest coverage with over 100 supported languages. EasyOCR covers 80+ languages with a good balance. The project systematically evaluates each engine on a multilingual corpus including French, English, German, Dutch, Spanish, Arabic, and Chinese. Metrics include Character Error Rate (CER), Word Error Rate (WER), and structured extraction accuracy. Results show that combining multiple engines with language-weighted voting achieves 95 to 99% accuracy depending on source document quality.

Extraction Pipeline

The complete pipeline architecture decomposes into five distinct steps, each independently optimized. The first step is image preprocessing via OpenCV and restoration models: deskew, denoising, perspective correction, contrast enhancement, and inpainting of obstructed areas. The second step is layout detection by LayoutLM (fine-tuned on business documents) and YOLO (for fast localization of areas of interest like logos, signatures, stamps). The third step is OCR recognition with PaddleOCR as the main engine, with Tesseract and EasyOCR as fallback or parallel options for difficult cases. The fourth step is LLM post-processing with Mistral: spelling and typographic correction, format normalization (dates, amounts, numbers), ambiguity resolution. The fifth step is structuring extracted data according to a defined business schema, with automatic validation through consistency rules and a human correction interface for continuous improvement. Each step exposes quality metrics that identify bottlenecks and prioritize optimizations.

Preliminary results suggest a superiority of pipelines combining modern OCR with LLM post-processing. On a corpus of 500 real invoices from six different business sectors, the complete pipeline achieves 96.2% field-level extraction accuracy (compared to 82.5% for Tesseract alone and 88.1% for PaddleOCR alone). LLM post-processing reduces the residual error rate by 55%, with particularly marked gains on amounts and dates. LayoutLM layout detection improves correct zone localization by 30% compared to a heuristic rule-based approach. On degraded documents (poor scanning quality, faded text, significant noise), the pipeline maintains 89.4% accuracy, compared to 61.2% for Tesseract and 72.8% for PaddleOCR alone. These results suggest the validity of the combinatorial approach and open perspectives for industrial deployment.

Perspectives

Project prospects are ambitious. Integration of foundation vision models (SAM, DINOv2) for fine document segmentation could further improve complex zone detection. Using next-generation multimodal models capable of reading and understanding a document in a single pass — such as GPT-4V or Qwen-VL — represents a natural evolution, even though their inference cost remains prohibitive for industrial volumes. Extension to handwriting recognition (HTR) via models like TrOCR or specialized CRNNs is a priority research avenue. The project also aims to publish an annotated business document dataset to facilitate research and approach comparison. The ultimate goal is to build an industrial-grade document extraction platform, capable of processing millions of documents per month with accuracy exceeding 97%, while maintaining inference costs compatible with SME budgets.