TECHNICAL_ARCHITECTURE.md

Technical Architecture & Implementation Guide

This document provides a deep dive into the internal workings of the IDP (Intelligent Document Processing) system. It explains the "how" and "why" behind the implementation decisions.

1. System Architecture

The system follows a modern decoupled architecture:

graph TD
    User[User] -->|Uploads File| Frontend[Next.js Frontend]
    Frontend -->|POST /process| API[FastAPI Backend]
    
    subgraph "Inference Pipeline"
        API -->|Raw Image/PDF| Preprocessor[Preprocessing Module]
        Preprocessor -->|Cleaned Image| OCR[OCR Engine]
        OCR -->|Raw Text + BBoxes| Classifier[Document Classifier]
        OCR -->|Raw Text| NER[NER Model]
        
        Classifier -->|Doc Type| PostProcessor[Post-Processing]
        NER -->|Entities| PostProcessor
        OCR -->|Text + BBoxes| PostProcessor
        
        PostProcessor -->|Structured JSON| API
    end
    
    API -->|JSON Response| Frontend
    Frontend -->|Render| UI[Results Display]

2. Component Deep Dive

A. Frontend (/frontend)

Built with Next.js 14 (App Router), React 18, and Tailwind CSS.

• app/page.tsx: The main entry point. It manages the global state for result, loading, and isSystemOnline. It uses a polling mechanism (setInterval) to check the backend health status every 30 seconds.
• components/DocumentUpload.tsx: Handles file selection and drag-and-drop.
  • Auto-Processing: As soon as a valid file is selected, it automatically triggers the upload.
  • Validation: Checks file size (<10MB) and type (PDF, PNG, JPEG).
• components/ResultsDisplay.tsx: Renders the JSON response.
  • Dynamic Badges: Uses getDocTypeStyle to color-code document types (Purple for Invoice, Pink for Receipt, Blue for Bank Statement).
  • Confidence Indicators: Color-codes confidence scores (Green > 80%, Yellow > 60%, Red < 60%).
• Styling: Uses globals.css for custom animations (animate-fade-in-up) and utility classes (glass-panel, bento-card).

B. Backend API (api_server.py)

Built with FastAPI.

• Endpoints:
  • GET /health: Simple status check.
  • POST /process: The core endpoint. Accepts multipart/form-data.
• File Handling: Uses tempfile to safely store uploaded files temporarily for processing, ensuring they are cleaned up afterwards.
• CORS: Configured to allow requests from the frontend (localhost or production domain).

C. Inference Pipeline (inference_pipeline.py)

The heart of the system. It orchestrates the flow of data through various ML models.

1. Preprocessing (preprocessing.py)

Before OCR, images are cleaned to improve accuracy:

• Resizing: Images are resized to a standard width (2048px) to maintain aspect ratio.
• Denoising: cv2.fastNlMeansDenoising removes grain/noise.
• Thresholding: Adaptive thresholding converts images to binary (black/white) for clearer text.
• Deskewing: Detects text orientation and rotates the image to be horizontal.

2. OCR Engine (ocr_engine.py)

• Engine: Uses EasyOCR (or PaddleOCR) to extract text and bounding boxes.
• Output: Returns a list of text lines with their coordinates and confidence scores.

3. Classification (classifier_model.py & demo_mode.py)

Determines if the document is an Invoice, Receipt, Bank Statement, etc.

• Model: Uses a fine-tuned MiniLM transformer model.
• Demo Mode / Heuristics: If the model isn't loaded, it falls back to smart keyword matching:
  • Invoice: Contains "invoice".
  • Bank Statement: Contains "account statement", "summary", etc.
  • Receipt: Contains "receipt", "total".
• Refinement: Even with a trained model, we use _refine_classification in the pipeline to override the model if strong keywords are present (e.g., correcting "Receipt" to "Invoice" if the word "Invoice" is clearly visible).

4. NER (Named Entity Recognition) (ner_model.py)

Extracts specific fields like Dates, Amounts, and IDs.

• Model: DistilBERT fine-tuned for token classification.
• Tags: Uses BIO tagging (e.g., B-DATE, I-DATE) to identify the start and continuation of entities.

5. Post-Processing (postprocessing.py)

Cleans up and validates the raw ML outputs.

• Regex Fallbacks: If NER misses a field (like a date), regex patterns (\d{2}/\d{2}/\d{4}) attempt to find it in the raw text.
• Validation: Checks if extracted values make sense (e.g., is the "Total" actually a number?).
• Normalization: Converts dates to YYYY-MM-DD and amounts to standard float format (removing currency symbols).
• Confidence Scoring: Combines Model Confidence + OCR Confidence to give a final reliability score.

3. Data Flow Example

1. Upload: User uploads invoice.pdf.
2. Conversion: PDFtoImageConverter converts it to invoice.png.
3. Preprocess: Image is deskewed and binarized.
4. OCR: Text "Invoice #123" is detected at [100, 100].
5. Classify: Model sees "Invoice" -> Classifies as INVOICE (Conf: 0.98).
6. NER: Model identifies "123" as INVOICE_NUMBER.
7. Post-Process:
   • INVOICE_NUMBER "123" is validated.
   • Date "Jan 1, 2024" is normalized to 2024-01-01.
8. Response: JSON returned to frontend.
9. Display: Frontend shows "Invoice" badge and lists extracted fields.

4. Key Technologies & Libraries

• PyTorch: Deep learning framework for Classifier and NER.
• Transformers (Hugging Face): Model architecture (DistilBERT, MiniLM).
• OpenCV: Image manipulation.
• EasyOCR: Optical Character Recognition.
• NumPy: Numerical operations.
• Uvicorn: ASGI server for FastAPI.

5. Customization

• Adding New Doc Types:

  1. Update classifier_model.py labels.
  2. Add keywords to _refine_classification in inference_pipeline.py.
  3. Add field extraction logic to postprocessing.py.
  4. Update ResultsDisplay.tsx in frontend to handle new badge types.

• Tuning Accuracy:

  • Adjust ocr_confidence_threshold in api_server.py.
  • Add more regex patterns in postprocessing.py.