TECHNICAL_ARCHITECTURE.md

# Technical Architecture & Implementation Guide

This document provides a deep dive into the internal workings of the IDP (Intelligent Document Processing) system. It explains the "how" and "why" behind the implementation decisions.

## 1. System Architecture

The system follows a modern decoupled architecture:

```mermaid
graph TD
    User[User] -->|Uploads File| Frontend[Next.js Frontend]
    Frontend -->|POST /process| API[FastAPI Backend]
    
    subgraph "Inference Pipeline"
        API -->|Raw Image/PDF| Preprocessor[Preprocessing Module]
        Preprocessor -->|Cleaned Image| OCR[OCR Engine]
        OCR -->|Raw Text + BBoxes| Classifier[Document Classifier]
        OCR -->|Raw Text| NER[NER Model]
        
        Classifier -->|Doc Type| PostProcessor[Post-Processing]
        NER -->|Entities| PostProcessor
        OCR -->|Text + BBoxes| PostProcessor
        
        PostProcessor -->|Structured JSON| API
    end
    
    API -->|JSON Response| Frontend
    Frontend -->|Render| UI[Results Display]
```

## 2. Component Deep Dive

### A. Frontend (`/frontend`)

Built with **Next.js 14** (App Router), **React 18**, and **Tailwind CSS**.

*   **`app/page.tsx`**: The main entry point. It manages the global state for `result`, `loading`, and `isSystemOnline`. It uses a polling mechanism (`setInterval`) to check the backend health status every 30 seconds.
*   **`components/DocumentUpload.tsx`**: Handles file selection and drag-and-drop.
    *   **Auto-Processing**: As soon as a valid file is selected, it automatically triggers the upload.
    *   **Validation**: Checks file size (<10MB) and type (PDF, PNG, JPEG).
*   **`components/ResultsDisplay.tsx`**: Renders the JSON response.
    *   **Dynamic Badges**: Uses `getDocTypeStyle` to color-code document types (Purple for Invoice, Pink for Receipt, Blue for Bank Statement).
    *   **Confidence Indicators**: Color-codes confidence scores (Green > 80%, Yellow > 60%, Red < 60%).
*   **Styling**: Uses `globals.css` for custom animations (`animate-fade-in-up`) and utility classes (`glass-panel`, `bento-card`).

### B. Backend API (`api_server.py`)

Built with **FastAPI**.

*   **Endpoints**:
    *   `GET /health`: Simple status check.
    *   `POST /process`: The core endpoint. Accepts `multipart/form-data`.
*   **File Handling**: Uses `tempfile` to safely store uploaded files temporarily for processing, ensuring they are cleaned up afterwards.
*   **CORS**: Configured to allow requests from the frontend (localhost or production domain).

### C. Inference Pipeline (`inference_pipeline.py`)

The heart of the system. It orchestrates the flow of data through various ML models.

#### 1. Preprocessing (`preprocessing.py`)
Before OCR, images are cleaned to improve accuracy:
*   **Resizing**: Images are resized to a standard width (2048px) to maintain aspect ratio.
*   **Denoising**: `cv2.fastNlMeansDenoising` removes grain/noise.
*   **Thresholding**: Adaptive thresholding converts images to binary (black/white) for clearer text.
*   **Deskewing**: Detects text orientation and rotates the image to be horizontal.

#### 2. OCR Engine (`ocr_engine.py`)
*   **Engine**: Uses **EasyOCR** (or PaddleOCR) to extract text and bounding boxes.
*   **Output**: Returns a list of text lines with their coordinates and confidence scores.

#### 3. Classification (`classifier_model.py` & `demo_mode.py`)
Determines if the document is an Invoice, Receipt, Bank Statement, etc.
*   **Model**: Uses a fine-tuned **MiniLM** transformer model.
*   **Demo Mode / Heuristics**: If the model isn't loaded, it falls back to smart keyword matching:
    *   *Invoice*: Contains "invoice".
    *   *Bank Statement*: Contains "account statement", "summary", etc.
    *   *Receipt*: Contains "receipt", "total".
*   **Refinement**: Even with a trained model, we use `_refine_classification` in the pipeline to override the model if strong keywords are present (e.g., correcting "Receipt" to "Invoice" if the word "Invoice" is clearly visible).

#### 4. NER (Named Entity Recognition) (`ner_model.py`)
Extracts specific fields like Dates, Amounts, and IDs.
*   **Model**: **DistilBERT** fine-tuned for token classification.
*   **Tags**: Uses BIO tagging (e.g., `B-DATE`, `I-DATE`) to identify the start and continuation of entities.

#### 5. Post-Processing (`postprocessing.py`)
Cleans up and validates the raw ML outputs.
*   **Regex Fallbacks**: If NER misses a field (like a date), regex patterns (`\d{2}/\d{2}/\d{4}`) attempt to find it in the raw text.
*   **Validation**: Checks if extracted values make sense (e.g., is the "Total" actually a number?).
*   **Normalization**: Converts dates to `YYYY-MM-DD` and amounts to standard float format (removing currency symbols).
*   **Confidence Scoring**: Combines Model Confidence + OCR Confidence to give a final reliability score.

## 3. Data Flow Example

1.  **Upload**: User uploads `invoice.pdf`.
2.  **Conversion**: `PDFtoImageConverter` converts it to `invoice.png`.
3.  **Preprocess**: Image is deskewed and binarized.
4.  **OCR**: Text "Invoice #123" is detected at `[100, 100]`.
5.  **Classify**: Model sees "Invoice" -> Classifies as `INVOICE` (Conf: 0.98).
6.  **NER**: Model identifies "123" as `INVOICE_NUMBER`.
7.  **Post-Process**:
    *   `INVOICE_NUMBER` "123" is validated.
    *   Date "Jan 1, 2024" is normalized to `2024-01-01`.
8.  **Response**: JSON returned to frontend.
9.  **Display**: Frontend shows "Invoice" badge and lists extracted fields.

## 4. Key Technologies & Libraries

*   **PyTorch**: Deep learning framework for Classifier and NER.
*   **Transformers (Hugging Face)**: Model architecture (DistilBERT, MiniLM).
*   **OpenCV**: Image manipulation.
*   **EasyOCR**: Optical Character Recognition.
*   **NumPy**: Numerical operations.
*   **Uvicorn**: ASGI server for FastAPI.

## 5. Customization

*   **Adding New Doc Types**:
    1.  Update `classifier_model.py` labels.
    2.  Add keywords to `_refine_classification` in `inference_pipeline.py`.
    3.  Add field extraction logic to `postprocessing.py`.
    4.  Update `ResultsDisplay.tsx` in frontend to handle new badge types.

*   **Tuning Accuracy**:
    *   Adjust `ocr_confidence_threshold` in `api_server.py`.
    *   Add more regex patterns in `postprocessing.py`.