| # Project Structure |
|
|
| ``` |
| pdf-redaction-api/ |
| │ |
| ├── main.py # FastAPI application entry point |
| ├── Dockerfile # Docker configuration for deployment |
| ├── requirements.txt # Python dependencies |
| ├── README.md # Project documentation (for HuggingFace) |
| ├── DEPLOYMENT.md # Deployment guide |
| ├── .gitignore # Git ignore rules |
| ├── .dockerignore # Docker ignore rules |
| │ |
| ├── app/ # Application modules |
| │ ├── __init__.py # Package initialization |
| │ └── redaction.py # Core redaction logic (PDFRedactor class) |
| │ |
| ├── uploads/ # Temporary upload directory |
| │ └── .gitkeep # Keep directory in git |
| │ |
| ├── outputs/ # Redacted PDF output directory |
| │ └── .gitkeep # Keep directory in git |
| │ |
| ├── tests/ # Test suite |
| │ ├── __init__.py |
| │ └── test_api.py # API endpoint tests |
| │ |
| └── client_example.py # Example client for API usage |
| ``` |
|
|
| ## File Descriptions |
|
|
| ### Core Files |
|
|
| #### `main.py` |
| FastAPI application with endpoints: |
| - `POST /redact` - Upload and redact PDF |
| - `GET /download/{job_id}` - Download redacted PDF |
| - `GET /health` - Health check |
| - `GET /stats` - API statistics |
| - `DELETE /cleanup/{job_id}` - Manual cleanup |
|
|
| #### `app/redaction.py` |
| Core redaction logic: |
| - `PDFRedactor` class |
| - OCR processing with pytesseract |
| - NER using HuggingFace transformers |
| - Entity-to-box mapping |
| - PDF redaction with coordinate scaling |
|
|
| ### Configuration Files |
|
|
| #### `requirements.txt` |
| Python dependencies: |
| - FastAPI & Uvicorn (API framework) |
| - Transformers & Torch (NER model) |
| - PyPDF (PDF manipulation) |
| - pdf2image (PDF to image conversion) |
| - pytesseract (OCR) |
| - Pillow (Image processing) |
|
|
| #### `Dockerfile` |
| Multi-stage build: |
| 1. Install system dependencies (tesseract, poppler) |
| 2. Install Python dependencies |
| 3. Copy application code |
| 4. Configure for port 7860 (HuggingFace default) |
|
|
| ### Documentation |
|
|
| #### `README.md` |
| HuggingFace Space documentation: |
| - Features overview |
| - API endpoint documentation |
| - Usage examples (cURL, Python) |
| - Response format |
| - Local development setup |
|
|
| #### `DEPLOYMENT.md` |
| Step-by-step deployment guide: |
| - HuggingFace Spaces setup |
| - Git workflow |
| - Configuration options |
| - Security considerations |
| - Troubleshooting |
| - Cost estimation |
|
|
| ### Testing & Examples |
|
|
| #### `tests/test_api.py` |
| Unit tests for API endpoints: |
| - Health check tests |
| - Upload validation tests |
| - Error handling tests |
| |
| #### `client_example.py` |
| Example client implementation: |
| - Upload PDF |
| - Download redacted file |
| - Health check |
| - Statistics |
|
|
| ## Data Flow |
|
|
| ``` |
| ┌─────────────────────────────────────────────────────────┐ |
| │ 1. Client uploads PDF │ |
| │ POST /redact with file │ |
| └─────────────────────────────────────────────────────────┘ |
| ↓ |
| ┌─────────────────────────────────────────────────────────┐ |
| │ 2. FastAPI (main.py) │ |
| │ - Validates file │ |
| │ - Generates job_id │ |
| │ - Saves to uploads/ │ |
| └─────────────────────────────────────────────────────────┘ |
| ↓ |
| ┌─────────────────────────────────────────────────────────┐ |
| │ 3. PDFRedactor (app/redaction.py) │ |
| │ - perform_ocr() → Extract text + boxes │ |
| │ - run_ner() → Identify entities │ |
| │ - map_entities_to_boxes() → Link entities to coords │ |
| │ - create_redacted_pdf() → Generate output │ |
| └─────────────────────────────────────────────────────────┘ |
| ↓ |
| ┌─────────────────────────────────────────────────────────┐ |
| │ 4. Response │ |
| │ - Return job_id and entity list │ |
| │ - Save redacted PDF to outputs/ │ |
| └─────────────────────────────────────────────────────────┘ |
| ↓ |
| ┌─────────────────────────────────────────────────────────┐ |
| │ 5. Client downloads │ |
| │ GET /download/{job_id} │ |
| └─────────────────────────────────────────────────────────┘ |
| ``` |
|
|
| ## Key Components |
|
|
| ### 1. FastAPI Application (`main.py`) |
|
|
| **Endpoints:** |
| - RESTful API design |
| - File upload handling |
| - Background task cleanup |
| - CORS middleware for web access |
|
|
| **Features:** |
| - Automatic OpenAPI documentation at `/docs` |
| - JSON response models with Pydantic |
| - Error handling with HTTP exceptions |
| - Request validation |
|
|
| ### 2. Redaction Engine (`app/redaction.py`) |
|
|
| **Pipeline Steps:** |
|
|
| 1. **OCR Processing** |
| - Convert PDF pages to images (pdf2image) |
| - Extract text and bounding boxes (pytesseract) |
| - Store image dimensions for coordinate scaling |
|
|
| 2. **NER Processing** |
| - Load HuggingFace model |
| - Identify entities in text |
| - Return entity types and character positions |
|
|
| 3. **Mapping** |
| - Create character span index for OCR words |
| - Match NER entities to OCR bounding boxes |
| - Handle partial word matches |
|
|
| 4. **Redaction** |
| - Scale OCR image coordinates to PDF points |
| - Create black rectangle annotations |
| - Write redacted PDF with pypdf |
|
|
| ### 3. Docker Container |
|
|
| **Layers:** |
| - Base: Python 3.10 slim |
| - System packages: tesseract-ocr, poppler-utils |
| - Python packages: From requirements.txt |
| - Application code: Copied last for better caching |
|
|
| **Optimizations:** |
| - Multi-stage build (not used here, but possible) |
| - Minimal base image |
| - Cached dependency layers |
| - .dockerignore to reduce context size |
|
|
| ## Environment Variables |
|
|
| Default configuration (can be overridden): |
|
|
| ```bash |
| PYTHONUNBUFFERED=1 # Immediate log output |
| HF_HOME=/app/cache # HuggingFace cache directory |
| ``` |
|
|
| ## Port Configuration |
|
|
| - **Development**: 7860 (configurable in main.py) |
| - **Production (HF Spaces)**: 7860 (required) |
|
|
| ## Directory Permissions |
|
|
| Ensure write permissions for: |
| - `uploads/` - Temporary PDF storage |
| - `outputs/` - Redacted PDF storage |
| - `cache/` - Model cache (created automatically) |
|
|
| ## Adding New Features |
|
|
| ### Add New Endpoint |
|
|
| 1. Define in `main.py`: |
| ```python |
| @app.get("/new-endpoint") |
| async def new_endpoint(): |
| return {"message": "Hello"} |
| ``` |
|
|
| 2. Add response model if needed |
| 3. Update README.md documentation |
| 4. Add tests in `tests/test_api.py` |
|
|
| ### Add New Redaction Option |
|
|
| 1. Modify `PDFRedactor` class in `app/redaction.py` |
| 2. Add parameter to `redact_document()` method |
| 3. Update API endpoint in `main.py` |
| 4. Document in README.md |
|
|
| ### Add Authentication |
|
|
| 1. Install: `pip install python-jose passlib` |
| 2. Create `app/auth.py` with JWT logic |
| 3. Add middleware to `main.py` |
| 4. Protect endpoints with dependencies |
|
|
| ## Best Practices |
|
|
| 1. **Logging**: Use `logger` for all important events |
| 2. **Error Handling**: Catch exceptions and return meaningful errors |
| 3. **Validation**: Use Pydantic models for request/response validation |
| 4. **Cleanup**: Always clean up temporary files |
| 5. **Documentation**: Keep README.md and code comments updated |
| 6. **Testing**: Add tests for new features |
|
|
| ## Performance Considerations |
|
|
| ### Bottlenecks |
| 1. OCR processing (most time-consuming) |
| 2. Model inference (NER) |
| 3. File I/O |
|
|
| ### Optimizations |
| - Lower DPI for faster OCR (trade-off with accuracy) |
| - Cache loaded models in memory |
| - Use async file operations |
| - Implement request queuing for high load |
| - Consider GPU for NER model |
|
|
| ### Scaling |
| - Horizontal: Multiple container instances |
| - Vertical: Larger CPU/RAM allocation |
| - Caching: Redis for temporary results |
| - Queue: Celery for background processing |
|
|
