| # Product Requirements Document (PRD) |
| # Project: NCOS_S1 (Large Compliance LLM Pipeline) |
| |
| ## 1. Project Overview |
| Deploy a large compliance LLM (ACATECH/ncos, Llama-2-70B) on Hugging Face Spaces, with a Next.js frontend (Vercel), Supabase for test cases, and Redis for queueing. The backend is a FastAPI app running in a Docker container for full control (CUDA, dependencies, etc.). |
| |
| --- |
| |
| ## 2. Current State Analysis |
| - **Backend:** |
| - FastAPI app in Hugging Face Space, Dockerized. |
| - CUDA and torch set up for GPU inference. |
| - Permissions and cache issues resolved. |
| - Requirements are mostly correct and reproducible. |
| - **Frontend:** |
| - Next.js app on Vercel (not tightly integrated yet). |
| - **Test/Queue:** |
| - Supabase for test cases. |
| - Redis for queueing (not fully integrated). |
| - **Issues:** |
| - Dependency hell (CUDA, torch, flash-attn, numpy, etc.). |
| - File permission and cache issues. |
| - Model/tokenizer loading errors (corrupt/incompatible files). |
| - Manual syncing of requirements and Dockerfile. |
| - No robust, end-to-end pipeline from test case → queue → model → result → storage. |
| - No clear API contract between frontend, backend, and test/queue system. |
| - No health checks, monitoring, or error reporting. |
| - No automated deployment or CI/CD for the Space. |
| - Monolithic codebase, hard to debug. |
| |
| --- |
| |
| ## 3. Goals |
| - Modular, robust, and reproducible pipeline for LLM compliance testing. |
| - Clean separation of backend, frontend, and queue/storage. |
| - Automated, reliable deployment and monitoring. |
| - Clear API contract and documentation. |
| |
| --- |
| |
| ## 4. Recommended Architecture |
| ### A. Modular Structure |
| - **Backend (Hugging Face Space):** |
| - FastAPI app, Dockerized, REST API for inference. |
| - Handles model loading, inference, health checks. |
| - Connects to Redis for job queueing. |
| - Optionally connects to Supabase for test/result storage. |
| - **Frontend (Vercel/Next.js):** |
| - Calls backend API for inference. |
| - Displays results, test case status, health info. |
| - **Queue/Storage:** |
| - Redis for job queueing (decouples frontend/backend). |
| - Supabase for storing test cases/results. |
| |
| ### B. Key Features |
| - Robust error handling and logging in backend. |
| - Health check endpoints (`/healthz`, `/readyz`). |
| - Clear API contract (OpenAPI/Swagger for FastAPI). |
| - Automated Docker build and deployment (version pinning). |
| - CI/CD pipeline for backend and frontend. |
| - Documentation for setup, usage, troubleshooting. |
| |
| --- |
| |
| ## 5. Action Plan |
| ### Step 1: Design the API Contract |
| - Define endpoints for: |
| - `/infer` (POST): Accepts input, returns model output. |
| - `/healthz` (GET): Returns service health. |
| - `/queue` (POST/GET): For job submission/status (if using Redis). |
| - Use FastAPI's OpenAPI docs for clarity. |
| |
| ### Step 2: Clean Backend Implementation |
| - Start a new repo or clean branch. |
| - Write a minimal FastAPI app: |
| - Loads model/tokenizer (with robust error handling). |
| - Exposes `/infer` and `/healthz`. |
| - Logs errors and requests. |
| - Add Redis integration for queueing (optional, but recommended for scale). |
| - Add Supabase integration for test/result storage (optional, can be added after core works). |
| |
| ### Step 3: Dockerize the Backend |
| - Use a clean, minimal Dockerfile: |
| - Start from `nvidia/cuda:12.1.0-devel-ubuntu22.04`. |
| - Install Python, torch, dependencies in correct order. |
| - Set up cache and permissions. |
| - Pin all versions in `requirements.txt`. |
| - Add a health check in Dockerfile (`HEALTHCHECK`). |
| |
| ### Step 4: Model/Tokenizer Management |
| - Ensure model/tokenizer files are valid and compatible. |
| - Test loading locally before pushing to Hugging Face. |
| - Document the process for updating model files. |
| |
| ### Step 5: Frontend Integration |
| - Update Next.js frontend to call the new backend API. |
| - Show job status, results, and health info. |
| - Add error handling and user feedback. |
| |
| ### Step 6: Queue and Storage Integration |
| - Set up Redis for job queueing. |
| - Set up Supabase for test case/result storage. |
| - Ensure backend can pull jobs from Redis, process, and store results in Supabase. |
| |
| ### Step 7: Monitoring and Health |
| - Add logging and error reporting (e.g., to stdout, or a logging service). |
| - Implement `/healthz` and `/readyz` endpoints. |
| - Optionally, add Prometheus/Grafana metrics. |
| |
| ### Step 8: CI/CD and Documentation |
| - Add GitHub Actions or similar for automated build/test/deploy. |
| - Write clear README and API docs. |
| |
| --- |
| |
| ## 6. Success Criteria |
| - End-to-end pipeline works: test case → queue → model → result → storage. |
| - Robust error handling and health checks in place. |
| - Automated, reproducible builds and deployments. |
| - Clear, up-to-date documentation for all components. |
