Upload system_knowledge_base.md

system_knowledge_base.md

@@ -0,0 +1,225 @@
+# SYSTEM ARCHITECTURE & OPERATIONAL MANUAL (`KNOWLEDGE.md`)
+
+> **Target Audience:** AI Coding Agents, Automated CI/CD Pipelines, and Systems Engineers.
+> **Scope:** Architecture, configuration standards, service management SOPs, and troubleshooting guides for the `Tom716/set-api` Hugging Face Space.
+
+---
+
+## 1. System Overview & Architecture
+
+This repository hosts a multi-service backend environment on **Hugging Face Spaces (Free Tier)**. It utilizes a mono-repo structure where multiple isolated backend services run concurrently inside a single Docker container managed by **Supervisord** and routed externally via a **Caddy** reverse proxy.
+
+### System Stack Summary
+- **Infrastructure Provider:** Hugging Face Spaces (Docker SDK)
+- **Container Base OS:** Ubuntu 22.04 LTS
+- **Process Manager:** Supervisord (`supervisord`)
+- **Reverse Proxy / Router:** Caddy Server v2
+- **Public Ingress Port:** `7860` (Hard requirement by Hugging Face)
+- **Container User Security:** Non-root execution under User ID `1000` (`user:user`)
+- **Keep-Alive Engine:** External HTTP pinging (e.g., Better Stack) targeting `/status` every 5 minutes
+
+```
+                   Public Web Traffic (HTTPS Edge)
+                                │
+                                ▼
+                 ┌─────────────────────────────┐
+                 │  Hugging Face Edge Proxy   │
+                 └──────────────┬──────────────┘
+                                │ Port 7860
+                                ▼
+                 ┌─────────────────────────────┐
+                 │     Caddy Reverse Proxy     │
+                 └──────────────┬──────────────┘
+                                │
+        ┌───────────────────────┼───────────────────────┐
+        │                       │                       │
+        ▼                       ▼                       ▼
+┌───────────────┐       ┌───────────────┐       ┌───────────────┐
+│ Caddy /status │       │ NestJS API    │       │ FastAPI App   │
+│ Health Endpoint│      │ Telegram Demo │       │ K3s Test App  │
+│ (Internal)    │       │ (Port 3000)   │       │ (Port 8000)   │
+└───────────────┘       └───────────────┘       └───────────────┘
+```
+
+---
+
+## 2. Directory Layout & Routing Matrix
+
+### Directory Layout
+```text
+/ (Repository Root)
+├── Dockerfile                  # Container build instructions & runtime setup
+├── Caddyfile                   # Reverse proxy routing rules
+├── supervisord.conf            # Multi-process supervisor configuration
+├── KNOWLEDGE.md                # System documentation (This File)
+└── projects/                   # Multi-service monorepo directory
+    ├── telegram-send-demo/     # Service 1: NestJS + Telegram Bot Engine
+    │   └── backend/            # NestJS API source code
+    └── k3s-test-app/           # Service 2: Python FastAPI Service
+        ├── pyproject.toml      # Python build metadata
+        └── src/k3s_test_app/   # FastAPI source code
+```
+
+### Port Allocation & Ingress Matrix
+
+| Service | Internal Port | Ingress Path | Health / Docs Path |
+| :--- | :--- | :--- | :--- |
+| **Caddy Proxy** | `7860` (Public) | `/` | `GET /status` |
+| **Caddy Admin API** | `2019` (Local) | `localhost:2019` | N/A |
+| **NestJS Telegram Service** | `3000` (Internal)| `/api/telegram/*` | `GET /api/telegram/health`, `/api/docs` |
+| **Python FastAPI Service** | `8000` (Internal)| `/api/k3s/*` | `GET /api/k3s/health`, `/api/k3s/docs` |
+
+---
+
+## 3. Configuration File Standards
+
+### 3.1 `Caddyfile` Standard
+```caddy
+{
+    admin localhost:2019
+    auto_https off
+}
+
+:7860 {
+    # Keep-Alive Status Check
+    handle /status {
+        respond "Caddy Proxy Active" 200
+    }
+
+    # Service: Telegram NestJS Backend
+    handle /api/telegram/* {
+        reverse_proxy localhost:3000
+    }
+
+    # Service: Python FastAPI Backend (Strips prefix for internal app routing)
+    handle /api/k3s/* {
+        uri strip_prefix /api/k3s
+        reverse_proxy localhost:8000
+    }
+
+    # Default Fallback (Routes to primary API service)
+    handle {
+        reverse_proxy localhost:3000
+    }
+}
+```
+
+### 3.2 `supervisord.conf` Standard
+```ini
+[supervisord]
+nodaemon=true
+user=user
+
+[program:caddy]
+command=caddy run --config /etc/caddy/Caddyfile --adapter caddyfile
+autostart=true
+autorestart=true
+stdout_logfile=/dev/stdout
+stdout_logfile_maxbytes=0
+stderr_logfile=/dev/stderr
+stderr_logfile_maxbytes=0
+
+[program:nestjs_backend]
+command=node /app/projects/telegram-send-demo/backend/dist/main.js
+directory=/app/projects/telegram-send-demo/backend
+autostart=true
+autorestart=true
+stdout_logfile=/dev/stdout
+stdout_logfile_maxbytes=0
+stderr_logfile=/dev/stderr
+stderr_logfile_maxbytes=0
+
+[program:k3s_backend]
+command=/app/projects/k3s-test-app/.venv/bin/uvicorn k3s_test_app.main:app --host 127.0.0.1 --port 8000 --root-path /api/k3s
+directory=/app/projects/k3s-test-app
+autostart=true
+autorestart=true
+stdout_logfile=/dev/stdout
+stdout_logfile_maxbytes=0
+stderr_logfile=/dev/stderr
+stderr_logfile_maxbytes=0
+```
+
+---
+
+## 4. Standard Operating Procedures (SOPs)
+
+### 4.1 How to Add a New API Backend Service
+
+When adding a new backend (e.g., `analytics-service` running on port `4000`):
+
+1. **Upload Source:** Place the project files inside `/projects/analytics-service`.
+2. **Update `Dockerfile`:** Add the build steps for the new runtime/dependencies:
+   ```dockerfile
+   COPY projects/analytics-service /app/projects/analytics-service
+   RUN cd /app/projects/analytics-service && npm install && npm run build
+   ```
+3. **Update `supervisord.conf`:** Define a new supervisor process block:
+   ```ini
+   [program:analytics_service]
+   command=node /app/projects/analytics-service/dist/main.js
+   directory=/app/projects/analytics-service
+   autostart=true
+   autorestart=true
+   stdout_logfile=/dev/stdout
+   stdout_logfile_maxbytes=0
+   stderr_logfile=/dev/stderr
+   stderr_logfile_maxbytes=0
+   ```
+4. **Update `Caddyfile`:** Route a unique path prefix to the allocated internal port:
+   ```caddy
+   handle /api/analytics/* {
+       reverse_proxy localhost:4000
+   }
+   ```
+5. **Commit & Deploy:** Commit changes to `main`. Hugging Face will trigger a fresh container build automatically.
+
+---
+
+### 4.2 How to Remove an Existing Service
+
+To decommission a service (e.g., `projects/k3s-test-app`):
+
+1. **Update `Caddyfile`:** Remove the corresponding `handle /api/k3s/*` block.
+2. **Update `supervisord.conf`:** Remove the `[program:k3s_backend]` block.
+3. **Update `Dockerfile`:** Remove build lines associated with that project.
+4. **Delete Directory:** Delete `/projects/k3s-test-app`.
+5. **Commit Changes:** Push to `main`.
+
+---
+
+## 5. Critical Troubleshooting & System Rules
+
+Any agent modifying this repository must follow these rules to avoid production breakages:
+
+### 1. Node.js IPv6 DNS Resolution Fix (`ipv4first`)
+- **Problem:** Node.js 20+ defaults to IPv6 DNS resolution. Inside Hugging Face Docker containers without outbound IPv6 routing, outbound HTTP requests (e.g., grammY hitting `api.telegram.org`) hang until timing out, returning a `503` error.
+- **Mandatory Fix:** Always include the following at the top of Node.js entrypoints (`main.ts`):
+  ```typescript
+  import * as dns from 'dns';
+  dns.setDefaultResultOrder('ipv4first');
+  ```
+
+### 2. FastAPI Behind Reverse Proxy Subpaths (`--root-path`)
+- **Problem:** When FastAPI is proxied under a subpath like `/api/k3s/*`, Swagger UI fails to fetch OpenAPI specs (`/openapi.json` returns 404).
+- **Mandatory Fix:** Always launch Uvicorn with `--root-path /api/<service_name>` or configure `root_path="/api/<service_name>"` inside `FastAPI()`.
+
+### 3. NPM Lockfile Synchronization (`npm install` vs `npm ci`)
+- **Problem:** `npm ci` fails if `package.json` and `package-lock.json` are slightly out of sync during container builds.
+- **Mandatory Fix:** Use `npm install` inside the container build steps in `Dockerfile` to dynamically install missing transitive dependencies.
+
+### 4. Base Image Python Versioning
+- **Problem:** Ubuntu 22.04 LTS provides **Python 3.10.12**. Setting `requires-python = ">=3.14"` in `pyproject.toml` causes `pip` installation failures.
+- **Mandatory Fix:** Ensure all Python projects set `requires-python = ">=3.10"`.
+
+### 5. Non-Root File Permissions (`User 1000`)
+- **Problem:** Hugging Face executes containers strictly as User ID `1000` (`user`). System services expecting root access fail to start.
+- **Mandatory Fix:** Include `chown -R user:user /app /etc/caddy /var/log/supervisor /var/lib/caddy /var/log/caddy` in `Dockerfile` before switching to `USER user`.
+
+### 6. Ephemeral Filesystem Limitation
+- **Warning:** Storage in free Hugging Face Spaces is **ephemeral**. Any local files, database files (e.g., SQLite), or authentication keys (e.g., Baileys WhatsApp sessions) written to disk will be wiped when the container restarts.
+- **Mandatory Requirement:** All persistent application state must be stored in external managed databases (e.g., Supabase, Neon PostgreSQL, or Redis).
+
+### 7. Container Sleep Prevention
+- **Mechanism:** Hugging Face Space containers go to sleep after 15–45 minutes of HTTP inactivity.
+- **Mandatory Requirement:** An external monitoring service (Better Stack, UptimeRobot) must issue an HTTP `GET` ping to `https://tom716-set-api.hf.space/status` every **5 minutes**.