animetix-web / docs /ARCHITECTURE.md
MissawB's picture
Upload folder using huggingface_hub (part 3)
feef5d9 verified
|
Raw History Blame Contribute Delete
18.8 kB
# Technical & Modular Architecture (Atomic & Hexagonal)
This document describes the software architecture of the **Double_scenario_Project** (Anime Archetype Engine). The project adopts an **Atomic & Hexagonal** (Clean Architecture) approach to guarantee a strict decoupling between the business logic (Domain) and the infrastructure layer (Adapters).
---
## 1. Overview of the Hexagon
The architecture is divided into three distinct layers:
```mermaid
graph TD
subgraph Frameworks & Adapters (External)
Django[Django Backend & Channels]
ML_Adapters[Inference Adapters: LocalLlama, Diffusers, Transformers]
Persistence_Adapters[Persistence Adapters: Vertex AI, pgvector, Neo4j, Django DB]
end
subgraph Ports (Interfaces)
InferencePort[InferencePort - includes Reranking]
PersistencePort[PersistencePort - UnifiedRepository]
end
subgraph Core Domain (Business Logic)
Services[Domain Services: AgenticRAGService, PromptManager, Games]
Models[Pydantic Models: DTOs, AI Schemas]
end
Django --> Services
Services --> InferencePort
Services --> PersistencePort
ML_Adapters --> InferencePort
Persistence_Adapters --> PersistencePort
```
---
## 2. Source Code Structure
The backend code is organized under `backend/`:
- **`core/ports/`**: Abstractions (Abstract Base Classes) defining the business contracts.
- `InferencePort`: Text/Image generation, voice cloning, reranking, and advanced computer vision.
- `MlopsPort`: Handles telemetry, DPO logging, and AI feedback loops via Cloud Run Jobs / GCP Tasks.
- `PersistencePort`: Unified data access definition (`UnifiedRepositoryAdapter`).
- **`core/domain/services/`**: Pure business logic services, completely independent of infrastructure or frameworks.
- **`adapters/`**: Concrete infrastructure implementations.
- `adapters/persistence/`: Handles data multi-sources (Vertex AI, pgvector, Neo4j, Django DB).
- `adapters/inference/`: Adapter implementations for Google GenAI, BrainAPI, Ollama (Unified), and local Transformers. The larger adapters are composed from **mixins**, and the base order matters: `class Fallback(Mixin, InferencePort)`. Reversed, `InferencePort`'s stub wins the MRO, the mixin silently becomes dead code, and the capability stops working with no error β€” this is exactly how `generate_sprite` was broken for months. A composition guard (`test_mixin_composition.py`) pins the MRO and forbids two mixins sharing a method name.
- **`api/`**: Headless Django presentation layer, split by domain (`animetix/api/core/`, `animetix/api/labs/`, `animetix/api/games/`, SSE stream views). Dependencies are declared in `containers/` (Dependency-Injector) and **constructor-injected** into every view (`@inject __init__` with `Provide[...]` defaults) β€” there is no service-locator access left in the view layer; tests override the container providers.
---
## 3. Storage & Persistence
The project utilizes **PostgreSQL + pgvector** in production and **NumPy (SQLite)** as the zero-configuration local fallback for semantic vector searches; a **Vertex AI Vector Search (Collections)** adapter exists as an optional managed alternative. Data access is unified under the `UnifiedRepositoryAdapter`. Additionally, **Neo4j** (Aura in production) acts as the graph database mapping complex, topological creator-studio-character relations.
> **Which Postgres is production?** **Cloud SQL** (`animetix-db`, `europe-west9`). **Neon serverless is development-only** β€” an earlier revision of this document said the opposite, and the confusion is expensive: pointing a migration or a backup check at Neon while believing it is production means touching the wrong database.
> **One trap when reading `jsonb` back:** Django disables psycopg2's automatic `jsonb` decoding, so **a raw SQL cursor receives a `str`, not a `dict`**. Test doubles that hand back dicts will pass while production returns a 500 on every row.
---
## 4. Lazy Imports Strategy
To optimize startup performance and keep memory usage low, heavy AI libraries (`torch`, `transformers`, etc.) are imported lazily using an attribute-based wrapper (`lazy_import.py`). The actual module import is only triggered during the first attribute access, eliminating unnecessary loading overhead for non-AI tasks.
---
## 4bis. Async / Sync Strategy
The codebase is **synchronous by default**, with `async` deliberately confined to two edges. This is the canonical Django + Channels model β€” keep it; do **not** "async-ify" the core.
- **Core domain & DRF views are SYNC.** Domain services (`core/domain/services/`) expose synchronous public APIs (e.g. `ReasoningAgentService.solve_complex_query`, the RAG services, game logic). DRF request handling is sync. Inference adapters are sync.
- **Async edge #1 β€” ASGI / Channels consumers** (`api/animetix/consumers/`): WebSocket consumers are `async` because they run on the ASGI event loop (real-time duel, codemanga, notifications, Gemini Live…). This is the *only* place async is the default.
- **Async edge #2 β€” Native async SSE stream views** (`api/animetix/api/streams.py`): the five Server-Sent-Events views (Animinator, Emoji, Paradox, Tree-of-Thoughts, Agentic RAG) are plain async Django views (outside DRF) consuming the adapters' native `astream_generate` β€” the worker thread is freed for the whole duration of a stream. Session/ORM access goes through `sync_to_async`.
- **Async edge #3 β€” Multi-agent bus subsystem** (`multi_agent_bus`, `orchestrator_agent_service.execute_workflow`, `reasoning_agent_service._on_bus_message`): Redis pub/sub over `asyncio`. Its `async` methods run **only inside an async context** (the bus loop / a consumer). Note: this subsystem is currently *not wired to any HTTP/WS endpoint* (experimental) β€” its services expose sync entry points (`solve_complex_query`) for normal use, and the async methods are bus-internal.
**Boundary rules (to keep it coherent):**
1. Never call an `async` method directly from sync request code. If a sync view ever needs the bus, wrap with `asgiref.sync.async_to_sync` β€” do not spin up `asyncio.run()` per request (it breaks the running loop under ASGI). *(Current count of boundary crossings: zero β€” the edges don't leak into the core.)*
2. Never perform blocking I/O (DB, sync HTTP) directly inside an async consumer; wrap with `asgiref.sync.sync_to_async`.
3. If the multi-agent bus is ever exposed to users, drive it from an **async consumer**, not a sync DRF view.
For SSRF-safe HTTP, both sync (`safe_http_request`) and async (`safe_http_request_async`) helpers exist in `core/utils/security.py`; pick the one matching your context.
---
## 5. Extensibility & Port Implementation
Adapters implement the abstract ports. Any method not implemented by a specific adapter raises an `InferenceNotImplementedError`. Extending the platform follows a strict pattern:
1. Extend the abstract **Port** definition.
2. Implement the concrete logic in the corresponding **Adapter**.
3. Register or bind the new implementation inside the `containers/` package (and add the consuming module to the `container.wire()` list in `apps.py` if it uses `@inject`).
---
## 6. Deployment: Decoupled Single Page Application (SPA)
Animetix is designed and deployed as a fully decoupled **Pure SPA** (Single Page Application).
- **Frontend (Static)**: A modern React application built with **Vite** (`frontend/`). The production bundle (`dist/`) is built for high performance. In development mode, Vite runs on port `5173` and proxies `/api` and `/ws` requests to the Django backend.
- **Backend (Headless API)**: Django operates strictly as a headless API. All legacy HTML templates and view controllers have been completely removed.
- **Unified Client-Side Routing**: Django routes any non-API fallback paths (`re_path(r'^(?!api/|static/|admin/).*$', spa_view)`) directly to the SPA, allowing React Router DOM to manage application routing on the client side.
- **One exception to "pure SPA" β€” the crawler-facing body** (`animetix/seo/`, September 2026). The server fills `<div id="root">` with the page's real headings and prose before React mounts. React uses `createRoot`, **not** `hydrateRoot`, so the container is empty at mount time and the block is replaced, never duplicated. The text is composed from the *same* i18n keys the page displays β€” writing it separately would create two copies, and a server body that diverges from the rendered page is **cloaking**.
> **On this deployment, no 404 is a 404.** `LocaleMiddleware` rewrites a missing path into a 302 toward `/fr/<path>`, where the SPA catch-all answers **200 `text/html`**. A deleted sitemap therefore looked to Search Console like "a sitemap in HTML format" β€” a parse error β€” instead of "removed", so its URLs were never dropped. Routes that must be gone return an explicit **410 Gone**.
---
## 7. Inference Adapters Ecosystem (Host-Dependent Chain)
The project implements a resilient `FallbackInferenceAdapter`. **The chain order depends on the host it runs on** β€” this is decided in `build_inference_chain()` ([`containers/inference.py`](../backend/api/animetix/containers/inference.py)), not fixed globally. On a developer machine, free local compute comes first; **in production the chain is remote-only**, because trying local engines that cannot exist there would burn the whole retry budget before reaching a backend that works.
```mermaid
graph TD
subgraph OnHost ["On-host (development)"]
F1["FallbackInferenceAdapter"] --> Ollama["UnifiedInferenceAdapter (local Ollama, FREE)"]
F1 --> Transformers["LocalTextAdapter (Transformers, 4-bit)"]
F1 --> B1["BrainAPIAdapter"]
F1 --> G1["GoogleGenAIAdapter (Gemini)"]
end
subgraph OffHost ["Off-host (production): remote-only"]
F2["FallbackInferenceAdapter"] --> B2["BrainAPIAdapter"]
F2 --> G2["GoogleGenAIAdapter (Gemini, last resort)"]
end
```
### 7bis. Two GPU services, not one
Text generation was **extracted out of the brain** in August 2026, so `BrainAPIAdapter` no longer talks to a single host:
- **`animetix-brain`** β€” vision only: CLIP/CCIP encoding, image/video/audio, 3D, manga OCR.
- **`animetix-llm`** β€” text generation and moderation, through an Ollama baked into the image.
Each image **refuses the other's routes** via the `SERVES_TEXT` flag, rather than dialing an endpoint nothing listens on. A second, separate `BrainAPIAdapter` singleton is wired for moderation specifically, so the guardrail's small model does not occupy the 27B.
> ⚠️ **This describes the code, not production** (measured 2026-09-07). `animetix-llm` has never been built or deployed: no `llm` package in Artifact Registry, no such Cloud Run service, and `LLM_API_URL` resolves to the **brain** on the web service and on all nine jobs. The brain image in production predates the split by two days, so it still carries Ollama and answers `/generate` β€” 999 times over seven days, all 200. Consequences: (1) rebuilding the brain from current code would 503 every generation and moderation, with a green deploy and a healthy `/health`, so `scripts/deploy/gcp/verify_text_serving.py` blocks that rebuild until a distinct service serves text; (2) deploying `animetix-llm` would **add** a second L4 (8 vCPU / 32 GiB against the brain's 4/16), so the split costs more, it does not save.
> **The served model tag has exactly one source of truth**: `LLM_OLLAMA_MODEL` in [`core/utils/local_models.py`](../backend/core/utils/local_models.py), which the Dockerfile bakes. It is deliberately **not** pinned in `deployments.yaml`: pinning it there is precisely what once let the manifest drift from the image and return **404 on every generation while `/health` reported `online`**.
---
## 8. Economy & Monetization: The Berrix Model
Animetix operates on a **Rewarded Economy** where all advanced AI features are 100% free for users, financed by their engagement.
### 8.1. Tokens: The Berrix (Bx)
- **Passive Mining**: Users earn **+10 Bx** per 3-minute cooldown (`MINING_REWARD_BX`, enforced server-side in `WalletMineView`). This is an explicit, capped **loss-leader** β€” it is not financed by anything.
- **Rewarded ads β€” scaffolding only, deliberately dormant.** `ad_reward_bx()` (β‰ˆ 41 Bx, calibrated for the margin floor) and a verifier exist, but **no ad network is wired**: `REWARDED_ADS_PROVIDER` is unset, so the verifier stays in stub mode. Ad-gated Bx rewards were **stripped in July 2026 for AdSense policy reasons** and must not be re-enabled in production before AdSense itself is validated. The older "+250 Bx per 30s video" figure is obsolete on both the amount and the mechanism.
- **No purchases**: Berrix cannot be bought β€” the Stripe payment stack was removed entirely (2026-07-07); `developer.py` now grants the Pro tier manually and for free. Earned Bx is the only currency; the `tier` concept (free/pro) only gates the developer API.
> Consequence worth stating plainly: **there is currently no revenue path at all.** Mining is a pure cost, ads are off, purchases are gone. Any cost model that assumes ad or pack revenue (see [COST_AUDIT.md](COST_AUDIT.md) Β§4) is describing a design, not the running system.
### 8.2. Token Consumption & Protection
Business logic services (`InferencePort`, `Forge`) call the atomic `deduct_berrix` function. If the user's `wallet_balance` is insufficient, the API returns an **HTTP 402 Payment Required** error, which the frontend intercepts to redirect the user to the **Power Station**.
---
## 9. VisionTransformersAdapter Mixin Architecture
To maintain high readability and avoid a monolithic file, the `VisionTransformersAdapter` is modularized into **four specialized mixins**:
```mermaid
classDiagram
class InferencePort {
<<abstract>>
+generate()
+health_check()
}
class DepthEstimationMixin {
+estimate_depth()
+generate_3d_scene()
}
class MangaOcrMixin {
+process_manga_page()
}
class VideoAnalysisMixin {
+get_video_temporal_embeddings()
+localize_video_actions()
+generate_video_description()
}
class ClipVisionMixin {
+get_image_embedding()
+classify_image()
+calculate_visual_similarity()
+visual_rerank()
+get_multimodal_late_interaction()
}
class VisionTransformersAdapter {
+detect_objects()
+generate_image_description()
+health_check()
}
InferencePort <|-- VisionTransformersAdapter
DepthEstimationMixin <|-- VisionTransformersAdapter
MangaOcrMixin <|-- VisionTransformersAdapter
VideoAnalysisMixin <|-- VisionTransformersAdapter
ClipVisionMixin <|-- VisionTransformersAdapter
```
---
## 10. Error Hierarchy
All custom application errors derive from `AnimetixBaseError`:
```mermaid
classDiagram
class AnimetixBaseError {
+message: str
+context: dict
}
class DomainError
class InfrastructureError
class InferenceError
class InferenceTimeoutError
class SpatialComputingError
class MangaProcessingError
class VideoProcessingError
class ImageGenerationError
class AdapterLoadError
class ContentModerationError
class KnowledgeGraphQueryError
AnimetixBaseError <|-- InfrastructureError --|> AdapterLoadError
AnimetixBaseError <|-- InfrastructureError --|> ContentModerationError
AnimetixBaseError <|-- InfrastructureError --|> KnowledgeGraphQueryError
```
---
## 11. Access & Deployment Environments
### A. Local Development Environment
The frontend and backend run in isolation to support Hot Module Replacement (HMR).
- **Backend (Django)**:
- URL: `http://localhost:8000`
- Command: `python backend/api/manage.py runserver`
- **Frontend (Vite / React)**:
- URL: `http://localhost:5173`
- Command: `cd frontend && npm run dev`
- *Note*: Vite automatically proxies `/api/*` and `/ws/*` calls to the Django instance.
### B. Dev / Staging Environment (Docker)
Containers package the entire infrastructure stack, serving the pre-built React frontend directly from the Django static server.
- **Standard Docker**:
- URL: `http://localhost:8000`
- Command: `docker-compose -f deploy/docker-compose.yml up`
- **Staging Docker** (includes debugging & experimental feature flags):
- URL: `http://localhost:8080`
- Command: `docker-compose -f deploy/docker-compose.yml -f deploy/docker-compose.staging.yml up`
### C. Production Environment (Google Cloud Run)
Production runs on **Cloud Run** (region `europe-west9`), behind the custom domain **https://animetix.xyz** (proxied by a Cloudflare Worker, since europe-west9 has no native domain mapping).
- **Services**: `animetix-web` (Django ASGI + built SPA, `europe-west9`) plus **two** scale-to-zero L4 GPU services in `europe-west1` β€” `animetix-brain` (vision) and `animetix-llm` (text). A small **Compute Engine VM** (`europe-west9-b`) runs Suwayomi-Server behind Caddy, asleep by default, woken on demand, with an idle-shutdown timer; it is deployed by hand via `scripts/deploy/gcp/deploy_suwayomi.py`.
- **Cloud Run Jobs**: 10 declared, **7 scheduled**. Two are on-demand only (drift baselines, character-index backfill) and one β€” `animetix-data-ingestion` β€” is deliberately **unscheduled** after it was measured burning ~4 h of GPU a night while completing zero times. `deployments.yaml` carries the reason next to each.
- **Secrets**: every credential is mounted from **Secret Manager** (`secretKeyRef`) β€” no secrets ship in images or env files.
- **Pipeline**: deploys are **manual-only** via `gh workflow run ci.yml -f deploy_to_prod=true` (a push never deploys). All deployment parameters live in the declarative [deploy/deployments.yaml](../deploy/deployments.yaml).
> **The deploy workflow builds only the web image.** The two GPU images and the Suwayomi VM are built and deployed separately. A dependency added for a brain or LLM feature therefore **never reaches production through the normal deploy**, and the workflow stays green while the feature is dead. Verify the live service, not the workflow's colour β€” and read the logs, not `gcloud run services describe`, which reported `Ready: True` throughout a twelve-day outage.
### D. Mirror Deployment (Hugging Face Spaces)
A container mirror can be published to **Hugging Face Spaces** (`https://huggingface.co/spaces/MissawB/Animetix`, internal port `7860`) via `deploy_to_hf.yml` / `scripts/deploy/huggingface/hf_deploy.py`.
---
*Last reviewed against the codebase: 2026-09-07. The sections most likely to drift are Β§7 (the inference chain and the brain/LLM split) and Β§11 (deployment topology) β€” both changed substantially in August 2026.*