Spaces:
Sleeping
Sleeping
Dmitry Beresnev commited on
Commit Β·
a51a89f
1
Parent(s): 332826f
add new architectural diagram to readme file
Browse files
README.md
CHANGED
|
@@ -136,6 +136,50 @@ for model in cache_info['cached_models']:
|
|
| 136 |
|
| 137 |
## ποΈ Architecture
|
| 138 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 139 |
### Model Caching System
|
| 140 |
|
| 141 |
The API uses an intelligent LRU (Least Recently Used) cache to manage models in memory:
|
|
@@ -270,11 +314,12 @@ The web-augmented chat endpoint automatically:
|
|
| 270 |
|
| 271 |
```
|
| 272 |
AGI/
|
| 273 |
-
βββ
|
| 274 |
-
βββ
|
|
|
|
| 275 |
βββ Dockerfile # Docker configuration
|
| 276 |
-
βββ pyproject.toml
|
| 277 |
-
βββ README.md
|
| 278 |
```
|
| 279 |
|
| 280 |
### Adding New Models
|
|
|
|
| 136 |
|
| 137 |
## ποΈ Architecture
|
| 138 |
|
| 139 |
+
### Current Control Plane Architecture
|
| 140 |
+
|
| 141 |
+
The repository now contains a C++ control plane for `llama-server` in addition to the legacy Python path.
|
| 142 |
+
|
| 143 |
+
Detailed diagram:
|
| 144 |
+
- [`docs/CPP_MANAGER_ARCHITECTURE_ASCII.md`](/Users/dmitryberesnev/Project/huggingface/agi/AGI/docs/CPP_MANAGER_ARCHITECTURE_ASCII.md)
|
| 145 |
+
|
| 146 |
+
```text
|
| 147 |
+
client
|
| 148 |
+
|
|
| 149 |
+
v
|
| 150 |
+
[llm-manager / Boost.Beast]
|
| 151 |
+
|- auth
|
| 152 |
+
|- token validation
|
| 153 |
+
|- rate limiting
|
| 154 |
+
|- request registry
|
| 155 |
+
|- bounded priority queue
|
| 156 |
+
|- scheduler
|
| 157 |
+
|- queue metrics
|
| 158 |
+
v
|
| 159 |
+
[active llama-server worker]
|
| 160 |
+
```
|
| 161 |
+
|
| 162 |
+
### Components
|
| 163 |
+
|
| 164 |
+
- `llm-manager (Boost.Beast)`: Main C++ HTTP server that exposes chat, control, cancel, and queue-metrics endpoints.
|
| 165 |
+
- `Config Layer`: Loads runtime settings from `config.toml` and environment variables for server, queue, auth, limits, and backend process settings.
|
| 166 |
+
- `API Key Auth`: Authenticates `Authorization: Bearer <token>` requests and maps them to `admin` or `user` roles.
|
| 167 |
+
- `Token Validation`: Estimates prompt size from `messages + max_tokens` and rejects oversized requests before they hit the backend.
|
| 168 |
+
- `Rate Limiter`: Applies per-key request and estimated-token budgets to protect the system from overload.
|
| 169 |
+
- `Request Registry`: Tracks request lifecycle by `request_id`, including queued, running, cancelled, failed, and completed states.
|
| 170 |
+
- `Priority Queue`: Holds queued work with bounded size and bounded token budget, and separates `admin` and `user` traffic.
|
| 171 |
+
- `Scheduler`: Pulls work from the queue, applies fairness, and forwards one request at a time to the active backend worker.
|
| 172 |
+
- `ModelManager`: Owns `llama-server` process lifecycle, including startup, readiness checks, model switching, and hard restarts.
|
| 173 |
+
- `Queue Metrics`: Exposes queue and runtime telemetry at `GET /queue/metrics` in Prometheus-style text format.
|
| 174 |
+
- `Active llama-server Worker`: The currently loaded backend inference process that serves `/v1/chat/completions`.
|
| 175 |
+
|
| 176 |
+
### Runtime Notes
|
| 177 |
+
|
| 178 |
+
- Only one backend worker is active at a time.
|
| 179 |
+
- Queue admission is bounded by both request count and estimated tokens.
|
| 180 |
+
- Per-request cancel is best-effort for running inference and may restart the active worker.
|
| 181 |
+
- Streaming is not enabled by default and is explicitly rejected until relay support is implemented.
|
| 182 |
+
|
| 183 |
### Model Caching System
|
| 184 |
|
| 185 |
The API uses an intelligent LRU (Least Recently Used) cache to manage models in memory:
|
|
|
|
| 314 |
|
| 315 |
```
|
| 316 |
AGI/
|
| 317 |
+
βββ cpp/ # C++ llm-manager control plane
|
| 318 |
+
βββ python/ # Legacy Python application modules
|
| 319 |
+
βββ docs/ # Design docs and architecture notes
|
| 320 |
βββ Dockerfile # Docker configuration
|
| 321 |
+
βββ pyproject.toml # Python dependencies
|
| 322 |
+
βββ README.md # This file
|
| 323 |
```
|
| 324 |
|
| 325 |
### Adding New Models
|