Dmitry Beresnev commited on
Commit
a51a89f
Β·
1 Parent(s): 332826f

add new architectural diagram to readme file

Browse files
Files changed (1) hide show
  1. README.md +49 -4
README.md CHANGED
@@ -136,6 +136,50 @@ for model in cache_info['cached_models']:
136
 
137
  ## πŸ—οΈ Architecture
138
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
139
  ### Model Caching System
140
 
141
  The API uses an intelligent LRU (Least Recently Used) cache to manage models in memory:
@@ -270,11 +314,12 @@ The web-augmented chat endpoint automatically:
270
 
271
  ```
272
  AGI/
273
- β”œβ”€β”€ app.py # Main FastAPI application
274
- β”œβ”€β”€ client_multi_model.py # Example client
 
275
  β”œβ”€β”€ Dockerfile # Docker configuration
276
- β”œβ”€β”€ pyproject.toml # Python dependencies
277
- └── README.md # This file
278
  ```
279
 
280
  ### Adding New Models
 
136
 
137
  ## πŸ—οΈ Architecture
138
 
139
+ ### Current Control Plane Architecture
140
+
141
+ The repository now contains a C++ control plane for `llama-server` in addition to the legacy Python path.
142
+
143
+ Detailed diagram:
144
+ - [`docs/CPP_MANAGER_ARCHITECTURE_ASCII.md`](/Users/dmitryberesnev/Project/huggingface/agi/AGI/docs/CPP_MANAGER_ARCHITECTURE_ASCII.md)
145
+
146
+ ```text
147
+ client
148
+ |
149
+ v
150
+ [llm-manager / Boost.Beast]
151
+ |- auth
152
+ |- token validation
153
+ |- rate limiting
154
+ |- request registry
155
+ |- bounded priority queue
156
+ |- scheduler
157
+ |- queue metrics
158
+ v
159
+ [active llama-server worker]
160
+ ```
161
+
162
+ ### Components
163
+
164
+ - `llm-manager (Boost.Beast)`: Main C++ HTTP server that exposes chat, control, cancel, and queue-metrics endpoints.
165
+ - `Config Layer`: Loads runtime settings from `config.toml` and environment variables for server, queue, auth, limits, and backend process settings.
166
+ - `API Key Auth`: Authenticates `Authorization: Bearer <token>` requests and maps them to `admin` or `user` roles.
167
+ - `Token Validation`: Estimates prompt size from `messages + max_tokens` and rejects oversized requests before they hit the backend.
168
+ - `Rate Limiter`: Applies per-key request and estimated-token budgets to protect the system from overload.
169
+ - `Request Registry`: Tracks request lifecycle by `request_id`, including queued, running, cancelled, failed, and completed states.
170
+ - `Priority Queue`: Holds queued work with bounded size and bounded token budget, and separates `admin` and `user` traffic.
171
+ - `Scheduler`: Pulls work from the queue, applies fairness, and forwards one request at a time to the active backend worker.
172
+ - `ModelManager`: Owns `llama-server` process lifecycle, including startup, readiness checks, model switching, and hard restarts.
173
+ - `Queue Metrics`: Exposes queue and runtime telemetry at `GET /queue/metrics` in Prometheus-style text format.
174
+ - `Active llama-server Worker`: The currently loaded backend inference process that serves `/v1/chat/completions`.
175
+
176
+ ### Runtime Notes
177
+
178
+ - Only one backend worker is active at a time.
179
+ - Queue admission is bounded by both request count and estimated tokens.
180
+ - Per-request cancel is best-effort for running inference and may restart the active worker.
181
+ - Streaming is not enabled by default and is explicitly rejected until relay support is implemented.
182
+
183
  ### Model Caching System
184
 
185
  The API uses an intelligent LRU (Least Recently Used) cache to manage models in memory:
 
314
 
315
  ```
316
  AGI/
317
+ β”œβ”€β”€ cpp/ # C++ llm-manager control plane
318
+ β”œβ”€β”€ python/ # Legacy Python application modules
319
+ β”œβ”€β”€ docs/ # Design docs and architecture notes
320
  β”œβ”€β”€ Dockerfile # Docker configuration
321
+ β”œβ”€β”€ pyproject.toml # Python dependencies
322
+ └── README.md # This file
323
  ```
324
 
325
  ### Adding New Models