IDKHowToCodeFr commited on
Commit
216837a
·
1 Parent(s): c53fce6

docs: update architecture and README with actual model strings

Browse files
Files changed (2) hide show
  1. README.md +4 -4
  2. architecture.md +4 -4
README.md CHANGED
@@ -26,10 +26,10 @@ The system operates on a highly optimized, deterministic fail-fast fallback casc
26
  ### 🌊 4-Tier Fail-Fast Cascade
27
  Instead of relying on a single point of failure, the engine delegates composition to a highly available 4-tier chain:
28
 
29
- 1. **Tier 1 (Speed & Intelligence)**: NVIDIA NIM (`llama-3.3-70b-instruct`) - 5s timeout.
30
- 2. **Tier 2 (Fast Failover)**: Groq (`llama-3.3-70b-versatile`) - 4s timeout.
31
  3. **Tier 3 (Secondary Failover)**: Gemini (`gemini-2.5-flash`) - 4s timeout.
32
- 4. **Tier 4 (Local Airgap)**: Natively executed `qwen2.5-3b-instruct` via `llama-cpp-python` with **strict LlamaGrammar enforcement**.
33
 
34
  > [!TIP]
35
  > **Dynamic Time Budgeting**
@@ -62,5 +62,5 @@ Instead of relying on a single point of failure, the engine delegates compositio
62
 
63
  * **Framework**: FastAPI / Uvicorn (Port `8080`)
64
  * **Local SLM Execution**: `llama-cpp-python` (Pre-compiled CPU Wheels)
65
- * **Models**: `Llama 3.3 70B`, `Gemini 2.5 Flash`, `Qwen2.5 3B`
66
  * **Infrastructure**: Dockerized `python:3.10-slim` on Hugging Face Spaces
 
26
  ### 🌊 4-Tier Fail-Fast Cascade
27
  Instead of relying on a single point of failure, the engine delegates composition to a highly available 4-tier chain:
28
 
29
+ 1. **Tier 1 (Speed & Intelligence)**: NVIDIA NIM (`nvidia/nemotron-3-ultra-550b-a55b`) - 5s timeout.
30
+ 2. **Tier 2 (Fast Failover)**: Groq (`openai/gpt-oss-20b`) - 4s timeout.
31
  3. **Tier 3 (Secondary Failover)**: Gemini (`gemini-2.5-flash`) - 4s timeout.
32
+ 4. **Tier 4 (Local Airgap)**: Natively executed `qwen2.5-7b-instruct` or `qwen2.5-3b-instruct` (based on system latency) via `llama-cpp-python` with **strict LlamaGrammar enforcement**.
33
 
34
  > [!TIP]
35
  > **Dynamic Time Budgeting**
 
62
 
63
  * **Framework**: FastAPI / Uvicorn (Port `8080`)
64
  * **Local SLM Execution**: `llama-cpp-python` (Pre-compiled CPU Wheels)
65
+ * **Models**: `Nemotron 3 Ultra 550B`, `GPT OSS 20B`, `Gemini 2.5 Flash`, `Qwen2.5 7B/3B`
66
  * **Infrastructure**: Dockerized `python:3.10-slim` on Hugging Face Spaces
architecture.md CHANGED
@@ -112,10 +112,10 @@ To ensure 100% uptime and the best possible latency, the system utilizes a **Tie
112
  ```mermaid
113
  flowchart TD
114
  A[Generate Message Request] --> B{4-Tier Fallback Chain}
115
- B -->|Tier 1: Speed & Intel| C[NVIDIA NIM: llama-3.3-70b]
116
- B -->|Tier 2: Fast Failover| D[Groq: llama-3.3-70b]
117
  B -->|Tier 3: Sec. Failover| E[Gemini: 2.5-flash]
118
- B -->|Tier 4: Local Airgap| F[Local Llama-CPP: qwen2.5-3b-instruct]
119
 
120
  C -.->|Timeout / Error| D
121
  D -.->|Timeout / Error| E
@@ -129,7 +129,7 @@ flowchart TD
129
  ```
130
 
131
  * **Tier 1-3 (Cloud-Speed Models):** Defaults to robust APIs with fail-fast HTTP error handling. Since inference happens off-server, the system simply proxies JSON, operating well within the 30s limit.
132
- * **Tier 4 (The Local Safety Net):** Gracefully cascades to `qwen2.5-3b-instruct-q4_k_m.gguf` via `llama-cpp-python` if cloud providers fail or time out.
133
  * **Auto-Repair Loop**: Outputs are parsed through `validate_output`. If a taboo word, word-count breach, or malformed schema is detected, the engine dynamically recalculates the tier budget and issues a self-correction prompt *before* failing over to the next tier.
134
 
135
  ---
 
112
  ```mermaid
113
  flowchart TD
114
  A[Generate Message Request] --> B{4-Tier Fallback Chain}
115
+ B -->|Tier 1: Speed & Intel| C[NVIDIA NIM: nemotron-3-ultra-550b-a55b]
116
+ B -->|Tier 2: Fast Failover| D[Groq: gpt-oss-20b]
117
  B -->|Tier 3: Sec. Failover| E[Gemini: 2.5-flash]
118
+ B -->|Tier 4: Local Airgap| F[Local Llama-CPP: qwen2.5-7b-instruct / 3b-instruct]
119
 
120
  C -.->|Timeout / Error| D
121
  D -.->|Timeout / Error| E
 
129
  ```
130
 
131
  * **Tier 1-3 (Cloud-Speed Models):** Defaults to robust APIs with fail-fast HTTP error handling. Since inference happens off-server, the system simply proxies JSON, operating well within the 30s limit.
132
+ * **Tier 4 (The Local Safety Net):** Gracefully cascades to `qwen2.5-7b-instruct-q4_k_m.gguf` or `qwen2.5-3b-instruct-q4_k_m.gguf` via `llama-cpp-python` if cloud providers fail or time out.
133
  * **Auto-Repair Loop**: Outputs are parsed through `validate_output`. If a taboo word, word-count breach, or malformed schema is detected, the engine dynamically recalculates the tier budget and issues a self-correction prompt *before* failing over to the next tier.
134
 
135
  ---