Spaces:
Running
Running
Commit ·
216837a
1
Parent(s): c53fce6
docs: update architecture and README with actual model strings
Browse files- README.md +4 -4
- architecture.md +4 -4
README.md
CHANGED
|
@@ -26,10 +26,10 @@ The system operates on a highly optimized, deterministic fail-fast fallback casc
|
|
| 26 |
### 🌊 4-Tier Fail-Fast Cascade
|
| 27 |
Instead of relying on a single point of failure, the engine delegates composition to a highly available 4-tier chain:
|
| 28 |
|
| 29 |
-
1. **Tier 1 (Speed & Intelligence)**: NVIDIA NIM (`
|
| 30 |
-
2. **Tier 2 (Fast Failover)**: Groq (`
|
| 31 |
3. **Tier 3 (Secondary Failover)**: Gemini (`gemini-2.5-flash`) - 4s timeout.
|
| 32 |
-
4. **Tier 4 (Local Airgap)**: Natively executed `qwen2.5-3b-instruct` via `llama-cpp-python` with **strict LlamaGrammar enforcement**.
|
| 33 |
|
| 34 |
> [!TIP]
|
| 35 |
> **Dynamic Time Budgeting**
|
|
@@ -62,5 +62,5 @@ Instead of relying on a single point of failure, the engine delegates compositio
|
|
| 62 |
|
| 63 |
* **Framework**: FastAPI / Uvicorn (Port `8080`)
|
| 64 |
* **Local SLM Execution**: `llama-cpp-python` (Pre-compiled CPU Wheels)
|
| 65 |
-
* **Models**: `
|
| 66 |
* **Infrastructure**: Dockerized `python:3.10-slim` on Hugging Face Spaces
|
|
|
|
| 26 |
### 🌊 4-Tier Fail-Fast Cascade
|
| 27 |
Instead of relying on a single point of failure, the engine delegates composition to a highly available 4-tier chain:
|
| 28 |
|
| 29 |
+
1. **Tier 1 (Speed & Intelligence)**: NVIDIA NIM (`nvidia/nemotron-3-ultra-550b-a55b`) - 5s timeout.
|
| 30 |
+
2. **Tier 2 (Fast Failover)**: Groq (`openai/gpt-oss-20b`) - 4s timeout.
|
| 31 |
3. **Tier 3 (Secondary Failover)**: Gemini (`gemini-2.5-flash`) - 4s timeout.
|
| 32 |
+
4. **Tier 4 (Local Airgap)**: Natively executed `qwen2.5-7b-instruct` or `qwen2.5-3b-instruct` (based on system latency) via `llama-cpp-python` with **strict LlamaGrammar enforcement**.
|
| 33 |
|
| 34 |
> [!TIP]
|
| 35 |
> **Dynamic Time Budgeting**
|
|
|
|
| 62 |
|
| 63 |
* **Framework**: FastAPI / Uvicorn (Port `8080`)
|
| 64 |
* **Local SLM Execution**: `llama-cpp-python` (Pre-compiled CPU Wheels)
|
| 65 |
+
* **Models**: `Nemotron 3 Ultra 550B`, `GPT OSS 20B`, `Gemini 2.5 Flash`, `Qwen2.5 7B/3B`
|
| 66 |
* **Infrastructure**: Dockerized `python:3.10-slim` on Hugging Face Spaces
|
architecture.md
CHANGED
|
@@ -112,10 +112,10 @@ To ensure 100% uptime and the best possible latency, the system utilizes a **Tie
|
|
| 112 |
```mermaid
|
| 113 |
flowchart TD
|
| 114 |
A[Generate Message Request] --> B{4-Tier Fallback Chain}
|
| 115 |
-
B -->|Tier 1: Speed & Intel| C[NVIDIA NIM:
|
| 116 |
-
B -->|Tier 2: Fast Failover| D[Groq:
|
| 117 |
B -->|Tier 3: Sec. Failover| E[Gemini: 2.5-flash]
|
| 118 |
-
B -->|Tier 4: Local Airgap| F[Local Llama-CPP: qwen2.5-3b-instruct]
|
| 119 |
|
| 120 |
C -.->|Timeout / Error| D
|
| 121 |
D -.->|Timeout / Error| E
|
|
@@ -129,7 +129,7 @@ flowchart TD
|
|
| 129 |
```
|
| 130 |
|
| 131 |
* **Tier 1-3 (Cloud-Speed Models):** Defaults to robust APIs with fail-fast HTTP error handling. Since inference happens off-server, the system simply proxies JSON, operating well within the 30s limit.
|
| 132 |
-
* **Tier 4 (The Local Safety Net):** Gracefully cascades to `qwen2.5-3b-instruct-q4_k_m.gguf` via `llama-cpp-python` if cloud providers fail or time out.
|
| 133 |
* **Auto-Repair Loop**: Outputs are parsed through `validate_output`. If a taboo word, word-count breach, or malformed schema is detected, the engine dynamically recalculates the tier budget and issues a self-correction prompt *before* failing over to the next tier.
|
| 134 |
|
| 135 |
---
|
|
|
|
| 112 |
```mermaid
|
| 113 |
flowchart TD
|
| 114 |
A[Generate Message Request] --> B{4-Tier Fallback Chain}
|
| 115 |
+
B -->|Tier 1: Speed & Intel| C[NVIDIA NIM: nemotron-3-ultra-550b-a55b]
|
| 116 |
+
B -->|Tier 2: Fast Failover| D[Groq: gpt-oss-20b]
|
| 117 |
B -->|Tier 3: Sec. Failover| E[Gemini: 2.5-flash]
|
| 118 |
+
B -->|Tier 4: Local Airgap| F[Local Llama-CPP: qwen2.5-7b-instruct / 3b-instruct]
|
| 119 |
|
| 120 |
C -.->|Timeout / Error| D
|
| 121 |
D -.->|Timeout / Error| E
|
|
|
|
| 129 |
```
|
| 130 |
|
| 131 |
* **Tier 1-3 (Cloud-Speed Models):** Defaults to robust APIs with fail-fast HTTP error handling. Since inference happens off-server, the system simply proxies JSON, operating well within the 30s limit.
|
| 132 |
+
* **Tier 4 (The Local Safety Net):** Gracefully cascades to `qwen2.5-7b-instruct-q4_k_m.gguf` or `qwen2.5-3b-instruct-q4_k_m.gguf` via `llama-cpp-python` if cloud providers fail or time out.
|
| 133 |
* **Auto-Repair Loop**: Outputs are parsed through `validate_output`. If a taboo word, word-count breach, or malformed schema is detected, the engine dynamically recalculates the tier budget and issues a self-correction prompt *before* failing over to the next tier.
|
| 134 |
|
| 135 |
---
|