olanigan commited on
Commit
eab0cb2
·
verified ·
1 Parent(s): b1b2f5e

Update README: streamline benchmarks and quickstarts

Browse files
Files changed (1) hide show
  1. README.md +9 -28
README.md CHANGED
@@ -69,29 +69,26 @@ model-index:
69
 
70
  ## ⚡ Executive Summary
71
 
72
- Modern AI agent frameworks (Antigravity, Claude Code, Cursor, LangChain) spend up to 40% of their compute budget and hundreds of milliseconds waiting for frontier LLMs (Claude 3.5 Sonnet, GPT-4o) to decide which tool or execution tier to invoke.
73
 
74
  **`HomeRouter-S1`** is a non-autoregressive, calibrated Small Decision Model (SDM) fine-tuned on 20,582 multi-turn agent execution turns. Built upon ModernBERT and the `LayaForDecision` architecture, it classifies agent tool intents and tier triage in a single forward pass without autoregressive text decoding:
75
 
76
  * **Sub-26ms CPU Latency (ONNX)** / **Sub-43ms Apple Silicon Latency (Ollama Metal)**.
77
- * **Zero Output Tokens ($N_{\text{out}} = 0$)**: Bypasses generative LLM decode loops entirely.
78
  * **100% Strict Enum Guarantee**: 0.00% schema parse errors, eliminating JSON formatting repair loops.
79
- * **Calibrated Confidence Gating**: Fitted temperature ($T = 1.0618$) achieves $\text{ECE} \le 0.0363$. Applying a threshold of $\tau = 0.90$ escalates ambiguous prompts to Tier-2 frontier models while safely auto-dispatching high-confidence turns.
80
- * **Avoided Frontier Spend**: Saves up to **$4,500.00 per 1M turns** in avoided frontier API calls (FOCUS 1.5 normalized).
81
 
82
  ---
83
 
84
  ## 📊 Dual-Engine Silicon Benchmark
85
 
86
- Tested across local host CPU and Apple Silicon Metal against frontier cloud models:
87
 
88
- | Model / Architecture | Mode / Runtime | TTFD (P50) | TTFD (P95) | Output Tokens ($N_{\text{out}}$) | Parse Error Rate | Calibrated ECE | FOCUS 1.5 Avoided Spend / 1M Turns |
89
- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- |
90
- | **Claude 3.5 Sonnet** | Cloud API (Prompt JSON) | 1,420 ms | 3,180 ms | ~280 tokens | 2.14% | 0.0820 | Baseline ($0.00) |
91
- | **GPT-4o** | Cloud API (Structured Output) | 960 ms | 2,410 ms | ~210 tokens | 1.80% | 0.0760 | -$700.00 |
92
- | **DeepSeek R1** | Cloud API (Reasoning Loop) | 4,200 ms | 8,900 ms | ~1,200 tokens | 0.95% | 0.0910 | +$1,700.00 |
93
- | **Homestead Native ONNX (CPU)** | **Direct FP32 Forward Pass** | **25.69 ms** | **26.00 ms** | **0 tokens** | **0.00%** | **0.0363** | **$4,500.00 (100%)** |
94
- | **Ollama MLX (Metal)** | **Native LayaForDecision** | **40.97 ms** | **42.50 ms** | **0 tokens** | **0.00%** | **0.0363** | **$4,500.00 (100%)** |
95
 
96
  ---
97
 
@@ -168,22 +165,6 @@ ollama create olanigan/HomeRouter-S1 -f ollama_laya/Modelfile
168
  ollama run olanigan/HomeRouter-S1 "find all *.ts files matching client"
169
  ```
170
 
171
- ### 4. TypeScript / Node via `s1sdk`
172
- ```typescript
173
- import { S1Client } from "@homestead/s1sdk";
174
-
175
- const s1 = new S1Client();
176
- const route = await s1.route("Update the database migration script", [
177
- "edit_file",
178
- "write",
179
- "run_command",
180
- "tier_frontier"
181
- ]);
182
-
183
- console.log(route.choice); // "edit_file"
184
- console.log(route.confidence); // 0.9412
185
- ```
186
-
187
  ---
188
 
189
  ## 📦 Package Contents & File Sizes
 
69
 
70
  ## ⚡ Executive Summary
71
 
72
+ Modern AI agent frameworks (Antigravity, Claude Code, Cursor, LangChain) spend substantial compute budgets and latency classifying routing intents, triaging tool dispatches, or determining next actions.
73
 
74
  **`HomeRouter-S1`** is a non-autoregressive, calibrated Small Decision Model (SDM) fine-tuned on 20,582 multi-turn agent execution turns. Built upon ModernBERT and the `LayaForDecision` architecture, it classifies agent tool intents and tier triage in a single forward pass without autoregressive text decoding:
75
 
76
  * **Sub-26ms CPU Latency (ONNX)** / **Sub-43ms Apple Silicon Latency (Ollama Metal)**.
77
+ * **Zero Output Tokens ($N_{\text{out}} = 0$)**: Evaluates entire decision candidate spaces in a single forward pass, bypassing generative decode loops entirely.
78
  * **100% Strict Enum Guarantee**: 0.00% schema parse errors, eliminating JSON formatting repair loops.
79
+ * **Calibrated Confidence Gating**: Fitted temperature ($T = 1.0618$) achieves $\text{ECE} \le 0.0363$. Applying a threshold of $\tau = 0.90$ escalates ambiguous prompts to secondary tiers while safely auto-dispatching high-confidence turns.
80
+ * **Local Silicon Efficiency**: Eliminates external network roundtrips and recurring per-token cloud inference costs.
81
 
82
  ---
83
 
84
  ## 📊 Dual-Engine Silicon Benchmark
85
 
86
+ Performance telemetry across local host CPU and Apple Silicon Metal runtimes:
87
 
88
+ | Model / Architecture | Mode / Runtime | TTFD (P50) | TTFD (P95) | Output Tokens ($N_{\text{out}}$) | Parse Error Rate | Calibrated ECE |
89
+ | :--- | :--- | :--- | :--- | :--- | :--- | :--- |
90
+ | **Homestead Native ONNX (CPU)** | **Direct FP32 Forward Pass** | **25.69 ms** | **26.00 ms** | **0 tokens** | **0.00%** | **0.0363** |
91
+ | **Ollama MLX (Metal)** | **Native LayaForDecision** | **40.97 ms** | **42.50 ms** | **0 tokens** | **0.00%** | **0.0363** |
 
 
 
92
 
93
  ---
94
 
 
165
  ollama run olanigan/HomeRouter-S1 "find all *.ts files matching client"
166
  ```
167
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
168
  ---
169
 
170
  ## 📦 Package Contents & File Sizes