Update README: streamline benchmarks and quickstarts

eab0cb2 verified 23 days ago

7.34 kB

---
language:
- en
license: apache-2.0
tags:
- system-1
- decision-model
- small-decision-model
- modernbert
- intent-routing
- agentic-router
- onnx
- ollama
- finops
- sovereign-ai
- focus-1.5
- s1sdk
- homestead
pipeline_tag: text-classification
base_model: convaiinnovations/laya
library_name: transformers
metrics:
- accuracy
- f1
- brier_score
- expected_calibration_error
model-index:
- name: HomeRouter-S1
  results:
  - task:
      type: text-classification
      name: Agentic Tool Intent Routing & Tier Triage
    dataset:
      name: Multi-Turn Agentic Trajectories (20,582 turns across 10 tools)
      type: proprietary
    metrics:
    - name: Test Accuracy
      type: accuracy
      value: 0.6563
    - name: Validation Macro-F1
      type: f1
      value: 0.5082
    - name: Calibrated ECE
      type: expected_calibration_error
      value: 0.0363
    - name: Calibrated Brier Score
      type: brier_score
      value: 0.4486
    - name: Temperature T
      type: temperature
      value: 1.0618
    - name: Recommended Gate Threshold
      type: threshold
      value: 0.90
---
# HomeRouter-S1 (`olanigan/HomeRouter-S1`)

> **Calibrated 120M System-1 Small Decision Model (SDM) for sub-26ms, zero-parsing-failure agentic tool intent routing and tier triage.**

[![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
[![Base Model](https://img.shields.io/badge/Base_Model-convaiinnovations%2Flaya-orange)](https://huggingface.co/convaiinnovations/laya)
[![Parameters](https://img.shields.io/badge/Parameters-120M-green)](https://huggingface.co/olanigan/HomeRouter-S1)
[![ONNX CPU Latency](https://img.shields.io/badge/P95_Latency_ONNX-26.0ms-brightgreen)](https://github.com/homestead-labs)
[![Ollama Metal Latency](https://img.shields.io/badge/P95_Latency_Metal-42.5ms-brightgreen)](https://github.com/homestead-labs)
[![Ecosystem](https://img.shields.io/badge/Ecosystem-s1sdk%20%7C%20ollama%20%7C%20onnx-purple)](https://github.com/homestead-labs)

---

## ⚡ Executive Summary

Modern AI agent frameworks (Antigravity, Claude Code, Cursor, LangChain) spend substantial compute budgets and latency classifying routing intents, triaging tool dispatches, or determining next actions.

**`HomeRouter-S1`** is a non-autoregressive, calibrated Small Decision Model (SDM) fine-tuned on 20,582 multi-turn agent execution turns. Built upon ModernBERT and the `LayaForDecision` architecture, it classifies agent tool intents and tier triage in a single forward pass without autoregressive text decoding:

* **Sub-26ms CPU Latency (ONNX)** / **Sub-43ms Apple Silicon Latency (Ollama Metal)**.
* **Zero Output Tokens ($N_{\text{out}} = 0$)**: Evaluates entire decision candidate spaces in a single forward pass, bypassing generative decode loops entirely.
* **100% Strict Enum Guarantee**: 0.00% schema parse errors, eliminating JSON formatting repair loops.
* **Calibrated Confidence Gating**: Fitted temperature ($T = 1.0618$) achieves $\text{ECE} \le 0.0363$. Applying a threshold of $\tau = 0.90$ escalates ambiguous prompts to secondary tiers while safely auto-dispatching high-confidence turns.
* **Local Silicon Efficiency**: Eliminates external network roundtrips and recurring per-token cloud inference costs.
---
## 📊 Dual-Engine Silicon Benchmark
Performance telemetry across local host CPU and Apple Silicon Metal runtimes:
| Model / Architecture | Mode / Runtime | TTFD (P50) | TTFD (P95) | Output Tokens ($N_{\text{out}}$) | Parse Error Rate | Calibrated ECE |
| :--- | :--- | :--- | :--- | :--- | :--- | :--- |
| **Homestead Native ONNX (CPU)** | **Direct FP32 Forward Pass** | **25.69 ms** | **26.00 ms** | **0 tokens** | **0.00%** | **0.0363** |
| **Ollama MLX (Metal)** | **Native LayaForDecision** | **40.97 ms** | **42.50 ms** | **0 tokens** | **0.00%** | **0.0363** |

---

## 🏷️ Supported Tool & Tier Catalog (10 Classes)

The model routes incoming agent dialogue turns across 10 discrete tools and meta-triage tiers:

```
0: bash           - Direct terminal command execution
1: edit_file      - Exact in-place text replacement
2: grep_search    - Codebase symbol & pattern search
3: read           - File inspection & document reading
4: run_command    - Workspace command & test runner execution
5: tier_cloud     - Cloud utility / mid-tier model routing
6: tier_frontier  - High-complexity escalation to Claude Sonnet / R1
7: tier_local     - On-device local SLM routing
8: view_file      - Read-only file preview
9: write          - Full file creation & overwriting
```

---

## 🚀 Quickstarts

### 1. Python (`transformers`)
```python
import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification
model_id = "olanigan/HomeRouter-S1"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSequenceClassification.from_pretrained(model_id)
query = "Run the unit tests in packages/s1-train and check if they pass"
inputs = tokenizer(query, return_tensors="pt")
with torch.no_grad():
    logits = model(**inputs).logits
# Calibrated temperature scaling
T = 1.0618
probs = torch.softmax(logits / T, dim=-1)
pred_idx = torch.argmax(probs, dim=-1).item()
confidence = probs[0, pred_idx].item()
action = model.config.id2label[pred_idx]
print(f"Predicted Action: {action} (Confidence: {confidence:.4f})")
print(f"Escalate to Frontier: {confidence < 0.90}")
```

### 2. Fast CPU Serving with ONNX Runtime
```python
import numpy as np
import onnxruntime as ort
from transformers import AutoTokenizer
session = ort.InferenceSession("model.onnx")
tokenizer = AutoTokenizer.from_pretrained("olanigan/HomeRouter-S1")
inputs = tokenizer("cat src/index.ts", return_tensors="np")
ort_inputs = {
    "input_ids": inputs["input_ids"],
    "attention_mask": inputs["attention_mask"]
}
logits = session.run(None, ort_inputs)[0]
probs = np.exp(logits / 1.0618) / np.sum(np.exp(logits / 1.0618), axis=-1, keepdims=True)
print("Predicted tool index:", np.argmax(probs))
```

### 3. Local Silicon Execution with Ollama
```bash
# Pull or run directly once published, or build locally from the bundle:
ollama create olanigan/HomeRouter-S1 -f ollama_laya/Modelfile
ollama run olanigan/HomeRouter-S1 "find all *.ts files matching client"
```

---

## 📦 Package Contents & File Sizes

| File / Folder | Format / Target | Size | Description |
|---|---|---|---|
| `model.safetensors` | BF16 (PyTorch / Transformers) | **239.5 MB** | Standard weights for Python transformers & pipelines |
| `model.onnx` | FP32 (ONNX Runtime / CPU) | **479.1 MB** | Standalone zero-dependency graph for local CPU serving |
| `ollama_laya/` | FP32 Metal (Ollama MLX) | **460.2 MB** | 33-tensor layout optimized for Apple Silicon Metal runtime |
| `tokenizer/` | Fast Tokenizers BPE | **3.5 MB** | ModernBERT fast tokenizer configuration & vocab |
| `manifest.json` | JSON | **1.1 KB** | Calibration parameters, ECE $\le 0.0363$, seed, and provenance |
| `config.json` | JSON | **1.7 KB** | Architecture specification and `id2label` mapping |

---

## 📜 Citation

```bibtex
@software{homerouter_s1_2026,
  author = {olanigan and Homestead Labs},
  title = {HomeRouter-S1: Calibrated 120M System-1 Small Decision Model for Agentic Tool Dispatch},
  year = {2026},
  url = {https://huggingface.co/olanigan/HomeRouter-S1}
}
```

	---
	language:
	- en
	license: apache-2.0
	tags:
	- system-1
	- decision-model
	- small-decision-model
	- modernbert
	- intent-routing
	- agentic-router
	- onnx
	- ollama
	- finops
	- sovereign-ai
	- focus-1.5
	- s1sdk
	- homestead
	pipeline_tag: text-classification
	base_model: convaiinnovations/laya
	library_name: transformers
	metrics:
	- accuracy
	- f1
	- brier_score
	- expected_calibration_error
	model-index:
	- name: HomeRouter-S1
	results:
	- task:
	type: text-classification
	name: Agentic Tool Intent Routing & Tier Triage
	dataset:
	name: Multi-Turn Agentic Trajectories (20,582 turns across 10 tools)
	type: proprietary
	metrics:
	- name: Test Accuracy
	type: accuracy
	value: 0.6563
	- name: Validation Macro-F1
	type: f1
	value: 0.5082
	- name: Calibrated ECE
	type: expected_calibration_error
	value: 0.0363
	- name: Calibrated Brier Score
	type: brier_score
	value: 0.4486
	- name: Temperature T
	type: temperature
	value: 1.0618
	- name: Recommended Gate Threshold
	type: threshold
	value: 0.90
	---

	# HomeRouter-S1 (`olanigan/HomeRouter-S1`)

	> Calibrated 120M System-1 Small Decision Model (SDM) for sub-26ms, zero-parsing-failure agentic tool intent routing and tier triage.

	[![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
	[![Base Model](https://img.shields.io/badge/Base_Model-convaiinnovations%2Flaya-orange)](https://huggingface.co/convaiinnovations/laya)
	[![Parameters](https://img.shields.io/badge/Parameters-120M-green)](https://huggingface.co/olanigan/HomeRouter-S1)
	[![ONNX CPU Latency](https://img.shields.io/badge/P95_Latency_ONNX-26.0ms-brightgreen)](https://github.com/homestead-labs)
	[![Ollama Metal Latency](https://img.shields.io/badge/P95_Latency_Metal-42.5ms-brightgreen)](https://github.com/homestead-labs)
	[![Ecosystem](https://img.shields.io/badge/Ecosystem-s1sdk%20%7C%20ollama%20%7C%20onnx-purple)](https://github.com/homestead-labs)

	---

	## ⚡ Executive Summary

	Modern AI agent frameworks (Antigravity, Claude Code, Cursor, LangChain) spend substantial compute budgets and latency classifying routing intents, triaging tool dispatches, or determining next actions.

	`HomeRouter-S1` is a non-autoregressive, calibrated Small Decision Model (SDM) fine-tuned on 20,582 multi-turn agent execution turns. Built upon ModernBERT and the `LayaForDecision` architecture, it classifies agent tool intents and tier triage in a single forward pass without autoregressive text decoding:

	* Sub-26ms CPU Latency (ONNX) / Sub-43ms Apple Silicon Latency (Ollama Metal).
	* Zero Output Tokens ($N_{\text{out}} = 0$): Evaluates entire decision candidate spaces in a single forward pass, bypassing generative decode loops entirely.
	* 100% Strict Enum Guarantee: 0.00% schema parse errors, eliminating JSON formatting repair loops.
	* Calibrated Confidence Gating: Fitted temperature ($T = 1.0618$) achieves $\text{ECE} \le 0.0363$. Applying a threshold of $\tau = 0.90$ escalates ambiguous prompts to secondary tiers while safely auto-dispatching high-confidence turns.
	* Local Silicon Efficiency: Eliminates external network roundtrips and recurring per-token cloud inference costs.

	---

	## 📊 Dual-Engine Silicon Benchmark

	Performance telemetry across local host CPU and Apple Silicon Metal runtimes:

	\| Model / Architecture \| Mode / Runtime \| TTFD (P50) \| TTFD (P95) \| Output Tokens ($N_{\text{out}}$) \| Parse Error Rate \| Calibrated ECE \|
	\| :--- \| :--- \| :--- \| :--- \| :--- \| :--- \| :--- \|
	\| Homestead Native ONNX (CPU) \| Direct FP32 Forward Pass \| 25.69 ms \| 26.00 ms \| 0 tokens \| 0.00% \| 0.0363 \|
	\| Ollama MLX (Metal) \| Native LayaForDecision \| 40.97 ms \| 42.50 ms \| 0 tokens \| 0.00% \| 0.0363 \|

	---

	## 🏷️ Supported Tool & Tier Catalog (10 Classes)

	The model routes incoming agent dialogue turns across 10 discrete tools and meta-triage tiers:

	```
	0: bash - Direct terminal command execution
	1: edit_file - Exact in-place text replacement
	2: grep_search - Codebase symbol & pattern search
	3: read - File inspection & document reading
	4: run_command - Workspace command & test runner execution
	5: tier_cloud - Cloud utility / mid-tier model routing
	6: tier_frontier - High-complexity escalation to Claude Sonnet / R1
	7: tier_local - On-device local SLM routing
	8: view_file - Read-only file preview
	9: write - Full file creation & overwriting
	```

	---

	## 🚀 Quickstarts

	### 1. Python (`transformers`)
	```python
	import torch
	from transformers import AutoTokenizer, AutoModelForSequenceClassification

	model_id = "olanigan/HomeRouter-S1"
	tokenizer = AutoTokenizer.from_pretrained(model_id)
	model = AutoModelForSequenceClassification.from_pretrained(model_id)

	query = "Run the unit tests in packages/s1-train and check if they pass"
	inputs = tokenizer(query, return_tensors="pt")

	with torch.no_grad():
	logits = model(**inputs).logits

	# Calibrated temperature scaling
	T = 1.0618
	probs = torch.softmax(logits / T, dim=-1)
	pred_idx = torch.argmax(probs, dim=-1).item()
	confidence = probs[0, pred_idx].item()

	action = model.config.id2label[pred_idx]
	print(f"Predicted Action: {action} (Confidence: {confidence:.4f})")
	print(f"Escalate to Frontier: {confidence < 0.90}")
	```

	### 2. Fast CPU Serving with ONNX Runtime
	```python
	import numpy as np
	import onnxruntime as ort
	from transformers import AutoTokenizer

	session = ort.InferenceSession("model.onnx")
	tokenizer = AutoTokenizer.from_pretrained("olanigan/HomeRouter-S1")

	inputs = tokenizer("cat src/index.ts", return_tensors="np")
	ort_inputs = {
	"input_ids": inputs["input_ids"],
	"attention_mask": inputs["attention_mask"]
	}
	logits = session.run(None, ort_inputs)[0]
	probs = np.exp(logits / 1.0618) / np.sum(np.exp(logits / 1.0618), axis=-1, keepdims=True)
	print("Predicted tool index:", np.argmax(probs))
	```

	### 3. Local Silicon Execution with Ollama
	```bash
	# Pull or run directly once published, or build locally from the bundle:
	ollama create olanigan/HomeRouter-S1 -f ollama_laya/Modelfile
	ollama run olanigan/HomeRouter-S1 "find all *.ts files matching client"
	```

	---

	## 📦 Package Contents & File Sizes

	\| File / Folder \| Format / Target \| Size \| Description \|
	\|---\|---\|---\|---\|
	\| `model.safetensors` \| BF16 (PyTorch / Transformers) \| 239.5 MB \| Standard weights for Python transformers & pipelines \|
	\| `model.onnx` \| FP32 (ONNX Runtime / CPU) \| 479.1 MB \| Standalone zero-dependency graph for local CPU serving \|
	\| `ollama_laya/` \| FP32 Metal (Ollama MLX) \| 460.2 MB \| 33-tensor layout optimized for Apple Silicon Metal runtime \|
	\| `tokenizer/` \| Fast Tokenizers BPE \| 3.5 MB \| ModernBERT fast tokenizer configuration & vocab \|
	\| `manifest.json` \| JSON \| 1.1 KB \| Calibration parameters, ECE $\le 0.0363$, seed, and provenance \|
	\| `config.json` \| JSON \| 1.7 KB \| Architecture specification and `id2label` mapping \|

	---

	## 📜 Citation

	```bibtex
	@software{homerouter_s1_2026,
	author = {olanigan and Homestead Labs},
	title = {HomeRouter-S1: Calibrated 120M System-1 Small Decision Model for Agentic Tool Dispatch},
	year = {2026},
	url = {https://huggingface.co/olanigan/HomeRouter-S1}
	}
	```