ettin-32m-effort-v1

README.md

@@ -0,0 +1,65 @@
+---
+license: mit
+base_model: jhu-clsp/ettin-encoder-32m
+pipeline_tag: text-classification
+language: en
+tags:
+  - onnx
+  - coding-assistant
+  - reasoning-effort
+  - claude-code
+  - opencode
+---
+
+# System1Router effort classifier
+
+Predicts the reasoning effort a coding-assistant turn needs (`low`, `medium`, `high`, `xhigh`, `max`) from the prompt and its recent conversation. It is the local classifier of [System1Router](https://github.com/Valkea/System1Router), which sets the effort of each Claude Code or OpenCode prompt automatically while keeping the model and its prompt cache.
+
+## Files
+
+| File | What |
+|---|---|
+| `ettin-32m-effort-v1.onnx` | ONNX graph with embedded weights. Inputs `input_ids`, `attention_mask` (int64, batch × ≤256); output: 5 effort logits |
+| `ettin-32m-effort-v1.tokenizer.json` | Hugging Face tokenizer |
+| `ettin-32m-effort-v1.json` | Manifest: effort order, input layout, file sizes and SHA-256 |
+
+System1Router downloads these on `system1router install`. A new version gets a new name (`-v2`); old files stay.
+
+## Input
+
+`[CLS] prompt [SEP] context [SEP]`, at most 256 tokens. The prompt keeps up to a third; the context is cut from its oldest end. The context is the last 4,000 characters of:
+
+```
+Files in play: <paths>
+Tools used: <tool names>
+Recent conversation:
+<the newest 3,000 characters of the conversation>
+```
+
+So a short follow-up ("ok, do it") after a long debugging exchange is judged by what it refers to.
+
+## Training
+
+- Base: [jhu-clsp/ettin-encoder-32m](https://huggingface.co/jhu-clsp/ettin-encoder-32m), fully fine-tuned (embeddings included), 4 epochs, lr 8e-5.
+- Data: 2,535 prompts from real Claude Code and OpenCode sessions, each labelled in hindsight by Claude Opus with the effort the turn needed, given the prompt, context and outcome. The data is not published.
+- Loss: class-weighted cross-entropy plus an expected-distance cost that weighs a too-low effort twice a too-high one.
+
+## Evaluation
+
+509 prompts from 96 held-out sessions (split by session):
+
+| | Exact | Within one level | Mean abs. error |
+|---|---:|---:|---:|
+| random | 23–28% | 65–74% | 1.04–1.21 |
+| always `medium` | 33% | 91% | 0.76 |
+| **this model** | **60%** | **94%** | **0.47** |
+
+Too low 25%, too high 15%. Labels are one judge's opinion, so part of the remaining error is label noise. About 10 ms per prompt on 4 CPU threads after a 0.5 s load; 130 MB.
+
+## Limits
+
+- Trained on one developer's sessions: mostly Rust, TypeScript and Python projects.
+- The token embeddings were fine-tuned on those sessions, so they carry traces of the vocabulary that appeared in them.
+- An effort predictor, not a quality judge: it does not know which model runs the turn.
+
+Retrain on your own sessions: [experiments/TRAINING.md](https://github.com/Valkea/System1Router/blob/main/experiments/TRAINING.md).

ettin-32m-effort-v1.json

@@ -0,0 +1,27 @@
+{
+ "name": "ettin-32m-effort-v1",
+ "base": null,
+ "efforts": [
+  "low",
+  "medium",
+  "high",
+  "xhigh",
+  "max"
+ ],
+ "maxLength": 256,
+ "contextChars": 4000,
+ "cls": 50281,
+ "sep": 50282,
+ "files": {
+  "onnx": {
+   "name": "ettin-32m-effort-v1.onnx",
+   "bytes": 129405337,
+   "sha256": "940f313b8378af1484be58c3d9c75ed9708662ee76abfe121298615bd9f05be9"
+  },
+  "tokenizer.json": {
+   "name": "ettin-32m-effort-v1.tokenizer.json",
+   "bytes": 3583228,
+   "sha256": "6c8aaa9a542084f2457eab775d4eeb51f92a70c0fd9de28d5edb0ddec3c08d30"
+  }
+ }
+}

ettin-32m-effort-v1.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:940f313b8378af1484be58c3d9c75ed9708662ee76abfe121298615bd9f05be9
+size 129405337