| --- |
| license: apache-2.0 |
| base_model: distilroberta-base |
| tags: [system-one, structured-output, calibrated, classification, parallel-decoding] |
| --- |
| # Alpha (System-One-style typed decision model) |
|
|
| Alpha takes unstructured / structured state plus **typed questions with declared options** and returns calibrated probabilities for |
| **all questions in one parallel forward pass**. It cannot generate strings and cannot answer outside the declared options |
| (masked softmax -> schema violations are impossible by construction). Option embeddings are independent of the state, so a schema can be |
| compiled once and cached; cardinality is limited only by memory. |
|
|
| Architecture: `distilroberta-base` encoder (state + question slots with `<mask>` readout tokens) -> 2-layer **Slot Reasoner** (set transformer, squared-ReLU, |
| zero-init residuals) -> biaffine/cosine scorer against option embeddings with per-question learned temperature + post-hoc temperature (T=1.04). |
|
|
| Training: 6000 steps x 32 workflow-style examples (1-4 questions over named fields, mixed from 10 public datasets with paraphrased |
| questions/option wordings), proper-scoring-rule loss (NLL + Brier + ordinal term), EMA weights, functional-memory anchor, discovery controller. |
| Held-out families (never trained on): cola, dbpedia, offensive. |
|
|
| ## Results (test splits; see results.json, alpha_report.png) |
| | family | acc (seen wording) | acc (unseen wording) | ECE | |
| |---|---|---|---| |
| | ag_news | 0.926 | 0.924 | 0.032 | |
| | banking77 | 0.902 | 0.890 | 0.036 | |
| | emotion | 0.922 | 0.894 | 0.030 | |
| | imdb | 0.886 | 0.892 | 0.027 | |
| | mrpc | 0.822 | 0.822 | 0.042 | |
| | news20 | 0.674 | 0.678 | 0.068 | |
| | sms | 0.993 | 0.346 | 0.004 | |
| | snli | 0.772 | 0.538 | 0.046 | |
| | sst2 | 0.912 | 0.910 | 0.034 | |
| | yelp | 0.572 | 0.522 | 0.072 | |
|
|
| Workflow eval (4 questions per query): per-question acc 0.825, all-correct 0.437, joint-confidence ECE 0.101. |
|
|
| Held-out task families (unseen schema wording): zero-shot / after 10 / after 40 gradient steps |
| | family | zero-shot | +10 steps | +40 steps | |
| |---|---|---|---| |
| | cola | 0.374 | 0.658 | 0.686 | |
| | dbpedia | 0.312 | 0.586 | 0.952 | |
| | offensive | 0.528 | 0.632 | 0.694 | |
|
|
| Speed on Tesla T4: p50 11.1 ms, p95 13.5 ms for a 3-question query (local, no network); 354 workflows/s at batch 64. |
| Qwen2.5-0.5B-Instruct (generative, same 120 questions): acc 0.583, p50 latency 82 ms vs Alpha acc 0.875, p50 11.1 ms. |
|
|
| Framework diagnostics: DMR=1.08; FRR by perturbation: sigma=0.5: FRR 4250.0 sigma=1: FRR 4250.0 sigma=2: FRR 4250.0 sigma=4: FRR 4250.0 sigma=8: FRR 14.2* (* = not re-reached within budget); MGR by checkpoint in results.json. |
|
|
| ## Honest limits |
| * This is a small research model trained on public classification datasets. **It is not a replacement for, and has not been compared against, TypeSafe's Jev** |
| (a closed API; its claims were not verified here). Do not read the latency comparison as a Jev comparison. |
| * Narrow task coverage (sentiment/topic/intent/NLI/paraphrase/spam/rating); free-form reasoning is out of scope by design. |
| * Calibration is measured in-distribution; shift will degrade it. |
|
|
| ## Usage |
| ```python |
| from huggingface_hub import snapshot_download; import sys |
| p = snapshot_download("SofiTesfay2010/Alpha"); sys.path.insert(0, p) |
| from alpha_s1 import Alpha |
| m = Alpha.from_pretrained(p, device="cuda").eval() |
| print(m.predict({"ticket": "I was charged twice, refund me now."}, |
| [{"id": "refund", "question": "Does the customer want a refund?", "options": ["no", "yes"]}])) |
| ``` |
|
|
