Docs: fix SVG overflow, add requirements/latency/usage-guide/FAQ, correct journey table (DPO-4)
Browse files- README.md +103 -2
- arbiter-banner.svg +1 -1
- arbiter-how.svg +6 -6
README.md
CHANGED
|
@@ -57,11 +57,43 @@ out = arb.decide(
|
|
| 57 |
# "tone": {"level": "2", "score": 2.1, ...}}
|
| 58 |
```
|
| 59 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 60 |
> **No pip package (yet).** Copy `arbiter.py` from this repo (~200 lines, only
|
| 61 |
> `torch` + `transformers`) next to your code. No training stack, no server, no API
|
| 62 |
> key needed. The file mirrors the exact training prompt format — do not rephrase
|
| 63 |
> the prompt template inside it.
|
| 64 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 65 |
---
|
| 66 |
|
| 67 |
## How it works
|
|
@@ -82,6 +114,41 @@ out = arb.decide(
|
|
| 82 |
Latency scales with prompt length only (prefill cost) — there is no per-token
|
| 83 |
generation loop.
|
| 84 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 85 |
## API reference
|
| 86 |
|
| 87 |
`Arbiter(path, device=None, temperature=None, max_state_tokens=2048)`
|
|
@@ -169,8 +236,9 @@ anymore.
|
|
| 169 |
| v3 + moderation data | 0.787 | moderation slice relabeled |
|
| 170 |
| v4 cherry-pick | 0.811 | kept gains, restored casualties |
|
| 171 |
| DPO-1 | 0.822 | 16k preference pairs |
|
| 172 |
-
| v5–v7 + new domains | 0.818 | legal/medical/multilingual siblings |
|
| 173 |
-
|
|
|
|
|
| 174 |
|
| 175 |
Recipe: full fine-tune of the 0.6B backbone, CE + 0.3·Brier loss, 1 epoch,
|
| 176 |
held-out-Brier early stopping, then Decision-DPO (β=1.0 + 0.1 CE anchor, LR 1e-5).
|
|
@@ -188,12 +256,45 @@ held-out-Brier early stopping, then Decision-DPO (β=1.0 + 0.1 CE anchor, LR 1e-
|
|
| 188 |
## Limitations (please read before deploying)
|
| 189 |
|
| 190 |
- A classifier head — not a reasoner, generator, or judge. No free text, ever.
|
|
|
|
|
|
|
| 191 |
- Medical and legal outputs are triage signals for pipelines **with humans in the loop**.
|
| 192 |
- Moderation output is a tier-1 signal; escalate to a stronger judge and a human
|
| 193 |
on low confidence.
|
| 194 |
- Turkish + English only. Customer-action and irony are the weakest slices.
|
| 195 |
- Not hardened against adversarial prompt injection.
|
| 196 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 197 |
## Files
|
| 198 |
|
| 199 |
| File | What |
|
|
|
|
| 57 |
# "tone": {"level": "2", "score": 2.1, ...}}
|
| 58 |
```
|
| 59 |
|
| 60 |
+
Turkish works the same way:
|
| 61 |
+
|
| 62 |
+
```python
|
| 63 |
+
out = arb.decide(
|
| 64 |
+
state="Kargo iki günde geldi, ürün fotoğraftaki gibi. Teşekkürler!",
|
| 65 |
+
questions=[{"id": "sentiment", "type": "choice",
|
| 66 |
+
"instructions": "What is the overall sentiment polarity expressed in this text?",
|
| 67 |
+
"options": {"Positive": "satisfaction, praise, thanks",
|
| 68 |
+
"Neutral": "factual objective information",
|
| 69 |
+
"Negative": "complaint, anger, disappointment"}}],
|
| 70 |
+
)
|
| 71 |
+
# {"sentiment": {"choice": "Positive", "probabilities": {...}}}
|
| 72 |
+
```
|
| 73 |
+
|
| 74 |
> **No pip package (yet).** Copy `arbiter.py` from this repo (~200 lines, only
|
| 75 |
> `torch` + `transformers`) next to your code. No training stack, no server, no API
|
| 76 |
> key needed. The file mirrors the exact training prompt format — do not rephrase
|
| 77 |
> the prompt template inside it.
|
| 78 |
|
| 79 |
+
## Requirements
|
| 80 |
+
|
| 81 |
+
- Python 3.10+, `torch`, `transformers`. That is the whole dependency list.
|
| 82 |
+
- Weights are 1.2 GB (bf16). Comfortable in ~2 GB of VRAM; CPU works too
|
| 83 |
+
(float32, slower).
|
| 84 |
+
- No server, no API key, no Inference Providers deployment: you run `arbiter.py`
|
| 85 |
+
wherever your code already runs.
|
| 86 |
+
|
| 87 |
+
Measured latency (T4 GPU, September 2026, single questions, temperature 1.6):
|
| 88 |
+
|
| 89 |
+
| Situation | Latency |
|
| 90 |
+
|---|---|
|
| 91 |
+
| first call (warmup) | ~1 s |
|
| 92 |
+
| steady state, per question | 60–90 ms |
|
| 93 |
+
|
| 94 |
+
Latency grows with prompt length (prefill cost) and is flat in the number of
|
| 95 |
+
questions per pass: asking 6 questions at once costs barely more than asking 1.
|
| 96 |
+
|
| 97 |
---
|
| 98 |
|
| 99 |
## How it works
|
|
|
|
| 114 |
Latency scales with prompt length only (prefill cost) — there is no per-token
|
| 115 |
generation loop.
|
| 116 |
|
| 117 |
+
## Getting good answers (read this before prompting)
|
| 118 |
+
|
| 119 |
+
ARBITER is prompt-format sensitive: it answers from the exact prompt shape it
|
| 120 |
+
trained on. Three habits decide whether you get 0.88 or 0.37:
|
| 121 |
+
|
| 122 |
+
1. **Frame the state like training data.** A short domain prefix plus the raw
|
| 123 |
+
text works far better than a bare sentence, e.g.
|
| 124 |
+
`[SUPPORT_TIER: return_desk] | Ticket #43998 Customer purchased a wireless
|
| 125 |
+
headphones 15 days ago. Reason: 'item defective on arrival'...`
|
| 126 |
+
We measured this directly on these weights: 8 hand-written free-form probes
|
| 127 |
+
scored 3/8, while 96 verbatim held-out rows in training format scored 85/96.
|
| 128 |
+
2. **Describe every option.** `{"Refund": "wants money back"}` beats
|
| 129 |
+
`["Refund", "Whereabouts", ...]`. The description is the signal; the name is
|
| 130 |
+
just a key.
|
| 131 |
+
3. **Gate on confidence.** `confidence` is the top-1 minus top-2 probability gap:
|
| 132 |
+
|
| 133 |
+
```python
|
| 134 |
+
a = out["intent"]
|
| 135 |
+
if a["confidence"] < 0.2:
|
| 136 |
+
escalate(state) # human review or a stronger judge
|
| 137 |
+
else:
|
| 138 |
+
route(a["choice"])
|
| 139 |
+
```
|
| 140 |
+
|
| 141 |
+
## When to use it (and when not)
|
| 142 |
+
|
| 143 |
+
Good fit: support-ticket triage and routing, Discord/forum moderation
|
| 144 |
+
pre-filtering, dialogue turn-taking, code-review triage, sentiment buckets —
|
| 145 |
+
any pipeline step shaped "text in, one of N labels out" in Turkish or English.
|
| 146 |
+
|
| 147 |
+
Not a fit: arithmetic or reasoning (`2 + 2` with options `[3, 4, 5, 22]`
|
| 148 |
+
picks `22` — it matches strings, it does not compute), free-text generation,
|
| 149 |
+
standalone medical or legal decisions, more than 52 options in one question,
|
| 150 |
+
languages beyond TR/EN.
|
| 151 |
+
|
| 152 |
## API reference
|
| 153 |
|
| 154 |
`Arbiter(path, device=None, temperature=None, max_state_tokens=2048)`
|
|
|
|
| 236 |
| v3 + moderation data | 0.787 | moderation slice relabeled |
|
| 237 |
| v4 cherry-pick | 0.811 | kept gains, restored casualties |
|
| 238 |
| DPO-1 | 0.822 | 16k preference pairs |
|
| 239 |
+
| v5–v7 + new domains | 0.818 | legal/medical/multilingual siblings, irony-train contradiction found and dropped |
|
| 240 |
+
| v8 SFT | 0.840 | customer siblings (+19 pts on customer); moderation dipped 0.750 → 0.626 |
|
| 241 |
+
| **DPO-4 (this release)** | **0.844** | preference pairs recovered moderation to 0.704, kept the rest |
|
| 242 |
|
| 243 |
Recipe: full fine-tune of the 0.6B backbone, CE + 0.3·Brier loss, 1 epoch,
|
| 244 |
held-out-Brier early stopping, then Decision-DPO (β=1.0 + 0.1 CE anchor, LR 1e-5).
|
|
|
|
| 256 |
## Limitations (please read before deploying)
|
| 257 |
|
| 258 |
- A classifier head — not a reasoner, generator, or judge. No free text, ever.
|
| 259 |
+
- No arithmetic or multi-step reasoning. `2 + 2` with options `[3, 4, 5, 22]`
|
| 260 |
+
answers `22`: it matches surface strings, it does not compute.
|
| 261 |
- Medical and legal outputs are triage signals for pipelines **with humans in the loop**.
|
| 262 |
- Moderation output is a tier-1 signal; escalate to a stronger judge and a human
|
| 263 |
on low confidence.
|
| 264 |
- Turkish + English only. Customer-action and irony are the weakest slices.
|
| 265 |
- Not hardened against adversarial prompt injection.
|
| 266 |
|
| 267 |
+
## FAQ
|
| 268 |
+
|
| 269 |
+
**Why is temperature 1.6, above 1.0?**
|
| 270 |
+
Decision-DPO sharpens the distributions (expected side effect). 1.6 is a single
|
| 271 |
+
global value fit post-hoc on held-out data to re-calibrate them — not per-task,
|
| 272 |
+
not per-slice. If you fine-tune or DPO further, refit it.
|
| 273 |
+
|
| 274 |
+
**What does `confidence` mean?**
|
| 275 |
+
Top-1 probability minus top-2. A 0.90 prediction with a 0.08 runner-up
|
| 276 |
+
(gap 0.82) is a sure thing; 0.45 vs 0.44 (gap 0.01) is a coin flip. Threshold it.
|
| 277 |
+
|
| 278 |
+
**More than 52 options?**
|
| 279 |
+
Chunk them: score 40-option windows with a `none of the above` catcher, then
|
| 280 |
+
re-score the finalists in one pass. `arbiter.py` covers the single-pass case
|
| 281 |
+
(≤52); port the chunking from `s1/engine.py` for the rest.
|
| 282 |
+
|
| 283 |
+
**Several questions at once?**
|
| 284 |
+
Yes — pass a list. Every question gets its own answer slot in one forward pass
|
| 285 |
+
(the quickstart answers intent + urgency + tone together).
|
| 286 |
+
|
| 287 |
+
**Can I fine-tune it?**
|
| 288 |
+
It is a stock Qwen3-0.6B-Base fine-tune (full, not LoRA), so any HF trainer
|
| 289 |
+
works. Keep option shuffling on and early-stop on held-out Brier, not train
|
| 290 |
+
loss — 1.5 epochs already overfit this recipe once
|
| 291 |
+
(v1: 0.94 in-task, 0.50 held-out).
|
| 292 |
+
|
| 293 |
+
**Why no Inference Providers / widget?**
|
| 294 |
+
The hosted inference API generates text; this model never generates — it is read
|
| 295 |
+
at a logit slot. A widget would return nonsense, so the model is deliberately
|
| 296 |
+
left undeployed there. Run `arbiter.py` locally instead.
|
| 297 |
+
|
| 298 |
## Files
|
| 299 |
|
| 300 |
| File | What |
|
arbiter-banner.svg
CHANGED
|
|
|
|
arbiter-how.svg
CHANGED
|
|
|
|