Utiric commited on
Commit
01cd1d1
·
verified ·
1 Parent(s): a1ede48

Docs: fix SVG overflow, add requirements/latency/usage-guide/FAQ, correct journey table (DPO-4)

Browse files
Files changed (3) hide show
  1. README.md +103 -2
  2. arbiter-banner.svg +1 -1
  3. arbiter-how.svg +6 -6
README.md CHANGED
@@ -57,11 +57,43 @@ out = arb.decide(
57
  # "tone": {"level": "2", "score": 2.1, ...}}
58
  ```
59
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
60
  > **No pip package (yet).** Copy `arbiter.py` from this repo (~200 lines, only
61
  > `torch` + `transformers`) next to your code. No training stack, no server, no API
62
  > key needed. The file mirrors the exact training prompt format — do not rephrase
63
  > the prompt template inside it.
64
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
65
  ---
66
 
67
  ## How it works
@@ -82,6 +114,41 @@ out = arb.decide(
82
  Latency scales with prompt length only (prefill cost) — there is no per-token
83
  generation loop.
84
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
85
  ## API reference
86
 
87
  `Arbiter(path, device=None, temperature=None, max_state_tokens=2048)`
@@ -169,8 +236,9 @@ anymore.
169
  | v3 + moderation data | 0.787 | moderation slice relabeled |
170
  | v4 cherry-pick | 0.811 | kept gains, restored casualties |
171
  | DPO-1 | 0.822 | 16k preference pairs |
172
- | v5–v7 + new domains | 0.818 | legal/medical/multilingual siblings |
173
- | **DPO-3 (this release)** | **0.844** | same pairs recipe on v7 |
 
174
 
175
  Recipe: full fine-tune of the 0.6B backbone, CE + 0.3·Brier loss, 1 epoch,
176
  held-out-Brier early stopping, then Decision-DPO (β=1.0 + 0.1 CE anchor, LR 1e-5).
@@ -188,12 +256,45 @@ held-out-Brier early stopping, then Decision-DPO (β=1.0 + 0.1 CE anchor, LR 1e-
188
  ## Limitations (please read before deploying)
189
 
190
  - A classifier head — not a reasoner, generator, or judge. No free text, ever.
 
 
191
  - Medical and legal outputs are triage signals for pipelines **with humans in the loop**.
192
  - Moderation output is a tier-1 signal; escalate to a stronger judge and a human
193
  on low confidence.
194
  - Turkish + English only. Customer-action and irony are the weakest slices.
195
  - Not hardened against adversarial prompt injection.
196
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
197
  ## Files
198
 
199
  | File | What |
 
57
  # "tone": {"level": "2", "score": 2.1, ...}}
58
  ```
59
 
60
+ Turkish works the same way:
61
+
62
+ ```python
63
+ out = arb.decide(
64
+ state="Kargo iki günde geldi, ürün fotoğraftaki gibi. Teşekkürler!",
65
+ questions=[{"id": "sentiment", "type": "choice",
66
+ "instructions": "What is the overall sentiment polarity expressed in this text?",
67
+ "options": {"Positive": "satisfaction, praise, thanks",
68
+ "Neutral": "factual objective information",
69
+ "Negative": "complaint, anger, disappointment"}}],
70
+ )
71
+ # {"sentiment": {"choice": "Positive", "probabilities": {...}}}
72
+ ```
73
+
74
  > **No pip package (yet).** Copy `arbiter.py` from this repo (~200 lines, only
75
  > `torch` + `transformers`) next to your code. No training stack, no server, no API
76
  > key needed. The file mirrors the exact training prompt format — do not rephrase
77
  > the prompt template inside it.
78
 
79
+ ## Requirements
80
+
81
+ - Python 3.10+, `torch`, `transformers`. That is the whole dependency list.
82
+ - Weights are 1.2 GB (bf16). Comfortable in ~2 GB of VRAM; CPU works too
83
+ (float32, slower).
84
+ - No server, no API key, no Inference Providers deployment: you run `arbiter.py`
85
+ wherever your code already runs.
86
+
87
+ Measured latency (T4 GPU, September 2026, single questions, temperature 1.6):
88
+
89
+ | Situation | Latency |
90
+ |---|---|
91
+ | first call (warmup) | ~1 s |
92
+ | steady state, per question | 60–90 ms |
93
+
94
+ Latency grows with prompt length (prefill cost) and is flat in the number of
95
+ questions per pass: asking 6 questions at once costs barely more than asking 1.
96
+
97
  ---
98
 
99
  ## How it works
 
114
  Latency scales with prompt length only (prefill cost) — there is no per-token
115
  generation loop.
116
 
117
+ ## Getting good answers (read this before prompting)
118
+
119
+ ARBITER is prompt-format sensitive: it answers from the exact prompt shape it
120
+ trained on. Three habits decide whether you get 0.88 or 0.37:
121
+
122
+ 1. **Frame the state like training data.** A short domain prefix plus the raw
123
+ text works far better than a bare sentence, e.g.
124
+ `[SUPPORT_TIER: return_desk] | Ticket #43998 Customer purchased a wireless
125
+ headphones 15 days ago. Reason: 'item defective on arrival'...`
126
+ We measured this directly on these weights: 8 hand-written free-form probes
127
+ scored 3/8, while 96 verbatim held-out rows in training format scored 85/96.
128
+ 2. **Describe every option.** `{"Refund": "wants money back"}` beats
129
+ `["Refund", "Whereabouts", ...]`. The description is the signal; the name is
130
+ just a key.
131
+ 3. **Gate on confidence.** `confidence` is the top-1 minus top-2 probability gap:
132
+
133
+ ```python
134
+ a = out["intent"]
135
+ if a["confidence"] < 0.2:
136
+ escalate(state) # human review or a stronger judge
137
+ else:
138
+ route(a["choice"])
139
+ ```
140
+
141
+ ## When to use it (and when not)
142
+
143
+ Good fit: support-ticket triage and routing, Discord/forum moderation
144
+ pre-filtering, dialogue turn-taking, code-review triage, sentiment buckets —
145
+ any pipeline step shaped "text in, one of N labels out" in Turkish or English.
146
+
147
+ Not a fit: arithmetic or reasoning (`2 + 2` with options `[3, 4, 5, 22]`
148
+ picks `22` — it matches strings, it does not compute), free-text generation,
149
+ standalone medical or legal decisions, more than 52 options in one question,
150
+ languages beyond TR/EN.
151
+
152
  ## API reference
153
 
154
  `Arbiter(path, device=None, temperature=None, max_state_tokens=2048)`
 
236
  | v3 + moderation data | 0.787 | moderation slice relabeled |
237
  | v4 cherry-pick | 0.811 | kept gains, restored casualties |
238
  | DPO-1 | 0.822 | 16k preference pairs |
239
+ | v5–v7 + new domains | 0.818 | legal/medical/multilingual siblings, irony-train contradiction found and dropped |
240
+ | v8 SFT | 0.840 | customer siblings (+19 pts on customer); moderation dipped 0.750 → 0.626 |
241
+ | **DPO-4 (this release)** | **0.844** | preference pairs recovered moderation to 0.704, kept the rest |
242
 
243
  Recipe: full fine-tune of the 0.6B backbone, CE + 0.3·Brier loss, 1 epoch,
244
  held-out-Brier early stopping, then Decision-DPO (β=1.0 + 0.1 CE anchor, LR 1e-5).
 
256
  ## Limitations (please read before deploying)
257
 
258
  - A classifier head — not a reasoner, generator, or judge. No free text, ever.
259
+ - No arithmetic or multi-step reasoning. `2 + 2` with options `[3, 4, 5, 22]`
260
+ answers `22`: it matches surface strings, it does not compute.
261
  - Medical and legal outputs are triage signals for pipelines **with humans in the loop**.
262
  - Moderation output is a tier-1 signal; escalate to a stronger judge and a human
263
  on low confidence.
264
  - Turkish + English only. Customer-action and irony are the weakest slices.
265
  - Not hardened against adversarial prompt injection.
266
 
267
+ ## FAQ
268
+
269
+ **Why is temperature 1.6, above 1.0?**
270
+ Decision-DPO sharpens the distributions (expected side effect). 1.6 is a single
271
+ global value fit post-hoc on held-out data to re-calibrate them — not per-task,
272
+ not per-slice. If you fine-tune or DPO further, refit it.
273
+
274
+ **What does `confidence` mean?**
275
+ Top-1 probability minus top-2. A 0.90 prediction with a 0.08 runner-up
276
+ (gap 0.82) is a sure thing; 0.45 vs 0.44 (gap 0.01) is a coin flip. Threshold it.
277
+
278
+ **More than 52 options?**
279
+ Chunk them: score 40-option windows with a `none of the above` catcher, then
280
+ re-score the finalists in one pass. `arbiter.py` covers the single-pass case
281
+ (≤52); port the chunking from `s1/engine.py` for the rest.
282
+
283
+ **Several questions at once?**
284
+ Yes — pass a list. Every question gets its own answer slot in one forward pass
285
+ (the quickstart answers intent + urgency + tone together).
286
+
287
+ **Can I fine-tune it?**
288
+ It is a stock Qwen3-0.6B-Base fine-tune (full, not LoRA), so any HF trainer
289
+ works. Keep option shuffling on and early-stop on held-out Brier, not train
290
+ loss — 1.5 epochs already overfit this recipe once
291
+ (v1: 0.94 in-task, 0.50 held-out).
292
+
293
+ **Why no Inference Providers / widget?**
294
+ The hosted inference API generates text; this model never generates — it is read
295
+ at a logit slot. A widget would return nonsense, so the model is deliberately
296
+ left undeployed there. Run `arbiter.py` locally instead.
297
+
298
  ## Files
299
 
300
  | File | What |
arbiter-banner.svg CHANGED
arbiter-how.svg CHANGED