model card: order-invariant inference mode (fix for option-order bias)
Browse files
README.md
CHANGED
|
@@ -190,10 +190,12 @@ Thai sets and on NLI/topic tasks, and route low-confidence English yes/no decisi
|
|
| 190 |
- Public 13-subset macro 61.9 β **63.2**; Wisesight 38.7 β **51.5**; MASSIVE-en 75.7 β 79.1; MASSIVE-th 86.4 β 88.6;
|
| 191 |
MultiNLI 85.6 β 87.3 (ECE 0.035 β 0.016); Aegis2 58.0 β 61.6; PubMedQA 53.6 β 56.4.
|
| 192 |
- Regression: SIB-200 Thai 77.5 β 74.0 (sentiment set was weighted 4Γ; will be lowered next round). Unchanged: SQuAD2, SummEval-relevance.
|
| 193 |
-
- Known issue
|
| 194 |
-
|
| 195 |
-
|
| 196 |
-
|
|
|
|
|
|
|
| 197 |
- Live API switched to v0.2 (same checkpoint as these weights).
|
| 198 |
|
| 199 |
**v0.1 β 2026-09-20**
|
|
|
|
| 190 |
- Public 13-subset macro 61.9 β **63.2**; Wisesight 38.7 β **51.5**; MASSIVE-en 75.7 β 79.1; MASSIVE-th 86.4 β 88.6;
|
| 191 |
MultiNLI 85.6 β 87.3 (ECE 0.035 β 0.016); Aegis2 58.0 β 61.6; PubMedQA 53.6 β 56.4.
|
| 192 |
- Regression: SIB-200 Thai 77.5 β 74.0 (sentiment set was weighted 4Γ; will be lowered next round). Unchanged: SQuAD2, SummEval-relevance.
|
| 193 |
+
- Known issue and fix: like Jev, a single forward pass is sensitive to the *order* of the options (measured on v0.2:
|
| 194 |
+
5β12% of arg-max choices flip under a random permutation for β€ 10 options, 72% for 77-way banking77). The client and
|
| 195 |
+
API now have an **order-invariant mode** that averages the answer over several cyclic option orders in one batched
|
| 196 |
+
pass (`order_invariant: true` / `permutations: n`; automatic for choice questions with > 10 options). With it, on 360
|
| 197 |
+
held-out Thai questions: flips 18.9% β 3.3%, mean probability shift 0.19 β 0.08, accuracy 68.6 β 73.6, banking77
|
| 198 |
+
41.7 β 63.3. Cost: ~2Γ latency (batched), never more tokens per request in `usage.input_tokens`.
|
| 199 |
- Live API switched to v0.2 (same checkpoint as these weights).
|
| 200 |
|
| 201 |
**v0.1 β 2026-09-20**
|