kobkrit commited on
Commit
82b8a23
Β·
verified Β·
1 Parent(s): eaaae4f

model card: order-invariant inference mode (fix for option-order bias)

Browse files
Files changed (1) hide show
  1. README.md +6 -4
README.md CHANGED
@@ -190,10 +190,12 @@ Thai sets and on NLI/topic tasks, and route low-confidence English yes/no decisi
190
  - Public 13-subset macro 61.9 β†’ **63.2**; Wisesight 38.7 β†’ **51.5**; MASSIVE-en 75.7 β†’ 79.1; MASSIVE-th 86.4 β†’ 88.6;
191
  MultiNLI 85.6 β†’ 87.3 (ECE 0.035 β†’ 0.016); Aegis2 58.0 β†’ 61.6; PubMedQA 53.6 β†’ 56.4.
192
  - Regression: SIB-200 Thai 77.5 β†’ 74.0 (sentiment set was weighted 4Γ—; will be lowered next round). Unchanged: SQuAD2, SummEval-relevance.
193
- - Known issue (measured on v0.2): answers can depend on the *order* of the options, mildly for ≀ 10 options
194
- (5–12% of arg-max choices flip under a random permutation, mean probability shift 0.05–0.14) and strongly for very
195
- large option sets (77-way banking77: 72% flip). A permutation-averaging inference mode is in progress; until then, for
196
- large option sets query with several option orders and average the probabilities.
 
 
197
  - Live API switched to v0.2 (same checkpoint as these weights).
198
 
199
  **v0.1 β€” 2026-09-20**
 
190
  - Public 13-subset macro 61.9 β†’ **63.2**; Wisesight 38.7 β†’ **51.5**; MASSIVE-en 75.7 β†’ 79.1; MASSIVE-th 86.4 β†’ 88.6;
191
  MultiNLI 85.6 β†’ 87.3 (ECE 0.035 β†’ 0.016); Aegis2 58.0 β†’ 61.6; PubMedQA 53.6 β†’ 56.4.
192
  - Regression: SIB-200 Thai 77.5 β†’ 74.0 (sentiment set was weighted 4Γ—; will be lowered next round). Unchanged: SQuAD2, SummEval-relevance.
193
+ - Known issue and fix: like Jev, a single forward pass is sensitive to the *order* of the options (measured on v0.2:
194
+ 5–12% of arg-max choices flip under a random permutation for ≀ 10 options, 72% for 77-way banking77). The client and
195
+ API now have an **order-invariant mode** that averages the answer over several cyclic option orders in one batched
196
+ pass (`order_invariant: true` / `permutations: n`; automatic for choice questions with > 10 options). With it, on 360
197
+ held-out Thai questions: flips 18.9% β†’ 3.3%, mean probability shift 0.19 β†’ 0.08, accuracy 68.6 β†’ 73.6, banking77
198
+ 41.7 β†’ 63.3. Cost: ~2Γ— latency (batched), never more tokens per request in `usage.input_tokens`.
199
  - Live API switched to v0.2 (same checkpoint as these weights).
200
 
201
  **v0.1 β€” 2026-09-20**