munod commited on
Commit
ea9529c
Β·
verified Β·
1 Parent(s): affad59

Upload folder using huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +34 -14
README.md CHANGED
@@ -28,7 +28,26 @@ pipeline_tag: text-classification
28
  > [`munod/tachyone-multi`](https://huggingface.co/munod/tachyone-multi)); measured numbers below come
29
  > from `benchmarks/report.md`.
30
  >
31
- > **This revision is B-18: the English confidence chain was rebuilt and P3's in-domain
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
32
  > acceptance closes.** Diagnosis settled B-15's recorded follow-up: no single temperature
33
  > serves both regimes of the legal P3 basis (training slice acc 0.803 wants T β‰ˆ 1.5–2; the
34
  > hard remainder slice at acc 0.372 wants T β‰₯ 10) *and* the acceptance read (acc 0.9956
@@ -177,21 +196,21 @@ interleaved touch-up (`training/interleave_continue.py`).
177
 
178
  | Checkpoint | Split | Accuracy | ECE (calibrated) | p50 (ms) |
179
  | --- | --- | --- | --- | --- |
180
- | English (ModernBERT-large + five-domain LoRA r=16 + choice-head bank) | support | **1.0000** | 0.017 | 58.5 |
181
- | English, five-domain split | 5 domains | **0.9983** | 0.016 | 58.0 |
182
- | English, **holdout phrasing** (never in training) | 5 domains | **0.9825** | 0.032 | 58.3 |
183
- | Multilingual (mmBERT-base + LoRA r=64 + fitted bank) β€” routed runtime | support | **0.9300** | 0.039 | 53.1 |
184
- | Multilingual, five-domain split | 5 domains | **0.9221** | 0.034 | 53.5 |
185
- | Multilingual, **holdout phrasing** (never in training) | 5 domains | **0.9264** | 0.033 | 53.2 |
186
 
187
  The English ECE column reads the **B-18 fitted confidence** (below): a `peakedness Γ—
188
  strength` cell map for `choice` (fitted slice-balanced over the in-template and holdout
189
  slices) plus the P3 `noul` map, with every temperature byte-unchanged β€” the P3 in-domain
190
  `choice` miss (0.5243 at T=10.0) is history (0.0010, acceptance met). The multilingual rows
191
- are the **routed system**: ~13–15% of rows fall
192
- through to the English checkpoint by language routing (empty state or no Latin-language signal
193
- β€” `.specs/project/BACKLOG.md` **L-013**), which is why the five-domain multilingual row sits
194
- below the multilingual checkpoint's own **0.9961** when it answers everything.
195
 
196
  Per primitive (English, five-domain): `choice` **0.9956**, `noul` **0.9996**, `score`
197
  **0.9996**; (multilingual, five-domain): `choice` **0.9564**, `noul` **0.9136**, `score`
@@ -253,9 +272,10 @@ worst new domain **0.993**, per-domain ECE **0.0005–0.0038** (zero exceptions)
253
  benchmark: multilingual support 0.9760, worst new domain 0.9747, per-domain ECE 0.0019–0.0202
254
  (zero exceptions), strict 1.000, every language β‰₯ baseline; overall 0.9885 / ECE 0.0069 when
255
  the multilingual checkpoint answers everything** β€” the routed rows above show the pair as
256
- served. On the support-only *routed* split (13–15% of
257
- rows fall through to the English checkpoint by language routing, `.specs/project/BACKLOG.md`
258
- **L-013**) the per-language story is unchanged in kind: three of six languages sit above the
 
259
  0.05 ECE target on that split, and on the **holdout** the B-1 per-language ECE criterion was
260
  **NOT met** at B-15 (7 cells, worst `score/it` 0.1495) β€” the **B-16** cycle (serve-key
261
  detector + fitted confidence, 2026-10-09) moved it to **11/12 with `score/it` 0.0521
 
28
  > [`munod/tachyone-multi`](https://huggingface.co/munod/tachyone-multi)); measured numbers below come
29
  > from `benchmarks/report.md`.
30
  >
31
+ > **This revision is B-19: routing reads the same detector as calibration, and the routed
32
+ > pair finally matches the arms.** The docs' *"~13–15% of multilingual rows fall through to
33
+ > the English checkpoint"* was real β€” measured **14.4%** (five-domain) / **12.1%** (support):
34
+ > 405 empty states whose language lives only in the localized question plus 677 Latin states
35
+ > without signal (`nl` losing 35% of its rows). Routing now keys on
36
+ > `detect_calibration_language` over the state **and** the request's question texts (the B-16
37
+ > detector, diagonal 99.80/99.95% β€” the same function that keys temperatures and cells, so
38
+ > fit and serve cannot drift), through the additive `Router.route(state, question_texts=…)`;
39
+ > non-Latin scripts and the no-keyword path are byte-identical to history, and `und`/accented
40
+ > Latin now routes multilingual (accents are a non-English signal). **Misroutes 14.4% β†’
41
+ > 0.05%** (4/7,500; support 0; English 0, clean and noisy alike). Routed multilingual
42
+ > accuracy **0.9300 β†’ 0.9993 / 0.9221 β†’ 0.9957 / 0.9264 β†’ 0.9828** (ECE β†’ 0.006/0.003/
43
+ > 0.006); the six checkpoint-alone reports are byte-identical and English accuracy-identical
44
+ > (quality is gated checkpoint-alone per **L-013**). On the 1,082 rescued rows the English
45
+ > arm scored 0.4871 against the multilingual arm's **1.0000**. Disclosed as measured:
46
+ > MASSIVE 0.058 β†’ **0.051** (rows re-routed between arms), typed-decisions 0.306 and XNLI
47
+ > 0.333 byte-unchanged; four Dutch states with no signal anywhere still reach English (the
48
+ > recorded L-013 default). Record: `BACKLOG.md` **B-19**.
49
+ >
50
+ > **The v0.12.0 revision was B-18 (history):** the English confidence chain was rebuilt and P3's in-domain
51
  > acceptance closes.** Diagnosis settled B-15's recorded follow-up: no single temperature
52
  > serves both regimes of the legal P3 basis (training slice acc 0.803 wants T β‰ˆ 1.5–2; the
53
  > hard remainder slice at acc 0.372 wants T β‰₯ 10) *and* the acceptance read (acc 0.9956
 
196
 
197
  | Checkpoint | Split | Accuracy | ECE (calibrated) | p50 (ms) |
198
  | --- | --- | --- | --- | --- |
199
+ | English (ModernBERT-large + five-domain LoRA r=16 + choice-head bank) | support | **1.0000** | 0.017 | 58.2 |
200
+ | English, five-domain split | 5 domains | **0.9983** | 0.016 | 58.8 |
201
+ | English, **holdout phrasing** (never in training) | 5 domains | **0.9825** | 0.032 | 59.2 |
202
+ | Multilingual (mmBERT-base + LoRA r=64 + fitted bank) β€” routed runtime | support | **0.9993** | 0.006 | 54.5 |
203
+ | Multilingual, five-domain split | 5 domains | **0.9957** | 0.003 | 53.2 |
204
+ | Multilingual, **holdout phrasing** (never in training) | 5 domains | **0.9828** | 0.006 | 53.8 |
205
 
206
  The English ECE column reads the **B-18 fitted confidence** (below): a `peakedness Γ—
207
  strength` cell map for `choice` (fitted slice-balanced over the in-template and holdout
208
  slices) plus the P3 `noul` map, with every temperature byte-unchanged β€” the P3 in-domain
209
  `choice` miss (0.5243 at T=10.0) is history (0.0010, acceptance met). The multilingual rows
210
+ are the **routed system**: since **B-19** the router keys on the calibration detector (state
211
+ + localized question), so only **0.05%** of rows still reach the English checkpoint by
212
+ design (**L-013** β€” empty *and* unsignaled), and the routed rows sit where the arms sit
213
+ (five-domain routed **0.9957** against the multilingual checkpoint's own **0.9961**).
214
 
215
  Per primitive (English, five-domain): `choice` **0.9956**, `noul` **0.9996**, `score`
216
  **0.9996**; (multilingual, five-domain): `choice` **0.9564**, `noul` **0.9136**, `score`
 
272
  benchmark: multilingual support 0.9760, worst new domain 0.9747, per-domain ECE 0.0019–0.0202
273
  (zero exceptions), strict 1.000, every language β‰₯ baseline; overall 0.9885 / ECE 0.0069 when
274
  the multilingual checkpoint answers everything** β€” the routed rows above show the pair as
275
+ served. On the support-only *routed* split (before B-19, 13–15% of
276
+ rows fell through to the English checkpoint by language routing β€” `.specs/project/BACKLOG.md`
277
+ **L-013**; since B-19 that is 0.05%, one detector deciding language everywhere) the
278
+ per-language story is unchanged in kind: three of six languages sit above the
279
  0.05 ECE target on that split, and on the **holdout** the B-1 per-language ECE criterion was
280
  **NOT met** at B-15 (7 cells, worst `score/it` 0.1495) β€” the **B-16** cycle (serve-key
281
  detector + fitted confidence, 2026-10-09) moved it to **11/12 with `score/it` 0.0521