docs/EVALUATION.md

Giant model — measured evaluation (v3.1.3, one interleaved pass)

All numbers below are from real runs of the committed binary and corpora (deterministic; seed 77). Engine: SyFox 3.1.2 substrate + v3.1.3 CLI routing fix. energy-norm ON everywhere (mandatory at corpus scale — v3.1.0). Selection (calibration, router wording) used CAL only; hidden was scored once. Per-row records: datasets/eval/dump_pass1.jsonl.

Corpus

16 domains, 76,142 train rows (152,284 lessons per epoch), 278 globally unique opaque anchors, 3 full per-script calibration files + 1 latin merge. Recipe (the three failure-mode fixes): empty per-row instructions at learn; globally-unique anchors; round-robin domain interleave. Criteria: preserved verbatim for tickets/game/guard/bank77/sms (keys remapped to anchors), mined top-10 for the 8 new HF domains (train rows only). See datasets/MANIFEST.json.

Hidden results — latin substrate (model-giant-latin, 43,420 nodes / 1,905,170 lanes)

domain	n	router acc	class acc	end-to-end	deferred	ms/row
tickets_en	1,800	0.831	0.904	0.761	0	7.65
bank77	3,080	0.931	0.023	0.022	0	10.47
sms	838	0.233	0.752	0.222	2	6.42
clinc	4,500	0.334	0.309	0.123	0	11.02
enron	1,110	0.394	0.752	0.323	0	6.79
emotion	2,000	0.155	0.249	0.073	0	6.76
hate	1,500	0.381	0.653	0.277	0	6.52
agnews	2,000	0.320	0.604	0.206	0	7.02
polarity	2,000	0.339	0.571	0.189	0	6.52
dbpedia	2,000	0.403	0.475	0.204	0	7.55
boolq	1,500	0.097	0.531	0.059	0	6.18

Weighted over these 22,328 rows: router 0.432 · class 0.454 · e2e 0.196.

Hidden results — multilingual substrates (post-isolation-fix)

domain	substrate	n	router	class	e2e
tickets_bn	giant-bengali (579 nodes / 12,972 lanes)	540	1.000	0.943	0.943
tickets_hi	giant-devanagari (546 / 12,353)	539	1.000	0.920	0.920
tickets_ru	giant-cyrillic (680 / 14,915)	539	1.000	0.959	0.959

Reference dedicated fabrics (v3.0, README): bn 0.969 / hi 0.965 / ru 0.970 — the giant's per-script fabrics land within 1-4.5 points at a fraction of the size, with the router trivially perfect inside a single-domain substrate.

Handcrafted probes (41; 37 scored + 4 OOD)

Router 10/41 = 0.244 · class 11/37 = 0.297. OOD: the pure-gibberish probe deferred (zzqvv plork wumfus xanquu blorp); the other three OOD probes answered at confidence 0.000-0.031 (near-silence, honest). Full transcript reproducible via tools/giant_bench.py --probes.

Router-criteria wording sweep (cal only, never hidden)

variant	router cal acc (n=3,250)
short4 (trained wording)	0.426
mid8 specific	0.164
rich12 specific	0.079
long abstract	0.208

Richer SPECIFIC wording makes the router worse — the direct-term harvest mechanism of docs/PROBING.md: criteria words that occur in states harvest energy regardless of router meaning, and more words = more cross-domain harvest. Abstract register words recover part of the loss but do not beat the short anchor-led probe. Winner (short4) matches what pass-1 trained.

Probe-condition dependence (measured)

The router readout depends on what ELSE the decide call asks — every question's criteria tokens inject into the same settled field:

condition	router acc
blind (router question alone), hidden n=23,528	0.400
blind, cal n=1,500	0.399
informed (router + the TRUE domain's class question — the hidden table above)	0.432 weighted
router + ALL 15 class questions, cal	0.399

Asking everything does NOT recover the informed condition: the other domains' criteria dilute the anchoring (competitor dilution). Production contract: a caller that only asks for routing gets the honest blind estimate (0.400); the class numbers apply when the caller names the domain.

Calibration (per-script, on cal only)

• giant-latin: choice temperature 0.1777; multiclass ECE 0.333 → 0.230 (choice), noul Platt (a=12.64, b=-1.00) ECE 0.511 → 0.095.
• giant-bengali / -devanagari / -cyrillic: fitted on their own cal splits.

Honest reading

1. Class channels work where per-anchor data is thick: tickets_en 0.904, sms 0.752, enron 0.752, hate 0.653, agnews 0.604, polarity 0.571 in a single shared fabric alongside 15 other domains.
2. The single 16-way router is the e2e bottleneck (0.10-0.40 on the new domains; wording sweep shows the limit is the mechanism, not the wording). e2e = router x class; when the caller names the domain (API use), use the class numbers. Blind routing alone measures 0.400 on hidden (see the probe-condition section).
3. bank77's 77-way decision collapses in the shared fabric (0.023 class vs 0.081 for the dedicated fabric): 77 anchors competing in a 43k-node multi-domain fabric is a harder readout than the dedicated fabric — hub crowding at scale, consistent with the v3.1.0 anatomy.
4. boolq stays near majority (0.531 vs ~0.62 True-majority): boolean passage comprehension is beyond bag-of-words SI readout — measured limit, not a bug.
5. One interleaved pass, not three: pass-1 lanes sit at their decay equilibrium because every domain is re-touched throughout the round-robin stream (the condition that made 3-epoch consolidation necessary — sequential training with lane decay 0.995/lesson — does not arise here). The chunked corpus (data/giant/chunks/) supports additional passes; the v3.1.0 ladder measured epochs 2-3 adding +0.008-0.012 on bank77 — run them if margins matter more than latency of training.

Reproduce

make all                                  # build engine
python3 tools/hf_fetch.py                 # download HF datasets (cached)
python3 tools/giant_prepare.py            # build corpus + chunks + probes
for c in 0 1 ... 11:                      # one pass = 12 incremental learns
  ./build/syfox learn --examples data/giant/chunks/chunk_0$c.jsonl \
      --model model-giant --lang auto --epochs 1
./build/syfox calibrate --model model-giant-latin \
    --examples data/giant/cal_latin.jsonl --energy-norm   # (+ 3 scripts)
python3 tools/giant_bench.py --model-slug giant-latin --domains ... --dump out.jsonl
python3 tools/giant_bench.py --model-slug giant-latin --probes
python3 tools/router_sweep.py --model-slug giant-latin