docs/EVALUATION.md

# Giant model — measured evaluation (v3.1.3, one interleaved pass)

All numbers below are from real runs of the committed binary and corpora
(deterministic; seed 77). Engine: SyFox 3.1.2 substrate + v3.1.3 CLI routing
fix. `energy-norm` ON everywhere (mandatory at corpus scale — v3.1.0).
Selection (calibration, router wording) used CAL only; hidden was scored
once. Per-row records: `datasets/eval/dump_pass1.jsonl`.

## Corpus

16 domains, 76,142 train rows (152,284 lessons per epoch), 278 globally
unique opaque anchors, 3 full per-script calibration files + 1 latin merge.
Recipe (the three failure-mode fixes): empty per-row instructions at learn;
globally-unique anchors; round-robin domain interleave. Criteria: preserved
verbatim for tickets/game/guard/bank77/sms (keys remapped to anchors),
mined top-10 for the 8 new HF domains (train rows only). See
`datasets/MANIFEST.json`.

## Hidden results — latin substrate (model-giant-latin, 43,420 nodes / 1,905,170 lanes)

| domain | n | router acc | class acc | end-to-end | deferred | ms/row |
|---|---:|---:|---:|---:|---:|---:|
| tickets_en | 1,800 | 0.831 | 0.904 | 0.761 | 0 | 7.65 |
| bank77 | 3,080 | 0.931 | 0.023 | 0.022 | 0 | 10.47 |
| sms | 838 | 0.233 | 0.752 | 0.222 | 2 | 6.42 |
| clinc | 4,500 | 0.334 | 0.309 | 0.123 | 0 | 11.02 |
| enron | 1,110 | 0.394 | 0.752 | 0.323 | 0 | 6.79 |
| emotion | 2,000 | 0.155 | 0.249 | 0.073 | 0 | 6.76 |
| hate | 1,500 | 0.381 | 0.653 | 0.277 | 0 | 6.52 |
| agnews | 2,000 | 0.320 | 0.604 | 0.206 | 0 | 7.02 |
| polarity | 2,000 | 0.339 | 0.571 | 0.189 | 0 | 6.52 |
| dbpedia | 2,000 | 0.403 | 0.475 | 0.204 | 0 | 7.55 |
| boolq | 1,500 | 0.097 | 0.531 | 0.059 | 0 | 6.18 |

Weighted over these 22,328 rows: router 0.432 · class 0.454 · e2e 0.196.

## Hidden results — multilingual substrates (post-isolation-fix)

| domain | substrate | n | router | class | e2e |
|---|---|---:|---:|---:|---:|
| tickets_bn | giant-bengali (579 nodes / 12,972 lanes) | 540 | 1.000 | 0.943 | 0.943 |
| tickets_hi | giant-devanagari (546 / 12,353) | 539 | 1.000 | 0.920 | 0.920 |
| tickets_ru | giant-cyrillic (680 / 14,915) | 539 | 1.000 | 0.959 | 0.959 |

Reference dedicated fabrics (v3.0, README): bn 0.969 / hi 0.965 / ru 0.970 —
the giant's per-script fabrics land within 1-4.5 points at a fraction of the
size, with the router trivially perfect inside a single-domain substrate.

## Handcrafted probes (41; 37 scored + 4 OOD)

Router 10/41 = 0.244 · class 11/37 = 0.297. OOD: the pure-gibberish probe
deferred (`zzqvv plork wumfus xanquu blorp`); the other three OOD probes
answered at confidence 0.000-0.031 (near-silence, honest). Full transcript
reproducible via `tools/giant_bench.py --probes`.

## Router-criteria wording sweep (cal only, never hidden)

| variant | router cal acc (n=3,250) |
|---|---:|
| short4 (trained wording) | **0.426** |
| mid8 specific | 0.164 |
| rich12 specific | 0.079 |
| long abstract | 0.208 |

Richer SPECIFIC wording makes the router worse — the direct-term harvest
mechanism of docs/PROBING.md: criteria words that occur in states harvest
energy regardless of router meaning, and more words = more cross-domain
harvest. Abstract register words recover part of the loss but do not beat
the short anchor-led probe. Winner (short4) matches what pass-1 trained.

## Probe-condition dependence (measured)

The router readout depends on what ELSE the decide call asks — every
question's criteria tokens inject into the same settled field:

| condition | router acc |
|---|---:|
| blind (router question alone), hidden n=23,528 | **0.400** |
| blind, cal n=1,500 | 0.399 |
| informed (router + the TRUE domain's class question — the hidden table above) | 0.432 weighted |
| router + ALL 15 class questions, cal | 0.399 |

Asking everything does NOT recover the informed condition: the other
domains' criteria dilute the anchoring (competitor dilution). Production
contract: a caller that only asks for routing gets the honest blind
estimate (0.400); the class numbers apply when the caller names the domain.

## Calibration (per-script, on cal only)

- giant-latin: choice temperature 0.1777; multiclass ECE 0.333 → 0.230
  (choice), noul Platt (a=12.64, b=-1.00) ECE 0.511 → 0.095.
- giant-bengali / -devanagari / -cyrillic: fitted on their own cal splits.

## Honest reading

1. **Class channels work where per-anchor data is thick**: tickets_en 0.904,
   sms 0.752, enron 0.752, hate 0.653, agnews 0.604, polarity 0.571 in a
   single shared fabric alongside 15 other domains.
2. **The single 16-way router is the e2e bottleneck** (0.10-0.40 on the new
   domains; wording sweep shows the limit is the mechanism, not the wording).
   e2e = router x class; when the caller names the domain (API use), use the
   class numbers. Blind routing alone measures 0.400 on hidden (see the
   probe-condition section).
3. **bank77's 77-way decision collapses in the shared fabric** (0.023 class
   vs 0.081 for the dedicated fabric): 77 anchors competing in a 43k-node
   multi-domain fabric is a harder readout than the dedicated fabric —
   hub crowding at scale, consistent with the v3.1.0 anatomy.
4. **boolq stays near majority** (0.531 vs ~0.62 True-majority): boolean
   passage comprehension is beyond bag-of-words SI readout — measured limit,
   not a bug.
5. **One interleaved pass, not three**: pass-1 lanes sit at their decay
   equilibrium because every domain is re-touched throughout the round-robin
   stream (the condition that made 3-epoch consolidation necessary —
   sequential training with lane decay 0.995/lesson — does not arise here).
   The chunked corpus (`data/giant/chunks/`) supports additional passes; the
   v3.1.0 ladder measured epochs 2-3 adding +0.008-0.012 on bank77 — run
   them if margins matter more than latency of training.

## Reproduce

```
make all                                  # build engine
python3 tools/hf_fetch.py                 # download HF datasets (cached)
python3 tools/giant_prepare.py            # build corpus + chunks + probes
for c in 0 1 ... 11:                      # one pass = 12 incremental learns
  ./build/syfox learn --examples data/giant/chunks/chunk_0$c.jsonl \
      --model model-giant --lang auto --epochs 1
./build/syfox calibrate --model model-giant-latin \
    --examples data/giant/cal_latin.jsonl --energy-norm   # (+ 3 scripts)
python3 tools/giant_bench.py --model-slug giant-latin --domains ... --dump out.jsonl
python3 tools/giant_bench.py --model-slug giant-latin --probes
python3 tools/router_sweep.py --model-slug giant-latin
```