# RESULTS (Windows + ROCm PC, measured 2026-10-22 JST)

Hardware/software: Ryzen AI 9 HX 370 / Radeon 890M (gfx1150), Windows 11, torch 2.14.0+rocm10.1.0,
transformers 5.20.2, peft 0.21.2. Model Qwen/Qwen3.5-2B, bf16, batch 1, LoRA r=32 (LM only) + head (22.1M).
Trainable params 53.3M. flash-linear-attention / causal_conv1d NOT installed (reference PyTorch path).
Synthetic data only (DUMMY teacher) -> accuracy numbers are not meaningful.

## Training run (`runs/train-syn-2b-390.log`)
200 optimizer steps, max_pixels=430080 (390 image tokens, ~860 total tokens), AOTriton OFF.

| step | loss | kd | ce |
| ---: | ---: | ---: | ---: |
| 10 | 0.997 | 0.816 | 1.207 |
| 50 | 0.353 | 0.325 | 0.379 |
| 100 | 0.310 | 0.290 | 0.329 |
| 150 | 0.311 | 0.302 | 0.310 |
| 200 | 0.224 | 0.187 | 0.292 |

4.5-4.6 s/step, peak allocated 10.49 GB. Checkpoint: adapter 125 MB + head 44 MB.

Eval on 20 synthetic records (60 questions, `runs/eval-syn-2b-390.json`): accuracy 1.0, NLL 0.202,
Brier 0.052, ECE(15) 0.182 (under-confident: it imitates the dummy teacher's soft targets), teacher agreement 0.917.

## bench.py (`runs/bench-default.json`, `runs/bench-aotriton.json`)
iPhone portrait 1179x2556 synthetic screenshots, 3 questions / 15 options. Train = fwd+bwd+AdamW step, median of 8.
Inference = preprocess + forward, LoRA NOT merged, median of 8.

| setting | image tokens | seq tokens | train s/step | train peak GB | infer ms | infer peak GB |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
| default, 430080 px | 390 | 861 | 4.64 | 10.48 | 2359 | 5.07 |
| default, 1048576 px | 987 | 1458 | 16.37 | 14.65 | 8731 | 6.95 |
| AOTriton=1, 430080 px | 390 | 861 | 4.02 | 10.28 | 1653 | 4.79 |
| AOTriton=1, 1048576 px | 987 | 1458 | 8.45 | 14.16 | 3199 | 4.90 |

infer.py with LoRA merged (AOTriton OFF, 390 tokens): forward 1856-1866 ms, total ~1885 ms.

=> `TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1` is now set by default (os.environ.setdefault) in
train.py / infer.py / bench.py / scripts/evaluate.py. Set it to 0 to disable.
## Local Clef-Flash teacher (measured 2026-10-23 JST)

Teacher: `Cloudflare/clef-flash` (Apache-2.0; Qwen3.5-9B backbone + JointSchemaHead width 1024 / 4 layers),
revision `f0b7d3cd65296abfe459e6c228b618f5d8b26d2f`, 19.08 GB safetensors (4 shards + 0.24 GB head), downloaded in 246 s.
Loaded in bf16 entirely on the 890M (`load_release_model`, ~23 s, 17.75 GB allocated). No offload needed.
`scripts/label_local_clef.py`, AOTriton=1, batch 1, 3 questions / 15 options, reference (slow) Gated-DeltaNet path.

| max_pixels | image tokens | seq tokens | first item s | median s/image | peak alloc GB | peak reserved GB |
| ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| 430080 (= student) | 390 | ~860 | 27-29 (warm-up) | 3.10 | 18.02 | 18.14 |
| 1048576 | 987 | ~1455 | 68 (warm-up) | 5.53 | 18.20 | 18.51 |

Release default is `longest_edge=16777216` (native res, ~2.9k tokens for 1179x2556); not benchmarked.
Logs/stats: `runs/clef-label-{train,eval}-390.{log,json}`, `runs/clef-label-eval-987.{log,json}`.

Teacher vs synthetic gold (`scripts/teacher_report.py`, `runs/teacher-report-clef-390.json`):

| setting | records | questions | accuracy | record exact | NLL | mean conf | ECE(15) |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| 390 tok, train+eval | 80 | 240 | 1.000 | 1.000 | 0.035 | 0.966 | 0.034 |
| 987 tok, eval | 20 | 60 | 1.000 | 1.000 | 0.036 | 0.965 | 0.035 |

Example (syn-train-0000, gold error_dialog / true / retry): screen error_dialog 0.976, has_error true 0.967,
primary_button retry 0.846 (none 0.060, ok 0.036, delete 0.029, login 0.029). `primary_button` is the least confident field.

## Student distilled from Clef-Flash labels (`runs/train-clef-2b-390.log`)

Qwen3.5-2B + LoRA r=32 + head, `train.py --data data/synthetic/train.clef.jsonl --steps 200 --max-pixels 430080`
(alpha 0.7, T 2.0, beta 0.1, AOTriton=1). 727 s total, ~3.5 s/step, peak 10.29 GB. Checkpoint `checkpoints/clef-2b-390`.
Loss 2.05 (step 10) -> 0.42 (50) -> 0.27 (100) -> 0.082 (200); KD 0.031 at step 200.

Eval on `data/synthetic/eval.clef.jsonl` (20 records / 60 questions, `scripts/evaluate.py`; teacher agreement = argmax vs Clef-Flash):

| student | accuracy | NLL | Brier | ECE(15) | teacher agreement |
| --- | ---: | ---: | ---: | ---: | ---: |
| clef-2b-390 (Clef labels) | 1.000 | 0.032 | 0.0025 | 0.031 | 1.000 |
| syn-2b-390 (dummy labels) | 1.000 | 0.202 | 0.052 | 0.182 | 1.000 |
| Clef-Flash teacher itself | 1.000 | 0.036 | - | 0.034 | - |

=> The real teacher's sharper, well-calibrated targets fix the under-confidence of the dummy-label student
(ECE 0.18 -> 0.03). Student inference is ~1.65 s/image vs teacher 3.10 s/image at 390 tokens (~1.9x faster, 18 GB -> ~5 GB).
Caveat: 60 train / 20 eval synthetic images, easy task -> saturated accuracy; needs real iOS screenshots.