RESULTS (Windows + ROCm PC, measured 2026-10-22 JST)
Hardware/software: Ryzen AI 9 HX 370 / Radeon 890M (gfx1150), Windows 11, torch 2.14.0+rocm10.1.0, transformers 5.20.2, peft 0.21.2. Model Qwen/Qwen3.5-2B, bf16, batch 1, LoRA r=32 (LM only) + head (22.1M). Trainable params 53.3M. flash-linear-attention / causal_conv1d NOT installed (reference PyTorch path). Synthetic data only (DUMMY teacher) -> accuracy numbers are not meaningful.
Training run (runs/train-syn-2b-390.log)
200 optimizer steps, max_pixels=430080 (390 image tokens, ~860 total tokens), AOTriton OFF.
| step | loss | kd | ce |
|---|---|---|---|
| 10 | 0.997 | 0.816 | 1.207 |
| 50 | 0.353 | 0.325 | 0.379 |
| 100 | 0.310 | 0.290 | 0.329 |
| 150 | 0.311 | 0.302 | 0.310 |
| 200 | 0.224 | 0.187 | 0.292 |
4.5-4.6 s/step, peak allocated 10.49 GB. Checkpoint: adapter 125 MB + head 44 MB.
Eval on 20 synthetic records (60 questions, runs/eval-syn-2b-390.json): accuracy 1.0, NLL 0.202,
Brier 0.052, ECE(15) 0.182 (under-confident: it imitates the dummy teacher's soft targets), teacher agreement 0.917.
bench.py (runs/bench-default.json, runs/bench-aotriton.json)
iPhone portrait 1179x2556 synthetic screenshots, 3 questions / 15 options. Train = fwd+bwd+AdamW step, median of 8. Inference = preprocess + forward, LoRA NOT merged, median of 8.
| setting | image tokens | seq tokens | train s/step | train peak GB | infer ms | infer peak GB |
|---|---|---|---|---|---|---|
| default, 430080 px | 390 | 861 | 4.64 | 10.48 | 2359 | 5.07 |
| default, 1048576 px | 987 | 1458 | 16.37 | 14.65 | 8731 | 6.95 |
| AOTriton=1, 430080 px | 390 | 861 | 4.02 | 10.28 | 1653 | 4.79 |
| AOTriton=1, 1048576 px | 987 | 1458 | 8.45 | 14.16 | 3199 | 4.90 |
infer.py with LoRA merged (AOTriton OFF, 390 tokens): forward 1856-1866 ms, total ~1885 ms.
=> TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 is now set by default (os.environ.setdefault) in
train.py / infer.py / bench.py / scripts/evaluate.py. Set it to 0 to disable.
Local Clef-Flash teacher (measured 2026-10-23 JST)
Teacher: Cloudflare/clef-flash (Apache-2.0; Qwen3.5-9B backbone + JointSchemaHead width 1024 / 4 layers),
revision f0b7d3cd65296abfe459e6c228b618f5d8b26d2f, 19.08 GB safetensors (4 shards + 0.24 GB head), downloaded in 246 s.
Loaded in bf16 entirely on the 890M (load_release_model, ~23 s, 17.75 GB allocated). No offload needed.
scripts/label_local_clef.py, AOTriton=1, batch 1, 3 questions / 15 options, reference (slow) Gated-DeltaNet path.
| max_pixels | image tokens | seq tokens | first item s | median s/image | peak alloc GB | peak reserved GB |
|---|---|---|---|---|---|---|
| 430080 (= student) | 390 | ~860 | 27-29 (warm-up) | 3.10 | 18.02 | 18.14 |
| 1048576 | 987 | ~1455 | 68 (warm-up) | 5.53 | 18.20 | 18.51 |
Release default is longest_edge=16777216 (native res, ~2.9k tokens for 1179x2556); not benchmarked.
Logs/stats: runs/clef-label-{train,eval}-390.{log,json}, runs/clef-label-eval-987.{log,json}.
Teacher vs synthetic gold (scripts/teacher_report.py, runs/teacher-report-clef-390.json):
| setting | records | questions | accuracy | record exact | NLL | mean conf | ECE(15) |
|---|---|---|---|---|---|---|---|
| 390 tok, train+eval | 80 | 240 | 1.000 | 1.000 | 0.035 | 0.966 | 0.034 |
| 987 tok, eval | 20 | 60 | 1.000 | 1.000 | 0.036 | 0.965 | 0.035 |
Example (syn-train-0000, gold error_dialog / true / retry): screen error_dialog 0.976, has_error true 0.967,
primary_button retry 0.846 (none 0.060, ok 0.036, delete 0.029, login 0.029). primary_button is the least confident field.
Student distilled from Clef-Flash labels (runs/train-clef-2b-390.log)
Qwen3.5-2B + LoRA r=32 + head, train.py --data data/synthetic/train.clef.jsonl --steps 200 --max-pixels 430080
(alpha 0.7, T 2.0, beta 0.1, AOTriton=1). 727 s total, ~3.5 s/step, peak 10.29 GB. Checkpoint checkpoints/clef-2b-390.
Loss 2.05 (step 10) -> 0.42 (50) -> 0.27 (100) -> 0.082 (200); KD 0.031 at step 200.
Eval on data/synthetic/eval.clef.jsonl (20 records / 60 questions, scripts/evaluate.py; teacher agreement = argmax vs Clef-Flash):
| student | accuracy | NLL | Brier | ECE(15) | teacher agreement |
|---|---|---|---|---|---|
| clef-2b-390 (Clef labels) | 1.000 | 0.032 | 0.0025 | 0.031 | 1.000 |
| syn-2b-390 (dummy labels) | 1.000 | 0.202 | 0.052 | 0.182 | 1.000 |
| Clef-Flash teacher itself | 1.000 | 0.036 | - | 0.034 | - |
=> The real teacher's sharper, well-calibrated targets fix the under-confidence of the dummy-label student
(ECE 0.18 -> 0.03). Student inference is 1.65 s/image vs teacher 3.10 s/image at 390 tokens (1.9x faster, 18 GB -> ~5 GB).
Caveat: 60 train / 20 eval synthetic images, easy task -> saturated accuracy; needs real iOS screenshots.