| # RESULTS (Windows + ROCm PC, measured 2026-10-22 JST) |
|
|
| Hardware/software: Ryzen AI 9 HX 370 / Radeon 890M (gfx1150), Windows 11, torch 2.14.0+rocm10.1.0, |
| transformers 5.20.2, peft 0.21.2. Model Qwen/Qwen3.5-2B, bf16, batch 1, LoRA r=32 (LM only) + head (22.1M). |
| Trainable params 53.3M. flash-linear-attention / causal_conv1d NOT installed (reference PyTorch path). |
| Synthetic data only (DUMMY teacher) -> accuracy numbers are not meaningful. |
| |
| ## Training run (`runs/train-syn-2b-390.log`) |
| 200 optimizer steps, max_pixels=430080 (390 image tokens, ~860 total tokens), AOTriton OFF. |
|
|
| | step | loss | kd | ce | |
| | ---: | ---: | ---: | ---: | |
| | 10 | 0.997 | 0.816 | 1.207 | |
| | 50 | 0.353 | 0.325 | 0.379 | |
| | 100 | 0.310 | 0.290 | 0.329 | |
| | 150 | 0.311 | 0.302 | 0.310 | |
| | 200 | 0.224 | 0.187 | 0.292 | |
|
|
| 4.5-4.6 s/step, peak allocated 10.49 GB. Checkpoint: adapter 125 MB + head 44 MB. |
|
|
| Eval on 20 synthetic records (60 questions, `runs/eval-syn-2b-390.json`): accuracy 1.0, NLL 0.202, |
| Brier 0.052, ECE(15) 0.182 (under-confident: it imitates the dummy teacher's soft targets), teacher agreement 0.917. |
|
|
| ## bench.py (`runs/bench-default.json`, `runs/bench-aotriton.json`) |
| iPhone portrait 1179x2556 synthetic screenshots, 3 questions / 15 options. Train = fwd+bwd+AdamW step, median of 8. |
| Inference = preprocess + forward, LoRA NOT merged, median of 8. |
|
|
| | setting | image tokens | seq tokens | train s/step | train peak GB | infer ms | infer peak GB | |
| | --- | ---: | ---: | ---: | ---: | ---: | ---: | |
| | default, 430080 px | 390 | 861 | 4.64 | 10.48 | 2359 | 5.07 | |
| | default, 1048576 px | 987 | 1458 | 16.37 | 14.65 | 8731 | 6.95 | |
| | AOTriton=1, 430080 px | 390 | 861 | 4.02 | 10.28 | 1653 | 4.79 | |
| | AOTriton=1, 1048576 px | 987 | 1458 | 8.45 | 14.16 | 3199 | 4.90 | |
|
|
| infer.py with LoRA merged (AOTriton OFF, 390 tokens): forward 1856-1866 ms, total ~1885 ms. |
|
|
| => `TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1` is now set by default (os.environ.setdefault) in |
| train.py / infer.py / bench.py / scripts/evaluate.py. Set it to 0 to disable. |
| ## Local Clef-Flash teacher (measured 2026-10-23 JST) |
|
|
| Teacher: `Cloudflare/clef-flash` (Apache-2.0; Qwen3.5-9B backbone + JointSchemaHead width 1024 / 4 layers), |
| revision `f0b7d3cd65296abfe459e6c228b618f5d8b26d2f`, 19.08 GB safetensors (4 shards + 0.24 GB head), downloaded in 246 s. |
| Loaded in bf16 entirely on the 890M (`load_release_model`, ~23 s, 17.75 GB allocated). No offload needed. |
| `scripts/label_local_clef.py`, AOTriton=1, batch 1, 3 questions / 15 options, reference (slow) Gated-DeltaNet path. |
|
|
| | max_pixels | image tokens | seq tokens | first item s | median s/image | peak alloc GB | peak reserved GB | |
| | ---: | ---: | ---: | ---: | ---: | ---: | ---: | |
| | 430080 (= student) | 390 | ~860 | 27-29 (warm-up) | 3.10 | 18.02 | 18.14 | |
| | 1048576 | 987 | ~1455 | 68 (warm-up) | 5.53 | 18.20 | 18.51 | |
| |
| Release default is `longest_edge=16777216` (native res, ~2.9k tokens for 1179x2556); not benchmarked. |
| Logs/stats: `runs/clef-label-{train,eval}-390.{log,json}`, `runs/clef-label-eval-987.{log,json}`. |
|
|
| Teacher vs synthetic gold (`scripts/teacher_report.py`, `runs/teacher-report-clef-390.json`): |
|
|
| | setting | records | questions | accuracy | record exact | NLL | mean conf | ECE(15) | |
| | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | |
| | 390 tok, train+eval | 80 | 240 | 1.000 | 1.000 | 0.035 | 0.966 | 0.034 | |
| | 987 tok, eval | 20 | 60 | 1.000 | 1.000 | 0.036 | 0.965 | 0.035 | |
|
|
| Example (syn-train-0000, gold error_dialog / true / retry): screen error_dialog 0.976, has_error true 0.967, |
| primary_button retry 0.846 (none 0.060, ok 0.036, delete 0.029, login 0.029). `primary_button` is the least confident field. |
|
|
| ## Student distilled from Clef-Flash labels (`runs/train-clef-2b-390.log`) |
|
|
| Qwen3.5-2B + LoRA r=32 + head, `train.py --data data/synthetic/train.clef.jsonl --steps 200 --max-pixels 430080` |
| (alpha 0.7, T 2.0, beta 0.1, AOTriton=1). 727 s total, ~3.5 s/step, peak 10.29 GB. Checkpoint `checkpoints/clef-2b-390`. |
| Loss 2.05 (step 10) -> 0.42 (50) -> 0.27 (100) -> 0.082 (200); KD 0.031 at step 200. |
|
|
| Eval on `data/synthetic/eval.clef.jsonl` (20 records / 60 questions, `scripts/evaluate.py`; teacher agreement = argmax vs Clef-Flash): |
|
|
| | student | accuracy | NLL | Brier | ECE(15) | teacher agreement | |
| | --- | ---: | ---: | ---: | ---: | ---: | |
| | clef-2b-390 (Clef labels) | 1.000 | 0.032 | 0.0025 | 0.031 | 1.000 | |
| | syn-2b-390 (dummy labels) | 1.000 | 0.202 | 0.052 | 0.182 | 1.000 | |
| | Clef-Flash teacher itself | 1.000 | 0.036 | - | 0.034 | - | |
|
|
| => The real teacher's sharper, well-calibrated targets fix the under-confidence of the dummy-label student |
| (ECE 0.18 -> 0.03). Student inference is ~1.65 s/image vs teacher 3.10 s/image at 390 tokens (~1.9x faster, 18 GB -> ~5 GB). |
| Caveat: 60 train / 20 eval synthetic images, easy task -> saturated accuracy; needs real iOS screenshots. |
|
|
