File size: 1,656 Bytes
75a486a 771809c 75a486a 771809c 75a486a 771809c 75a486a 08e43d9 75a486a 08e43d9 75a486a 08e43d9 75a486a | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 | # mt92 extract — shipped working set (24 layers + outlier split)
`r24` is the checkpoint behind the live artifact: Qwen3-0.6B at lr 8e-5, then the
four least useful transformer blocks dropped by angular distance and retrained.
## The two things that mattered
**All the int8 damage lives in a handful of channels.** down_proj inputs peak
near 3800 while weights sit near 0.5, and onnxruntime scales activations per
tensor, so one shared scale erases everything small. Pulling the **32** widest
channels per layer into their own full-precision matmul drops the remaining peak
to about 10 and restores full-precision quality for ~3 MB. `split_onnx.py` does
it with Gather/MatMul/Add only, so `validate_graph` stays happy.
**Depth is nearly free on this task and it is what the cost coordinate measures.**
Cost is prefill of a fixed 249-token probe. 28 -> 24 layers costs nothing
measurable (0.9170 -> 0.9139) and 24 -> 20 costs 1.6 points (0.8984).
| artifact | micro-F1 (400) | probe TTFT | size |
|---|---|---|---|
| 28L, down_proj in fp32 (previous) | 0.9183 | 1579 ms | 984 MiB |
| 28L, outlier split | 0.9170 | 721 ms | 736 MiB |
| **24L, outlier split (shipped)** | **0.9139** | **602 ms** | **675 MiB** |
Full-precision down_proj was buying the same quality at 2.6x the latency.
## Dead ends
Qwen3-1.7B: even at 0.975 quality its cost puts it in a thin frontier slice --
0.22 share against the real round-1237 field, versus 0.5-0.88 for this artifact.
Static QDQ calibration, SmoothQuant migration, weight averaging, and a QAT phase
on a properly tuned base all measured worse than doing nothing.
Rebuild: `./build_final.sh <model_dir> 32`.
|