| # mt92 extract — shipped working set (24 layers + outlier split) |
|
|
| `r24` is the checkpoint behind the live artifact: Qwen3-0.6B at lr 8e-5, then the |
| four least useful transformer blocks dropped by angular distance and retrained. |
|
|
| ## The two things that mattered |
|
|
| **All the int8 damage lives in a handful of channels.** down_proj inputs peak |
| near 3800 while weights sit near 0.5, and onnxruntime scales activations per |
| tensor, so one shared scale erases everything small. Pulling the **32** widest |
| channels per layer into their own full-precision matmul drops the remaining peak |
| to about 10 and restores full-precision quality for ~3 MB. `split_onnx.py` does |
| it with Gather/MatMul/Add only, so `validate_graph` stays happy. |
|
|
| **Depth is nearly free on this task and it is what the cost coordinate measures.** |
| Cost is prefill of a fixed 249-token probe. 28 -> 24 layers costs nothing |
| measurable (0.9170 -> 0.9139) and 24 -> 20 costs 1.6 points (0.8984). |
|
|
| | artifact | micro-F1 (400) | probe TTFT | size | |
| |---|---|---|---| |
| | 28L, down_proj in fp32 (previous) | 0.9183 | 1579 ms | 984 MiB | |
| | 28L, outlier split | 0.9170 | 721 ms | 736 MiB | |
| | **24L, outlier split (shipped)** | **0.9139** | **602 ms** | **675 MiB** | |
| |
| Full-precision down_proj was buying the same quality at 2.6x the latency. |
|
|
| ## Dead ends |
|
|
| Qwen3-1.7B: even at 0.975 quality its cost puts it in a thin frontier slice -- |
| 0.22 share against the real round-1237 field, versus 0.5-0.88 for this artifact. |
| Static QDQ calibration, SmoothQuant migration, weight averaging, and a QAT phase |
| on a properly tuned base all measured worse than doing nothing. |
|
|
| Rebuild: `./build_final.sh <model_dir> 32`. |
|
|
