--- title: ICE — quantization by error-propagation class created: 2026-08-21 status: measured — ladder shipped 2026-08-21, one derived rule refuted scope: model-agnostic. Instantiated for Ornith-1.5-35B-A3B; written to be re-derived for any sparse MoE. --- # ICE **Freeze what propagates. Spend everything else on the library.** Every quantizer in the GGUF ecosystem — llama.cpp's presets, Unsloth Dynamic, mudler's APEX, AtomicChat's AD — minimises the same objective for every tensor: > imatrix-weighted MSE of *this tensor's output, for the current token.* That objective is correct for tensors whose error dies with the token. It is the wrong objective for three other kinds of tensor, and nobody separates them. --- ## 1. The four error-propagation classes Classify every tensor by **how far an error travels**, not by how big its activations are. | class | mechanism | failure shape | treatment | |---|---|---|---| | **DISCRETE** | error flips an argmax; a different computation runs | not graded — *categorical* | exact (F32) | | **RECURRENT** | error enters a state decay and compounds multiplicatively along the sequence | grows with context length | exact (F32) | | **CACHED** | error is written once and re-read by later tokens | frozen in, cannot be re-decided | near-exact (F16) | | **INSTANT** | error affects this token's output only | graded, bounded | the budget lives here | ### How to find them in an arbitrary architecture - **DISCRETE** — anything feeding a `top_k`, `argmax`, or hard gate. In an MoE that is the router (`ffn_gate_inp`). Also any hard routing/branching signal. - **RECURRENT** — anything multiplying or parameterising a carried state: SSM/Mamba decay and timestep terms (`ssm_alpha`, `ssm_beta`, `ssm_a`, `ssm_dt`), gated linear-attention decay, any RNN gate. **Not** the state's output projection — that one is INSTANT. - **CACHED** — whatever is written into the KV cache: `attn_k`, `attn_v`. Note `attn_q` is *not* cached (see §5 — this is the trap). - **INSTANT** — everything else. ### The finding that makes this cheap On Ornith-1.5-35B-A3B the entire propagating set is **47 M parameters — 0.14% of the model. Keeping all of it exact costs 0.15 GB.** It is not a trade-off; it is a line item you can simply pay. Expect the same on any sparse MoE: routers and state gates are structurally tiny. --- ## 2. The four measured laws ### Law 1 — the sparsity law: always-on bits are worth `1/sparsity` routed bits For anything that fires every token, compute-share and storage-cost are both proportional to parameter count, so *work per byte* is a constant. Routed experts divide that constant by the routing fraction. Measured on Ornith-1.5 (8 of 256 active): every always-on role scores ~270 units of work per GB-per-bpw; the experts score **8.5. The ratio is 32 — exactly 256/8.** > **A bit spent on the always-on core is worth `E_total / E_active` bits spent on > the expert bank.** Compute this ratio first for any new MoE; it sets the entire > shape of the recipe. ### Law 2 — the convexity law: allocation cleverness is capped Quantization error falls as `4^-b`, which is convex. By Jensen, non-uniform allocation only beats uniform when sensitivity varies enormously, and the gain scales with the **logarithm** of the spread. Measured, by water-filling over the real per-expert routing histogram: | sensitivity spread | optimal gain over uniform | |---|---| | 46× (real expert routing) | **+0.139 bpw** | | 3.7× (depth spread of `ffn_down`) | less | | ~250× | ~1 bpw | > **Corollary: stop tuning ladders.** This is why every published recipe lands > within ~0.3 bpw of every other one. Per-expert allocation, depth grading, and > per-layer bands are all worth less than 0.15 bpw. ICE therefore does **no depth > grading at all** — an explicit refusal, not an oversight. ### Law 3 — the leverage law: flat importance is the expensive case The imatrix stores per-channel importance `h`. Measure its concentration with the participation ratio `PR = (Σh)² / (n·Σh²)`. **Spiky `h` (low PR) is a gift** — the quantizer knows which few channels to spend the block scale on. **Flat `h` (high PR) is the hard case** — every channel matters and there is no leverage. So *flat tensors need more bits.* Measured on Ornith-1.5: | tensor | PR/n | |---|---:| | `attn_k/q/v` | 0.0069 (~14 of 2048 channels) | | `attn_qkv`, `attn_gate`, `ssm_alpha/beta` | 0.0128 | | `ffn_gate_exps`, `ffn_up_exps` | 0.2048 | | **`ffn_down_exps`** | **0.6104 — flattest in the model** | > ~~`down` should sit `log4(PR_down / PR_gateup)` = 0.79 bpw above `gate`/`up`.~~ > **REFUTED BY MEASUREMENT, 2026-08-21.** This looked like a derivation of what convention does by feel, and it agreed with the best published rungs (UD-Q4_K_XL and AtomicChat AD-Q5_K-Q4_K both run ~1.0). A controlled test at identical size says otherwise: | expert allocation | size | mean KLD | |---|---:|---:| | `down` = `gate/up` + 0.79 | 20.82 GB | 0.046449 | | **uniform (gap = 0)** | 20.85 GB | **0.041192** | **Uniform is 11.3% better**, reproduced exactly on a rebuild (0.041192 twice). Rebuilding the whole ladder uniform improved every rung: −7.3% / −11.3% / −3.3% / −3.1% at 19/21/23/25 GB. **What PR actually predicts is unknown.** It measures how concentrated the *input* importance is, which is not the same as how much a tensor's error reaches the output — `ffn_down` writes straight into the residual stream while `gate`/`up` errors are attenuated through the gating nonlinearity first. Position in the circuit is not captured by PR, and on this model it evidently dominates. **Practical rule until someone measures better: allocate the expert stack uniformly.** Do not bump `ffn_down`, despite convention. ### Law 4 — the floor is epistemic, not numeric Fit `KLD ≈ k₀ + c_d·4^(-bpw_dense) + c_e·4^(-bpw_expert)`. Two independent harnesses (ours, AtomicChat's) agreed: | | floor `k₀` | `c_e/c_d` | |---|---:|---:| | ours (wikitext/2048) | 0.0205 | 5.91 | | AtomicChat (eval_neutral/4096) | 0.0219 | 7.22 | `UD-Q6_K` measures 0.0221 against a 0.0205 floor — **it is 8% above the floor. The top of the ladder is saturated.** Above ~5.5 expert bpw you are buying calibration noise, not accuracy. > **You cannot out-bit a wrong prior.** Quantization is lossy compression *guided by > a belief about importance*. Improve the imatrix before adding bits. **Caveat:** this fitted model ranks the extremes well and gets close calls wrong (it misorders AtomicChat's `Q4_K_M` vs `AD-Q4_K-IQ4_XS`). Use it to find the floor and the saturation point, never to choose between two candidate recipes. --- ## 3. The ICE assignment ``` DISCRETE routers, hard gates F32 RECURRENT state decay / timestep terms F32 CACHED attn_k, attn_v F16 ALWAYS-ON output head, token_embd, attention, Q8_0 state output proj, shared expert INSTANT routed experts the dial down = gate = up (UNIFORM - measured, see Law 3) ``` Plus one decision specific to speculative-decoding models: > **A draft head does not need target-model precision.** Its errors are *verified* > by the target, so they cost **acceptance**, not correctness. ICE pins only the > draft block's projections and lets its experts follow the tier. On Ornith-1.5 > that frees ~0.40 GB. Everyone else pins the whole block Q8_0, drops it to Q4_0, > or deletes it — nobody asks what a drafter actually needs. --- ## 4. Adapting ICE to a new MoE — the checklist 1. **Inventory.** Dump the tensor list. Group by role. Compute params, % of file, and % of *active* compute per role (weight routed experts by `k/E`). 2. **Sparsity ratio `E/k`** → Law 1. This is your always-on-to-expert exchange rate. 3. **Classify** every tensor into the four propagation classes (§1). Sum the propagating set — if it is ≪1% of the model, freeze all of it. 4. **PR from the imatrix** per role → Law 3. Informative about *quantizer leverage*, but it did NOT predict the right expert split on Ornith-1.5 — allocate the expert stack uniformly unless a controlled test on your model says otherwise. `analysis/imatrix_anatomy.py` computes PR from the imatrix alone, no weights. 5. **Routing histogram** from the imatrix `counts` → Law 2 → confirm depth/expert grading is not worth the risk. `analysis/routing_skew.py`. 6. **Fixed body** = propagating set + always-on core + draft projections. Everything else is one dial: `size = body + E_params · bpw / 8`. 7. **Bound the ladder.** Fit Law 4 on any measured tiers to find `k₀`; the useful band runs from the cliff (where the expert term explodes) to saturation (where the floor dominates). On Ornith-1.5 that was 17–27 GB — five rungs, not eight. --- ## 5. What was tried and rejected — with numbers Negative results, kept because they are the expensive part to rediscover. - **Per-expert bit allocation by routing frequency.** Measured: load balancing works (209.5 effective experts of 256, gini 0.331), and convexity caps the gain at **+0.139 bpw**. A naive "hot 25% at 6.5, cold at 2.5" split is *worse than uniform* (0.00586 vs 0.00552). **Dead.** - **Demoting `attn_q`** on the argument that Q's error dies with the token while K/V's persists. The arithmetic kills it: each query distributes attention weight summing to 1, so total k-influence over a sequence is O(N) — **the same as q**. Buys 63 MB, costs 0.17 active bpw. Kept only as an A/B flag. - **Active bpw as a ranking metric.** Tested against measured KLD it misorders **4 of our 8 tiers** (Spearman +0.95) and **10 of AtomicChat's 13** (+0.71). It over-credits Q8_0-dense-path designs because error is not linear in bits. **Use it to explain where bits went; rank on measured KLD.** - **Depth grading.** Worth <0.15 bpw by Law 2, and the *direction* is unresolved: PR says the middle layers need more bits (they are flattest), convention says the edges do (they matter more functionally). PR measures quantizer leverage, not functional sensitivity. Unresolved → not spent. --- ## 6. Open, and worth more than everything above - **Orthogonal rotation.** `RMSNorm` is invariant to rotation but **not** to scaling — so a rotation of the residual stream folds into the norm weights exactly and for free, and yields a *standard GGUF*. With `attn_*` at PR 0.0069 the residual stream has ~14 dominant dimensions of 2048; spreading that energy is worth an estimated ~0.9 bpw. Nobody has published rotation results for a hybrid SSM/MoE, and the fixed SSM state may not tolerate it. **Test this next.** - **Shared basis + per-expert delta.** `W_i = W̄ + Δ_i`, store `W̄` once. If `‖Δ‖/‖W‖ ≈ ¼` that is ~2 bpw for 0.4% overhead, and `W̄ᵀx` is computed once per token rather than once per expert. Needs a llama.cpp op. **The one measurement that decides it is free: compute `‖Δ_i‖/‖W̄‖` from the bf16.** Counter-argument: this architecture already has a shared expert, which may have absorbed the common component. - **`up`↔`down` scale migration.** `scale(up row j, s)` and `scale(down col j, 1/s)` is *exactly* output-preserving. `up`'s output channels are rows and already get their own k-quant scales (free), while `down`'s input channels are one element in each of 2048 rows (polluting every row's range). Migrate the variance out of `down`'s columns into `up`'s rows. Format-compatible, never done in GGUF. Related: [[apex-standard-pipeline]] · [[map-transfers-imatrix-does-not]]