ICE technical report v1.0 with full evidence appendix

08ee00e verified 2 months ago

11.7 kB

title: ICE — quantization by error-propagation class
created: 2026-08-21T00:00:00.000Z
status: measured — ladder shipped 2026-08-21, one derived rule refuted
scope: >-
  model-agnostic. Instantiated for Ornith-1.5-35B-A3B; written to be re-derived
  for any sparse MoE.

ICE

Freeze what propagates. Spend everything else on the library.

Every quantizer in the GGUF ecosystem — llama.cpp's presets, Unsloth Dynamic, mudler's APEX, AtomicChat's AD — minimises the same objective for every tensor:

imatrix-weighted MSE of this tensor's output, for the current token.

That objective is correct for tensors whose error dies with the token. It is the wrong objective for three other kinds of tensor, and nobody separates them.

1. The four error-propagation classes

Classify every tensor by how far an error travels, not by how big its activations are.

class	mechanism	failure shape	treatment
DISCRETE	error flips an argmax; a different computation runs	not graded — categorical	exact (F32)
RECURRENT	error enters a state decay and compounds multiplicatively along the sequence	grows with context length	exact (F32)
CACHED	error is written once and re-read by later tokens	frozen in, cannot be re-decided	near-exact (F16)
INSTANT	error affects this token's output only	graded, bounded	the budget lives here

How to find them in an arbitrary architecture

DISCRETE — anything feeding a top_k, argmax, or hard gate. In an MoE that is the router (ffn_gate_inp). Also any hard routing/branching signal.
RECURRENT — anything multiplying or parameterising a carried state: SSM/Mamba decay and timestep terms (ssm_alpha, ssm_beta, ssm_a, ssm_dt), gated linear-attention decay, any RNN gate. Not the state's output projection — that one is INSTANT.
CACHED — whatever is written into the KV cache: attn_k, attn_v. Note attn_q is not cached (see §5 — this is the trap).
INSTANT — everything else.

The finding that makes this cheap

On Ornith-1.5-35B-A3B the entire propagating set is 47 M parameters — 0.14% of the model. Keeping all of it exact costs 0.15 GB. It is not a trade-off; it is a line item you can simply pay. Expect the same on any sparse MoE: routers and state gates are structurally tiny.

2. The four measured laws

Law 1 — the sparsity law: always-on bits are worth `1/sparsity` routed bits

For anything that fires every token, compute-share and storage-cost are both proportional to parameter count, so work per byte is a constant. Routed experts divide that constant by the routing fraction.

Measured on Ornith-1.5 (8 of 256 active): every always-on role scores ~270 units of work per GB-per-bpw; the experts score 8.5. The ratio is 32 — exactly 256/8.

A bit spent on the always-on core is worth E_total / E_active bits spent on the expert bank. Compute this ratio first for any new MoE; it sets the entire shape of the recipe.

Law 2 — the convexity law: allocation cleverness is capped

Quantization error falls as 4^-b, which is convex. By Jensen, non-uniform allocation only beats uniform when sensitivity varies enormously, and the gain scales with the logarithm of the spread.

Measured, by water-filling over the real per-expert routing histogram:

sensitivity spread	optimal gain over uniform
46× (real expert routing)	+0.139 bpw
3.7× (depth spread of `ffn_down`)	less
~250×	~1 bpw

Corollary: stop tuning ladders. This is why every published recipe lands within ~0.3 bpw of every other one. Per-expert allocation, depth grading, and per-layer bands are all worth less than 0.15 bpw. ICE therefore does no depth grading at all — an explicit refusal, not an oversight.

Law 3 — the leverage law: flat importance is the expensive case

The imatrix stores per-channel importance h. Measure its concentration with the participation ratio PR = (Σh)² / (n·Σh²).

Spiky h (low PR) is a gift — the quantizer knows which few channels to spend the block scale on. Flat h (high PR) is the hard case — every channel matters and there is no leverage. So flat tensors need more bits.

Measured on Ornith-1.5:

tensor	PR/n
`attn_k/q/v`	0.0069 (~14 of 2048 channels)
`attn_qkv`, `attn_gate`, `ssm_alpha/beta`	0.0128
`ffn_gate_exps`, `ffn_up_exps`	0.2048
`ffn_down_exps`	0.6104 — flattest in the model

~~down should sit log4(PR_down / PR_gateup) = 0.79 bpw above gate/up.~~ REFUTED BY MEASUREMENT, 2026-08-21.

This looked like a derivation of what convention does by feel, and it agreed with the best published rungs (UD-Q4_K_XL and AtomicChat AD-Q5_K-Q4_K both run ~1.0). A controlled test at identical size says otherwise:

expert allocation	size	mean KLD
`down` = `gate/up` + 0.79	20.82 GB	0.046449
uniform (gap = 0)	20.85 GB	0.041192

Uniform is 11.3% better, reproduced exactly on a rebuild (0.041192 twice). Rebuilding the whole ladder uniform improved every rung: −7.3% / −11.3% / −3.3% / −3.1% at 19/21/23/25 GB.

What PR actually predicts is unknown. It measures how concentrated the input importance is, which is not the same as how much a tensor's error reaches the output — ffn_down writes straight into the residual stream while gate/up errors are attenuated through the gating nonlinearity first. Position in the circuit is not captured by PR, and on this model it evidently dominates.

Practical rule until someone measures better: allocate the expert stack uniformly. Do not bump ffn_down, despite convention.

Law 4 — the floor is epistemic, not numeric

Fit KLD ≈ k₀ + c_d·4^(-bpw_dense) + c_e·4^(-bpw_expert). Two independent harnesses (ours, AtomicChat's) agreed:

	floor `k₀`	`c_e/c_d`
ours (wikitext/2048)	0.0205	5.91
AtomicChat (eval_neutral/4096)	0.0219	7.22

UD-Q6_K measures 0.0221 against a 0.0205 floor — it is 8% above the floor. The top of the ladder is saturated. Above ~5.5 expert bpw you are buying calibration noise, not accuracy.

You cannot out-bit a wrong prior. Quantization is lossy compression guided by a belief about importance. Improve the imatrix before adding bits.

Caveat: this fitted model ranks the extremes well and gets close calls wrong (it misorders AtomicChat's Q4_K_M vs AD-Q4_K-IQ4_XS). Use it to find the floor and the saturation point, never to choose between two candidate recipes.

3. The ICE assignment

DISCRETE  routers, hard gates                       F32
RECURRENT state decay / timestep terms              F32
CACHED    attn_k, attn_v                            F16
ALWAYS-ON output head, token_embd, attention,       Q8_0
          state output proj, shared expert
INSTANT   routed experts                            the dial
            down = gate = up  (UNIFORM - measured, see Law 3)

Plus one decision specific to speculative-decoding models:

A draft head does not need target-model precision. Its errors are verified by the target, so they cost acceptance, not correctness. ICE pins only the draft block's projections and lets its experts follow the tier. On Ornith-1.5 that frees ~0.40 GB. Everyone else pins the whole block Q8_0, drops it to Q4_0, or deletes it — nobody asks what a drafter actually needs.

4. Adapting ICE to a new MoE — the checklist

Inventory. Dump the tensor list. Group by role. Compute params, % of file, and % of active compute per role (weight routed experts by k/E).
Sparsity ratio E/k → Law 1. This is your always-on-to-expert exchange rate.
Classify every tensor into the four propagation classes (§1). Sum the propagating set — if it is ≪1% of the model, freeze all of it.
PR from the imatrix per role → Law 3. Informative about quantizer leverage, but it did NOT predict the right expert split on Ornith-1.5 — allocate the expert stack uniformly unless a controlled test on your model says otherwise. analysis/imatrix_anatomy.py computes PR from the imatrix alone, no weights.
Routing histogram from the imatrix counts → Law 2 → confirm depth/expert grading is not worth the risk. analysis/routing_skew.py.
Fixed body = propagating set + always-on core + draft projections. Everything else is one dial: size = body + E_params · bpw / 8.
Bound the ladder. Fit Law 4 on any measured tiers to find k₀; the useful band runs from the cliff (where the expert term explodes) to saturation (where the floor dominates). On Ornith-1.5 that was 17–27 GB — five rungs, not eight.

5. What was tried and rejected — with numbers

Negative results, kept because they are the expensive part to rediscover.

Per-expert bit allocation by routing frequency. Measured: load balancing works (209.5 effective experts of 256, gini 0.331), and convexity caps the gain at +0.139 bpw. A naive "hot 25% at 6.5, cold at 2.5" split is worse than uniform (0.00586 vs 0.00552). Dead.
Demoting attn_q on the argument that Q's error dies with the token while K/V's persists. The arithmetic kills it: each query distributes attention weight summing to 1, so total k-influence over a sequence is O(N) — the same as q. Buys 63 MB, costs 0.17 active bpw. Kept only as an A/B flag.
Active bpw as a ranking metric. Tested against measured KLD it misorders 4 of our 8 tiers (Spearman +0.95) and 10 of AtomicChat's 13 (+0.71). It over-credits Q8_0-dense-path designs because error is not linear in bits. Use it to explain where bits went; rank on measured KLD.
Depth grading. Worth <0.15 bpw by Law 2, and the direction is unresolved: PR says the middle layers need more bits (they are flattest), convention says the edges do (they matter more functionally). PR measures quantizer leverage, not functional sensitivity. Unresolved → not spent.

6. Open, and worth more than everything above

Orthogonal rotation. RMSNorm is invariant to rotation but not to scaling — so a rotation of the residual stream folds into the norm weights exactly and for free, and yields a standard GGUF. With attn_* at PR 0.0069 the residual stream has ~14 dominant dimensions of 2048; spreading that energy is worth an estimated ~0.9 bpw. Nobody has published rotation results for a hybrid SSM/MoE, and the fixed SSM state may not tolerate it. Test this next.
Shared basis + per-expert delta. W_i = W̄ + Δ_i, store W̄ once. If ‖Δ‖/‖W‖ ≈ ¼ that is ~2 bpw for 0.4% overhead, and W̄ᵀx is computed once per token rather than once per expert. Needs a llama.cpp op. The one measurement that decides it is free: compute ‖Δ_i‖/‖W̄‖ from the bf16. Counter-argument: this architecture already has a shared expert, which may have absorbed the common component.
up↔down scale migration. scale(up row j, s) and scale(down col j, 1/s) is exactly output-preserving. up's output channels are rows and already get their own k-quant scales (free), while down's input channels are one element in each of 2048 rows (polluting every row's range). Migrate the variance out of down's columns into up's rows. Format-compatible, never done in GGUF.

Related: [[apex-standard-pipeline]] · [[map-transfers-imatrix-does-not]]

ICE