ICE technical report v1.0 with full evidence appendix

08ee00e verified 2 months ago

11.7 kB

---
title: ICE — quantization by error-propagation class
created: 2026-08-21
status: measured — ladder shipped 2026-08-21, one derived rule refuted
scope: model-agnostic. Instantiated for Ornith-1.5-35B-A3B; written to be re-derived
       for any sparse MoE.
---
# ICE

**Freeze what propagates. Spend everything else on the library.**

Every quantizer in the GGUF ecosystem — llama.cpp's presets, Unsloth Dynamic,
mudler's APEX, AtomicChat's AD — minimises the same objective for every tensor:

> imatrix-weighted MSE of *this tensor's output, for the current token.*

That objective is correct for tensors whose error dies with the token. It is the
wrong objective for three other kinds of tensor, and nobody separates them.

---

## 1. The four error-propagation classes

Classify every tensor by **how far an error travels**, not by how big its
activations are.

| class | mechanism | failure shape | treatment |
|---|---|---|---|
| **DISCRETE** | error flips an argmax; a different computation runs | not graded — *categorical* | exact (F32) |
| **RECURRENT** | error enters a state decay and compounds multiplicatively along the sequence | grows with context length | exact (F32) |
| **CACHED** | error is written once and re-read by later tokens | frozen in, cannot be re-decided | near-exact (F16) |
| **INSTANT** | error affects this token's output only | graded, bounded | the budget lives here |

### How to find them in an arbitrary architecture

- **DISCRETE** — anything feeding a `top_k`, `argmax`, or hard gate. In an MoE
  that is the router (`ffn_gate_inp`). Also any hard routing/branching signal.
- **RECURRENT** — anything multiplying or parameterising a carried state: SSM/Mamba
  decay and timestep terms (`ssm_alpha`, `ssm_beta`, `ssm_a`, `ssm_dt`), gated
  linear-attention decay, any RNN gate. **Not** the state's output projection —
  that one is INSTANT.
- **CACHED** — whatever is written into the KV cache: `attn_k`, `attn_v`. Note
  `attn_q` is *not* cached (see §5 — this is the trap).
- **INSTANT** — everything else.

### The finding that makes this cheap

On Ornith-1.5-35B-A3B the entire propagating set is **47 M parameters — 0.14% of
the model. Keeping all of it exact costs 0.15 GB.** It is not a trade-off; it is a
line item you can simply pay. Expect the same on any sparse MoE: routers and state
gates are structurally tiny.

---

## 2. The four measured laws

### Law 1 — the sparsity law: always-on bits are worth `1/sparsity` routed bits

For anything that fires every token, compute-share and storage-cost are both
proportional to parameter count, so *work per byte* is a constant. Routed experts
divide that constant by the routing fraction.

Measured on Ornith-1.5 (8 of 256 active): every always-on role scores ~270 units of
work per GB-per-bpw; the experts score **8.5. The ratio is 32 — exactly 256/8.**

> **A bit spent on the always-on core is worth `E_total / E_active` bits spent on
> the expert bank.** Compute this ratio first for any new MoE; it sets the entire
> shape of the recipe.

### Law 2 — the convexity law: allocation cleverness is capped

Quantization error falls as `4^-b`, which is convex. By Jensen, non-uniform
allocation only beats uniform when sensitivity varies enormously, and the gain
scales with the **logarithm** of the spread.

Measured, by water-filling over the real per-expert routing histogram:

| sensitivity spread | optimal gain over uniform |
|---|---|
| 46× (real expert routing) | **+0.139 bpw** |
| 3.7× (depth spread of `ffn_down`) | less |
| ~250× | ~1 bpw |

> **Corollary: stop tuning ladders.** This is why every published recipe lands
> within ~0.3 bpw of every other one. Per-expert allocation, depth grading, and
> per-layer bands are all worth less than 0.15 bpw. ICE therefore does **no depth
> grading at all** — an explicit refusal, not an oversight.

### Law 3 — the leverage law: flat importance is the expensive case

The imatrix stores per-channel importance `h`. Measure its concentration with the
participation ratio `PR = (Σh)² / (n·Σh²)`.

**Spiky `h` (low PR) is a gift** — the quantizer knows which few channels to spend
the block scale on. **Flat `h` (high PR) is the hard case** — every channel matters
and there is no leverage. So *flat tensors need more bits.*

Measured on Ornith-1.5:

| tensor | PR/n |
|---|---:|
| `attn_k/q/v` | 0.0069 (~14 of 2048 channels) |
| `attn_qkv`, `attn_gate`, `ssm_alpha/beta` | 0.0128 |
| `ffn_gate_exps`, `ffn_up_exps` | 0.2048 |
| **`ffn_down_exps`** | **0.6104 — flattest in the model** |

>  ~~`down` should sit `log4(PR_down / PR_gateup)` = 0.79 bpw above `gate`/`up`.~~
> **REFUTED BY MEASUREMENT, 2026-08-21.**

This looked like a derivation of what convention does by feel, and it agreed with
the best published rungs (UD-Q4_K_XL and AtomicChat AD-Q5_K-Q4_K both run ~1.0).
A controlled test at identical size says otherwise:

| expert allocation | size | mean KLD |
|---|---:|---:|
| `down` = `gate/up` + 0.79 | 20.82 GB | 0.046449 |
| **uniform (gap = 0)** | 20.85 GB | **0.041192** |

**Uniform is 11.3% better**, reproduced exactly on a rebuild (0.041192 twice).
Rebuilding the whole ladder uniform improved every rung: −7.3% / −11.3% / −3.3% /
−3.1% at 19/21/23/25 GB.

**What PR actually predicts is unknown.** It measures how concentrated the *input*
importance is, which is not the same as how much a tensor's error reaches the
output — `ffn_down` writes straight into the residual stream while `gate`/`up`
errors are attenuated through the gating nonlinearity first. Position in the
circuit is not captured by PR, and on this model it evidently dominates.

**Practical rule until someone measures better: allocate the expert stack
uniformly.** Do not bump `ffn_down`, despite convention.

### Law 4 — the floor is epistemic, not numeric

Fit `KLD ≈ k₀ + c_d·4^(-bpw_dense) + c_e·4^(-bpw_expert)`. Two independent
harnesses (ours, AtomicChat's) agreed:

| | floor `k₀` | `c_e/c_d` |
|---|---:|---:|
| ours (wikitext/2048) | 0.0205 | 5.91 |
| AtomicChat (eval_neutral/4096) | 0.0219 | 7.22 |
`UD-Q6_K` measures 0.0221 against a 0.0205 floor — **it is 8% above the floor. The
top of the ladder is saturated.** Above ~5.5 expert bpw you are buying calibration
noise, not accuracy.

> **You cannot out-bit a wrong prior.** Quantization is lossy compression *guided by
> a belief about importance*. Improve the imatrix before adding bits.

**Caveat:** this fitted model ranks the extremes well and gets close calls wrong
(it misorders AtomicChat's `Q4_K_M` vs `AD-Q4_K-IQ4_XS`). Use it to find the floor
and the saturation point, never to choose between two candidate recipes.

---

## 3. The ICE assignment

```
DISCRETE  routers, hard gates                       F32
RECURRENT state decay / timestep terms              F32
CACHED    attn_k, attn_v                            F16
ALWAYS-ON output head, token_embd, attention,       Q8_0
          state output proj, shared expert
INSTANT   routed experts                            the dial
            down = gate = up  (UNIFORM - measured, see Law 3)
```

Plus one decision specific to speculative-decoding models:

> **A draft head does not need target-model precision.** Its errors are *verified*
> by the target, so they cost **acceptance**, not correctness. ICE pins only the
> draft block's projections and lets its experts follow the tier. On Ornith-1.5
> that frees ~0.40 GB. Everyone else pins the whole block Q8_0, drops it to Q4_0,
> or deletes it — nobody asks what a drafter actually needs.

---

## 4. Adapting ICE to a new MoE — the checklist

1. **Inventory.** Dump the tensor list. Group by role. Compute params, % of file,
   and % of *active* compute per role (weight routed experts by `k/E`).
2. **Sparsity ratio `E/k`** → Law 1. This is your always-on-to-expert exchange rate.
3. **Classify** every tensor into the four propagation classes (§1). Sum the
   propagating set — if it is ≪1% of the model, freeze all of it.
4. **PR from the imatrix** per role → Law 3. Informative about *quantizer leverage*,
   but it did NOT predict the right expert split on Ornith-1.5 — allocate the expert
   stack uniformly unless a controlled test on your model says otherwise.
   `analysis/imatrix_anatomy.py` computes PR from the imatrix alone, no weights.
5. **Routing histogram** from the imatrix `counts` → Law 2 → confirm depth/expert
   grading is not worth the risk. `analysis/routing_skew.py`.
6. **Fixed body** = propagating set + always-on core + draft projections.
   Everything else is one dial: `size = body + E_params · bpw / 8`.
7. **Bound the ladder.** Fit Law 4 on any measured tiers to find `k₀`; the useful
   band runs from the cliff (where the expert term explodes) to saturation
   (where the floor dominates). On Ornith-1.5 that was 17–27 GB — five rungs, not
   eight.

---

## 5. What was tried and rejected — with numbers

Negative results, kept because they are the expensive part to rediscover.

- **Per-expert bit allocation by routing frequency.** Measured: load balancing
  works (209.5 effective experts of 256, gini 0.331), and convexity caps the gain
  at **+0.139 bpw**. A naive "hot 25% at 6.5, cold at 2.5" split is *worse than
  uniform* (0.00586 vs 0.00552). **Dead.**
- **Demoting `attn_q`** on the argument that Q's error dies with the token while
  K/V's persists. The arithmetic kills it: each query distributes attention weight
  summing to 1, so total k-influence over a sequence is O(N) — **the same as q**.
  Buys 63 MB, costs 0.17 active bpw. Kept only as an A/B flag.
- **Active bpw as a ranking metric.** Tested against measured KLD it misorders
  **4 of our 8 tiers** (Spearman +0.95) and **10 of AtomicChat's 13** (+0.71).
  It over-credits Q8_0-dense-path designs because error is not linear in bits.
  **Use it to explain where bits went; rank on measured KLD.**
- **Depth grading.** Worth <0.15 bpw by Law 2, and the *direction* is unresolved:
  PR says the middle layers need more bits (they are flattest), convention says
  the edges do (they matter more functionally). PR measures quantizer leverage,
  not functional sensitivity. Unresolved → not spent.

---

## 6. Open, and worth more than everything above

- **Orthogonal rotation.** `RMSNorm` is invariant to rotation but **not** to
  scaling — so a rotation of the residual stream folds into the norm weights
  exactly and for free, and yields a *standard GGUF*. With `attn_*` at PR 0.0069
  the residual stream has ~14 dominant dimensions of 2048; spreading that energy
  is worth an estimated ~0.9 bpw. Nobody has published rotation results for a
  hybrid SSM/MoE, and the fixed SSM state may not tolerate it. **Test this next.**
- **Shared basis + per-expert delta.** `W_i = W̄ + Δ_i`, store `W̄` once. If
  `‖Δ‖/‖W‖ ≈ ¼` that is ~2 bpw for 0.4% overhead, and `W̄ᵀx` is computed once per
  token rather than once per expert. Needs a llama.cpp op. **The one measurement
  that decides it is free: compute `‖Δ_i‖/‖W̄‖` from the bf16.** Counter-argument:
  this architecture already has a shared expert, which may have absorbed the
  common component.
- **`up`↔`down` scale migration.** `scale(up row j, s)` and `scale(down col j, 1/s)`
  is *exactly* output-preserving. `up`'s output channels are rows and already get
  their own k-quant scales (free), while `down`'s input channels are one element
  in each of 2048 rows (polluting every row's range). Migrate the variance out of
  `down`'s columns into `up`'s rows. Format-compatible, never done in GGUF.
Related: [[apex-standard-pipeline]] · [[map-transfers-imatrix-does-not]]

	---
	title: ICE — quantization by error-propagation class
	created: 2026-08-21
	status: measured — ladder shipped 2026-08-21, one derived rule refuted
	scope: model-agnostic. Instantiated for Ornith-1.5-35B-A3B; written to be re-derived
	for any sparse MoE.
	---

	# ICE

	Freeze what propagates. Spend everything else on the library.

	Every quantizer in the GGUF ecosystem — llama.cpp's presets, Unsloth Dynamic,
	mudler's APEX, AtomicChat's AD — minimises the same objective for every tensor:

	> imatrix-weighted MSE of this tensor's output, for the current token.

	That objective is correct for tensors whose error dies with the token. It is the
	wrong objective for three other kinds of tensor, and nobody separates them.

	---

	## 1. The four error-propagation classes

	Classify every tensor by how far an error travels, not by how big its
	activations are.

	\| class \| mechanism \| failure shape \| treatment \|
	\|---\|---\|---\|---\|
	\| DISCRETE \| error flips an argmax; a different computation runs \| not graded — categorical \| exact (F32) \|
	\| RECURRENT \| error enters a state decay and compounds multiplicatively along the sequence \| grows with context length \| exact (F32) \|
	\| CACHED \| error is written once and re-read by later tokens \| frozen in, cannot be re-decided \| near-exact (F16) \|
	\| INSTANT \| error affects this token's output only \| graded, bounded \| the budget lives here \|

	### How to find them in an arbitrary architecture

	- DISCRETE — anything feeding a `top_k`, `argmax`, or hard gate. In an MoE
	that is the router (`ffn_gate_inp`). Also any hard routing/branching signal.
	- RECURRENT — anything multiplying or parameterising a carried state: SSM/Mamba
	decay and timestep terms (`ssm_alpha`, `ssm_beta`, `ssm_a`, `ssm_dt`), gated
	linear-attention decay, any RNN gate. Not the state's output projection —
	that one is INSTANT.
	- CACHED — whatever is written into the KV cache: `attn_k`, `attn_v`. Note
	`attn_q` is not cached (see §5 — this is the trap).
	- INSTANT — everything else.

	### The finding that makes this cheap

	On Ornith-1.5-35B-A3B the entire propagating set is **47 M parameters — 0.14% of
	the model. Keeping all of it exact costs 0.15 GB.** It is not a trade-off; it is a
	line item you can simply pay. Expect the same on any sparse MoE: routers and state
	gates are structurally tiny.

	---

	## 2. The four measured laws

	### Law 1 — the sparsity law: always-on bits are worth `1/sparsity` routed bits

	For anything that fires every token, compute-share and storage-cost are both
	proportional to parameter count, so work per byte is a constant. Routed experts
	divide that constant by the routing fraction.

	Measured on Ornith-1.5 (8 of 256 active): every always-on role scores ~270 units of
	work per GB-per-bpw; the experts score 8.5. The ratio is 32 — exactly 256/8.

	> **A bit spent on the always-on core is worth `E_total / E_active` bits spent on
	> the expert bank.** Compute this ratio first for any new MoE; it sets the entire
	> shape of the recipe.

	### Law 2 — the convexity law: allocation cleverness is capped

	Quantization error falls as `4^-b`, which is convex. By Jensen, non-uniform
	allocation only beats uniform when sensitivity varies enormously, and the gain
	scales with the logarithm of the spread.

	Measured, by water-filling over the real per-expert routing histogram:

	\| sensitivity spread \| optimal gain over uniform \|
	\|---\|---\|
	\| 46× (real expert routing) \| +0.139 bpw \|
	\| 3.7× (depth spread of `ffn_down`) \| less \|
	\| ~250× \| ~1 bpw \|

	> Corollary: stop tuning ladders. This is why every published recipe lands
	> within ~0.3 bpw of every other one. Per-expert allocation, depth grading, and
	> per-layer bands are all worth less than 0.15 bpw. ICE therefore does **no depth
	> grading at all** — an explicit refusal, not an oversight.

	### Law 3 — the leverage law: flat importance is the expensive case

	The imatrix stores per-channel importance `h`. Measure its concentration with the
	participation ratio `PR = (Σh)² / (n·Σh²)`.

	Spiky `h` (low PR) is a gift — the quantizer knows which few channels to spend
	the block scale on. Flat `h` (high PR) is the hard case — every channel matters
	and there is no leverage. So flat tensors need more bits.

	Measured on Ornith-1.5:

	\| tensor \| PR/n \|
	\|---\|---:\|
	\| `attn_k/q/v` \| 0.0069 (~14 of 2048 channels) \|
	\| `attn_qkv`, `attn_gate`, `ssm_alpha/beta` \| 0.0128 \|
	\| `ffn_gate_exps`, `ffn_up_exps` \| 0.2048 \|
	\| `ffn_down_exps` \| 0.6104 — flattest in the model \|

	> ~~`down` should sit `log4(PR_down / PR_gateup)` = 0.79 bpw above `gate`/`up`.~~
	> REFUTED BY MEASUREMENT, 2026-08-21.

	This looked like a derivation of what convention does by feel, and it agreed with
	the best published rungs (UD-Q4_K_XL and AtomicChat AD-Q5_K-Q4_K both run ~1.0).
	A controlled test at identical size says otherwise:

	\| expert allocation \| size \| mean KLD \|
	\|---\|---:\|---:\|
	\| `down` = `gate/up` + 0.79 \| 20.82 GB \| 0.046449 \|
	\| uniform (gap = 0) \| 20.85 GB \| 0.041192 \|

	Uniform is 11.3% better, reproduced exactly on a rebuild (0.041192 twice).
	Rebuilding the whole ladder uniform improved every rung: −7.3% / −11.3% / −3.3% /
	−3.1% at 19/21/23/25 GB.

	What PR actually predicts is unknown. It measures how concentrated the input
	importance is, which is not the same as how much a tensor's error reaches the
	output — `ffn_down` writes straight into the residual stream while `gate`/`up`
	errors are attenuated through the gating nonlinearity first. Position in the
	circuit is not captured by PR, and on this model it evidently dominates.

	**Practical rule until someone measures better: allocate the expert stack
	uniformly.** Do not bump `ffn_down`, despite convention.

	### Law 4 — the floor is epistemic, not numeric

	Fit `KLD ≈ k₀ + c_d·4^(-bpw_dense) + c_e·4^(-bpw_expert)`. Two independent
	harnesses (ours, AtomicChat's) agreed:

	\| \| floor `k₀` \| `c_e/c_d` \|
	\|---\|---:\|---:\|
	\| ours (wikitext/2048) \| 0.0205 \| 5.91 \|
	\| AtomicChat (eval_neutral/4096) \| 0.0219 \| 7.22 \|

	`UD-Q6_K` measures 0.0221 against a 0.0205 floor — **it is 8% above the floor. The
	top of the ladder is saturated.** Above ~5.5 expert bpw you are buying calibration
	noise, not accuracy.

	> You cannot out-bit a wrong prior. Quantization is lossy compression *guided by
	> a belief about importance*. Improve the imatrix before adding bits.

	Caveat: this fitted model ranks the extremes well and gets close calls wrong
	(it misorders AtomicChat's `Q4_K_M` vs `AD-Q4_K-IQ4_XS`). Use it to find the floor
	and the saturation point, never to choose between two candidate recipes.

	---

	## 3. The ICE assignment

	```
	DISCRETE routers, hard gates F32
	RECURRENT state decay / timestep terms F32
	CACHED attn_k, attn_v F16
	ALWAYS-ON output head, token_embd, attention, Q8_0
	state output proj, shared expert
	INSTANT routed experts the dial
	down = gate = up (UNIFORM - measured, see Law 3)
	```

	Plus one decision specific to speculative-decoding models:

	> A draft head does not need target-model precision. Its errors are verified
	> by the target, so they cost acceptance, not correctness. ICE pins only the
	> draft block's projections and lets its experts follow the tier. On Ornith-1.5
	> that frees ~0.40 GB. Everyone else pins the whole block Q8_0, drops it to Q4_0,
	> or deletes it — nobody asks what a drafter actually needs.

	---

	## 4. Adapting ICE to a new MoE — the checklist

	1. Inventory. Dump the tensor list. Group by role. Compute params, % of file,
	and % of active compute per role (weight routed experts by `k/E`).
	2. Sparsity ratio `E/k` → Law 1. This is your always-on-to-expert exchange rate.
	3. Classify every tensor into the four propagation classes (§1). Sum the
	propagating set — if it is ≪1% of the model, freeze all of it.
	4. PR from the imatrix per role → Law 3. Informative about quantizer leverage,
	but it did NOT predict the right expert split on Ornith-1.5 — allocate the expert
	stack uniformly unless a controlled test on your model says otherwise.
	`analysis/imatrix_anatomy.py` computes PR from the imatrix alone, no weights.
	5. Routing histogram from the imatrix `counts` → Law 2 → confirm depth/expert
	grading is not worth the risk. `analysis/routing_skew.py`.
	6. Fixed body = propagating set + always-on core + draft projections.
	Everything else is one dial: `size = body + E_params · bpw / 8`.
	7. Bound the ladder. Fit Law 4 on any measured tiers to find `k₀`; the useful
	band runs from the cliff (where the expert term explodes) to saturation
	(where the floor dominates). On Ornith-1.5 that was 17–27 GB — five rungs, not
	eight.

	---

	## 5. What was tried and rejected — with numbers

	Negative results, kept because they are the expensive part to rediscover.

	- Per-expert bit allocation by routing frequency. Measured: load balancing
	works (209.5 effective experts of 256, gini 0.331), and convexity caps the gain
	at +0.139 bpw. A naive "hot 25% at 6.5, cold at 2.5" split is *worse than
	uniform* (0.00586 vs 0.00552). Dead.
	- Demoting `attn_q` on the argument that Q's error dies with the token while
	K/V's persists. The arithmetic kills it: each query distributes attention weight
	summing to 1, so total k-influence over a sequence is O(N) — the same as q.
	Buys 63 MB, costs 0.17 active bpw. Kept only as an A/B flag.
	- Active bpw as a ranking metric. Tested against measured KLD it misorders
	4 of our 8 tiers (Spearman +0.95) and 10 of AtomicChat's 13 (+0.71).
	It over-credits Q8_0-dense-path designs because error is not linear in bits.
	Use it to explain where bits went; rank on measured KLD.
	- Depth grading. Worth <0.15 bpw by Law 2, and the direction is unresolved:
	PR says the middle layers need more bits (they are flattest), convention says
	the edges do (they matter more functionally). PR measures quantizer leverage,
	not functional sensitivity. Unresolved → not spent.

	---

	## 6. Open, and worth more than everything above

	- Orthogonal rotation. `RMSNorm` is invariant to rotation but not to
	scaling — so a rotation of the residual stream folds into the norm weights
	exactly and for free, and yields a standard GGUF. With `attn_*` at PR 0.0069
	the residual stream has ~14 dominant dimensions of 2048; spreading that energy
	is worth an estimated ~0.9 bpw. Nobody has published rotation results for a
	hybrid SSM/MoE, and the fixed SSM state may not tolerate it. Test this next.
	- Shared basis + per-expert delta. `W_i = W̄ + Δ_i`, store `W̄` once. If
	`‖Δ‖/‖W‖ ≈ ¼` that is ~2 bpw for 0.4% overhead, and `W̄ᵀx` is computed once per
	token rather than once per expert. Needs a llama.cpp op. **The one measurement
	that decides it is free: compute `‖Δ_i‖/‖W̄‖` from the bf16.** Counter-argument:
	this architecture already has a shared expert, which may have absorbed the
	common component.
	- `up`↔`down` scale migration. `scale(up row j, s)` and `scale(down col j, 1/s)`
	is exactly output-preserving. `up`'s output channels are rows and already get
	their own k-quant scales (free), while `down`'s input channels are one element
	in each of 2048 rows (polluting every row's range). Migrate the variance out of
	`down`'s columns into `up`'s rows. Format-compatible, never done in GGUF.

	Related: [[apex-standard-pipeline]] · [[map-transfers-imatrix-does-not]]