---
title: ICE for Ornith-1.5-35B-A3B — the concrete instantiation
created: 2026-08-21
status: SHIPPED 2026-08-21 — 4 tiers live, uniform expert allocation
principle: see PRINCIPLE.md (model-agnostic)
---

# ICE — Ornith-1.5-35B-A3B

The general method is in [PRINCIPLE.md](PRINCIPLE.md). This file is what it
produced for *this* model, with the arithmetic shown.

## Architecture inputs

| | |
|---|---|
| blocks | 41 = 30 linear/SSM + 10 full-attention (3,7,…,39) + 1 MTP (`blk.40`) |
| residual width | 2048 |
| experts | 256, 8 active → **sparsity ratio 32** |
| expert FFN / shared FFN | 512 / 512 |
| KV heads | 2, head_dim 256, on 10 blocks only → ~20 KB/token |
| vocab | 248,320 → `output.weight` = 508.6 M in one tensor |

**Every block is 95.6% expert weights** (805.3 M of ~842 M). The surgical field is
the remaining ~35 M per block plus the two 508 M end-caps.

## Active-compute geography (measured)

| role | params | % of file | % of *active* compute |
|---|---:|---:|---:|
| routed experts (8/256) | 32.21 B | 92.94% | 34.2% |
| attention | 1.03 B | 2.96% | **34.9%** |
| output head | 0.51 B | 1.47% | **17.3%** |
| ssm | 0.26 B | 0.74% | 8.7% |
| shared expert | 0.13 B | 0.36% | 4.3% |
| token_embd | 0.51 B | 1.47% | 0% (`get_rows`) |

**65.8% of active parameters are non-routed.** Active total 2.95 B — which is what
the "A3B" label means, and a useful check that the accounting is right.

## The assignment

| tensor | × | params | class | type |
|---|---:|---:|---|---|
| `ffn_gate_inp`, `ffn_gate_inp_shexp` | 80 | 21.0 M | DISCRETE | F32 † |
| `ssm_conv1d` | 30 | 1.0 M | RECURRENT | F32 † |
| `ssm_alpha`, `ssm_beta` | 60 | 3.9 M | RECURRENT | **F32 (pinned)** |
| `ssm_a`, `ssm_dt`, all norms | — | 0.3 M | RECURRENT / — | F32 † |
| `attn_k`, `attn_v` | 20 | 21.0 M | **CACHED** | **F16** |
| `attn_q` | 10 | 167.8 M | INSTANT | Q8_0 |
| `attn_output` | 10 | 83.9 M | INSTANT | Q8_0 |
| `attn_qkv` | 30 | 503.3 M | INSTANT | Q8_0 |
| `attn_gate` | 30 | 251.7 M | INSTANT | Q8_0 |
| `ssm_out` | 30 | 251.7 M | INSTANT | Q8_0 |
| `ffn_*_shexp` | 120 | 125.8 M | always-on | Q8_0 |
| `output.weight` | 1 | 508.6 M | always-on | Q8_0 |
| `token_embd` | 1 | 508.6 M | — | Q8_0 |
| `blk.40.nextn.eh_proj` + blk.40 attn/shexp | — | 39.3 M | draft projections | Q8_0 |
| **`ffn_{down,gate,up}_exps`** | 123 | **33.02 B** | INSTANT | **the dial** |

† already F32 in llama.cpp: `ffn_gate_inp*` is excluded by name, `ssm_conv1d` has
first dim 4 (not a multiple of 256), 1-D tensors are never quantized. Pinning them
would be *inert* and fail GATE 0 — the intent is satisfied by the quantizer itself.

**Fixed body ≈ 2.6 GB.** Propagating set = 47 M params = **0.14% of the model**,
exact for 0.15 GB.

### Expert split

**UNIFORM** — `down` = `gate` = `up`. The first ladder shipped a 0.79 bpw bump on
`down` derived from `log4(0.6104/0.2048)`; a controlled test at identical size
refuted it (0.041192 uniform vs 0.046449 bumped, **11.3% better uniform**), and the
ladder was rebuilt. See PRINCIPLE.md Law 3.

### Draft head

`blk.40`'s **experts follow the tier**; only its projections are Q8_0. Draft errors
are verified by the target model, so they cost *acceptance*, not correctness.
Frees ~0.40 GB versus pinning the whole block.

### No depth grading

Refused deliberately. Water-filling over the real routing histogram shows the
optimal gain is **+0.139 bpw**, and the direction of a depth profile is unresolved
(PR says middle, convention says edges). Not worth the risk budget.

## The ladder

| tier | built | expert bpw | down = gate = up |
|---|---:|---:|---|
| `19G-ICE` | 18.82 GB | 3.899 | IQ3_S / IQ4_XS |
| `21G-ICE` | 20.84 GB | 4.383 | IQ4_XS / Q4_K |
| `23G-ICE` | 22.83 GB | 4.868 | Q4_K / Q5_K |
| `25G-ICE` | 24.84 GB | 5.353 | Q5_K / Q6_K |

GATE 0: **0 double-matched, 0 uncovered, 0 inert** on all four.

## Predicted position (active bpw, pre-measurement)

| build | disk | ACTIVE bpw | dense | blk.40 |
|---|---:|---:|---:|---:|
| UD-IQ4_XS | 18.68 G | 6.762 | 2.47 G | 0.90 G |
| **19G-ICE** | 18.84 G | **7.191** | 2.61 G | 0.43 G |
| **23G-ICE** | 22.83 G | **7.521** | 2.61 G | 0.53 G |
| UD-Q4_K_XL | 23.21 G | 7.471 | 2.59 G | 0.90 G |
| **25G-ICE** | 24.84 G | **7.691** | 2.61 G | 0.53 G |
| UD-Q5_K_S | 25.83 G | 7.693 | 2.59 G | 0.90 G |

`25G-ICE` matches `UD-Q5_K_S`'s active precision at **0.99 GB less**.

## Falsifiable predictions — scored

1. ~~`25G-ICE` ≥ `UD-Q5_K_S` on measured KLD at ~1 GB less.~~ **FAILED.** 0.0303 vs
   0.0272; ICE is 0.99 GB smaller but not better. The 0.79 gap it rested on was then
   refuted outright.
2. **`19G-ICE` beats `UD-IQ4_XS`** — **HELD.** 0.060844 vs 0.0723, **15.8% better**
   at 0.14 GB *smaller* (18.82 vs 18.68 GB is +0.14; after the uniform rebuild the
   file is 18.82 GB and still wins).
3. **Un-pinning `blk.40` costs <2 pp acceptance** — **HELD, direction unverified.**
   96.04% measured, above the 92.97% baseline, but on a different tier so not a
   clean comparison. No collapse.

Same harness as the UD/APEX ladder — wikitext-2-raw, 16 chunks, 2048 ctx — so the
numbers drop straight into the existing table. Changing the harness would make the
comparison meaningless.

## Results — shipped

| tier | size | mean KLD | top-1 | vs UD curve at same size |
|---|---:|---:|---:|---:|
| `25G-ICE` | 24.84 GB | 0.030304 | 93.16% | −1.9% |
| `23G-ICE` | 22.83 GB | 0.036069 | 92.65% | −6.3% |
| `21G-ICE` | 20.84 GB | 0.041192 | 92.03% | **−18.6%** |
| `19G-ICE` | 18.82 GB | 0.060844 | 89.72% | **−13.6%** |

**MTP draft acceptance: 96.04%** (388/404), measured on the shrunk `blk.40`.
Baseline with the block fully Q8_0 was 92.97% on a different tier, so this is not a
clean comparison — but it establishes the smaller draft head still drafts.

The advantage grows as size falls: at the top of the ladder the calibration floor
(~0.0185) dominates and no allocation choice can move it.

### What did not survive measurement

| idea | verdict |
|---|---|
| `down` bumped above `gate`/`up` | **refuted, −11.3%** — ship uniform |
| `token_embd` → Q4_K | **not replicated** — helps at 21G, hurts at 19G |
| `attn_gate` / `ssm_out` → Q6_K | **harmful** — always-on, 32× law |
| `attn_q` demotion | dropped before shipping — arithmetic didn't hold |
| up↔down scale migration | **dead** — `down` columns uniform (p95/p05 = 1.09) |