docs: descriptive model card

README.md

@@ -1,87 +1,339 @@
 ---
+license: gemma
+license_link: https://ai.google.dev/gemma/terms
 base_model: google/gemma-4-26B-A4B-it-qat-q4_0-unquantized
 library_name: coreai
-license: gemma
+pipeline_tag: text-generation
 tags:
-- core-ai
-- aimodel
-- moe
-- on-device
-- apple-silicon
-extra_gated_heading: Access Gemma on Hugging Face
-extra_gated_description: This artifact is a derivative of
-  google/gemma-4-26B-A4B-it-qat-q4_0-unquantized and is governed by the Gemma Terms of
-  Use, the Gemma Prohibited Use Policy and the Gemma license.
+  - core-ai
+  - aimodel
+  - apple-silicon
+  - on-device
+  - quantized
+  - int4
+  - qat
+  - moe
+  - mixture-of-experts
+  - gemma4
 ---
 
-# Gemma 4 26B-A4B - Core AI (MoE, int4)
+# Gemma 4 26B-A4B (MoE) — Core AI (.aimodel)
+
+`google/gemma-4-26B-A4B-it-qat-q4_0-unquantized` converted to a Core AI `.aimodel` bundle
+for Apple silicon by the [Visible](https://visible.cx) project. **Derivative artifact**:
+Google's QAT-trained weights rounded onto int4 and re-expressed as a Core AI graph, with the
+128-expert sparse branch of every layer lowered onto a Metal gather kernel.
+
+This is a **sparse mixture-of-experts** model — 26.5B total parameters, roughly 4B active
+per token — and it is the first Gemma 4 MoE to exist as a Core AI bundle at all. Producing
+it required authoring a Gemma 4 MoE decoder layer, not just wiring two finished halves
+together: `coreai_models` shipped five Gemma 4 modules and not one line of MoE in any of
+them.
 
-`gpu-pipelined/gemma4_26b_a4b_qat_decode_int4linsym_moeaff4_msdpa_g8` - a decode-only (S=1) Core AI bundle for Apple's pipelined GPU
-engine, converted from
-[`google/gemma-4-26B-A4B-it-qat-q4_0-unquantized`](https://huggingface.co/google/gemma-4-26B-A4B-it-qat-q4_0-unquantized).
+> ⚠️ **UNQUALIFIED, community tier.** Nothing here has produced a token, on any machine, and
+> **no Mac-side qualification is scheduled** — the machine that commissioned the conversion
+> cannot load the result. It is published so people with larger Macs have the artifact. It
+> is not published as a verified one. Read the qualification section before using it for
+> anything.
+>
+> ⚠️ **The 16 GB tier is excluded outright.** Weights are ~17.6 GB resident, against a
+> 16 GB Mac's entire ~10.7 GB Metal working set. Declared target: **Macs with ≥24–32 GB
+> unified memory.**
 
-26.5B total parameters, **128 experts, top-8 routed** (~4B active per token), 30 layers
-(25 sliding / 5 full attention), vocab 262,144.
+## Provenance
 
 | | |
 |---|---|
-| Bundle bytes | 17,580,059,414 |
-| Context (manifest) | 16384 |
-| Functions | main |
-| Producer | `coreai-core 1.0.0b2` |
-| Created | `20260817T215629Z` |
-| KV cache | fp16, one growing pair `[30, 1, 8, S, 512]` |
+| Base checkpoint | `google/gemma-4-26B-A4B-it-qat-q4_0-unquantized` @ `f1e06dc520982d9b9edd76859fdb7ab209449949` |
+| Checkpoint size | 51,644,341,801 B, bf16, 1,013 tensors across 2 shards; ungated |
+| Recipe | `export_gemma4_moe_decode_pipelined.py int4lin --lin-sym --metal-sdpa --max-ctx 16384` |
+| Toolchain base | `apple/coreai-models` @ `b1cb71b8522d99408059fa0b98b8742171bcb0b8` + the [coreai-model-zoo](https://github.com/john-rocky/coreai-model-zoo) python overlay, plus three new modules (below) |
+| Toolchain | `coreai-torch 0.4.1`, `coreai-core 1.0.0b2`, `coreai-opt 0.2.1`, `torch 2.9.0` |
+| Producer fingerprint | `coreai-core 1.0.0b2` — **PASS**, verified on the inner asset `metadata.json` |
+| Asset creation date | `20260817T215629Z` |
+| Vocab | 262,144 |
+| Conversion | **102.25 GB peak RSS** (`VmHWM`), Linux x86_64 on a 247 GB box |
+
+"QAT-unquantized" means QAT-*trained*, stored bf16; the int4 rounding happens at export onto
+the ggml q4_0 grid the training already targeted. There is no separate pre-quantised int4
+artifact to prefer.
+
+### Architecture, and why this is a small 31B rather than a big E4B
+
+| | 26B-A4B | 31B | E4B |
+|---|---|---|---|
+| `hidden_size` | **2816** | 5376 | 2560 |
+| layers | **30** (25 sliding / 5 full) | 60 | 42 |
+| dense MLP `intermediate_size` | **2112** | 21504 | 10240 |
+| `num_experts` / `top_k` | **128 / 8** | — | — |
+| `moe_intermediate_size` | **704** | — | — |
+| attention heads | **16** | 32 | 8 |
+| KV heads (sliding / full) | **8 / 2** | 16 / 4 | 2 / — |
+| `head_dim` / `global_head_dim` | **256 / 512** | 256 / 512 | 256 / 512 |
+| `sliding_window` | **1024** | 1024 | 512 |
+| Per-Layer Embeddings | **none** | none | 256 |
+| KV-shared layers | **0** | 0 | 18 |
+
+Same attention family as the dense 12B/31B, same dual head_dim, same `attention_k_eq_v`
+(full layers carry no `v_proj`), same dual RoPE, same softcap. **Everything the E2B/E4B path
+exists to handle is absent here**, which is why the port started from the dense model rather
+than the pipelined PLE one, and why there is no gather-table sidecar in this repo.
+
+### The block that had to be authored
+
+Every layer carries a dense MLP **and** a sparse branch, in parallel — this is not "MoE
+replaces the FFN", and getting that wrong would have produced a plausible model with wrong
+outputs:
+
+```python
+residual = x                                              # post-attention hidden
+h  = mlp(pre_feedforward_layernorm(x))                    # dense branch
+h1 = post_feedforward_layernorm_1(h)
+w, idx = router(residual)                                 # routes on the RAW residual
+h2 = experts(pre_feedforward_layernorm_2(residual), idx)  # sparse branch
+h2 = post_feedforward_layernorm_2(sum_k w_k * h2_k)
+x  = residual + post_feedforward_layernorm(h1 + h2)
+x  = x * layer_scalar
+```
+
+Two details are load-bearing and both are silently wrong if missed: **the router reads the
+un-normalised residual**, and **the experts are GELU-gated** (`gelu_pytorch_tanh`), where the
+`SwitchGLU` primitive's default is SiLU. Building it with the default would have produced a
+model that loads, traces, exports and fingerprints — and generates wrong text.
+
+The router itself: scale-free RMSNorm → `× scale × hidden**-0.5` → linear to 128 → fp32
+softmax over all experts → top-8 → renormalise to sum 1 → multiply by a learned
+`per_expert_scale` gathered at the selected ids.
+
+### Compression recipe
 
-## Compression
+| Weights | Scheme | Why |
+|---|---|---|
+| **Routed experts** (128/layer, 22.84 G params) | **affine int4** (`aff4`), per-(output row, K-block-32) scale + bias, MSE-optimal clip over 6 candidates | see below |
+| Attention, dense MLP, untied LM head | linear **int4** per-block-32, plain absmax (`--lin-sym`) | the ggml q4_0 grid the QAT checkpoint was trained on — the 12B/31B recipe verbatim |
+| **Router** (`proj`, `scale`, `per_expert_scale`) | **fp16**, excluded by name | routers are the quantization-sensitive part of an MoE and the whole set is 0.02 GB |
+| `embed_tokens` (in-graph) | fp16 | the gather stays exact; the head is untied and quantized separately |
 
-| Weights | Scheme |
+`sym8` on the experts would have put them at 22.8 GB before anything else — a bundle nobody
+can load — so int4 is not optional. `aff4` was chosen over a k-means palette (`km4`)
+deliberately and against the usual zoo preference, because that preference is about
+**non-QAT** weight distributions: this checkpoint was QAT-trained on per-block-32 q4_0
+(`w = (q-8)·d`, 16 uniform levels), and an affine int4 block-32 grid can represent that
+**exactly** (`scale = d`, `bias = -8d`). A palette fits one 16-entry codebook per 32 output
+rows across the whole K axis and structurally cannot follow per-block QAT scales — choosing
+it would throw away the reason for using a QAT checkpoint. The price is aux bytes: `aff4`
+stores an fp16 scale *and* bias per 32-element K block, i.e. **5 bits/param effective**
+against `km4`'s ~4.03. That is ~3 GB of the bundle, and it is the right ~3 GB.
+
+The `down` projection's K of 704 is padded to 768 by the kernel's automatic `_kpad256`
+treatment — 64 zero columns, ~0.5% wasted expert bytes. `gate`/`up` need no padding.
+
+**The gather kernel is the whole point.** Without `MetalSwitchGLU`, the sparse branch lowers
+to a dense matmul that reads **all 128 experts every token** — a 16× over-read that would
+make this bundle slower per token than the dense 31B while being worse at everything.
+
+### New source, kept with the documentation
+
+Three files were authored for this port and are a clean, self-contained zoo contribution;
+nothing in the shared toolchain was modified:
+
+| File | What |
 |---|---|
-| Routed experts (128/layer) | **affine int4**, per-(output-row, K-block-32) scale+bias, MSE-optimal clip - read through the `gather_qmm` custom Metal kernel so decode fetches only the 8 routed experts, not all 128 |
-| Attention, dense MLP, LM head (untied) | linear int4, per-block-32, plain absmax (`symmetric`) - the ggml q4_0 grid the QAT checkpoint was trained on |
-| Router (`router.proj`, `scale`, `per_expert_scale`) | **fp16** - routers are the quantization-sensitive part of an MoE and this one is ~0.02 GB |
-| `embed_tokens` (in-graph) | fp16 - the gather stays exact |
-| Norms / RoPE / SDPA | no quantized weight |
-
-Affine int4 block-32 was chosen over k-means int4 because it can represent the
-checkpoint's QAT q4_0 grid exactly (`scale = d`, `bias = -8d`), whereas a 16-entry
-codebook spanning the whole K axis cannot follow per-block QAT scales.
-
-## Hardware target
-
-**Macs with >= 24-32 GB unified memory.** Resident weights are ~17.6 GB (the bundle
-size above is the weights), before any KV cache; KV costs **491,520 B per token of
-context** (fp16, one growing `[30, 1, 8, S, 512]` pair), i.e. 2.0 GB at 4096 and 8.1 GB
-at the manifest's 16384.
-
-**This bundle is NOT for the 16 GB tier.** A 16 GB Mac's Metal
-`recommendedMaxWorkingSetSize` is roughly 10.7 GB - the weights alone are over that
-before a single token of context. Do not select it there.
-
-## Status: UNQUALIFIED - community tier
-
-**Converted and fingerprinted; never run.** The conversion box is Linux with no GPU
-delegate, so this bundle has never produced a token, and **no Mac-side qualification is
-scheduled** - the machine that commissioned it cannot load it. Every performance or
-quality claim you might expect here is therefore absent on purpose: there are no
-measured tok/s, no oracle margin, no residency measurement.
-
-What *was* gated, on the box: the MoE branch against a literal transcription of
-`transformers`' `Gemma4TextDecoderLayer` / `Gemma4TextRouter` / `Gemma4TextExperts` on
-the real layer-0 weights - fp32 relative error **3.5e-7**, identical top-8 expert
-selection, bit-identical routing weights. That gates the **authoring** (block wiring,
-the fused `gate_up_proj` split, the GELU gate, router maths). It does not gate the
-exported graph, the int4 rounding, or either Metal kernel.
-
-Two Metal kernels run in this graph - `gather_qmm` (`aff4`) for the experts and the
-flash-decode SDPA for the full-attention layers - and **they have never been combined in
-any model before**. If decode produces garbage, re-export without `--metal-sdpa` first:
-it is an optimisation, the MoE kernel is not, and that isolates the fault in one run.
-
-Decode-only (`main`, S=1). There is no prefill function: the flash-decode SDPA kernel is
-structurally q=1 with no causal mask, and `MetalSwitchGLU` is likewise decode-only, so a
-prefill variant needs two new kernels rather than a flag.
-
-## Licence
-
-Derivative of a Gemma model; the **Gemma Terms of Use**, the Gemma Prohibited Use Policy
-and the Gemma licence apply.
+| `coreai_models/models/macos/gemma4_moe_text.py` | `Gemma4MoeConfig`, `Gemma4MoeRouter`, `GeluGLU`, `Gemma4MoeDecoderLayer`, `Gemma4MoeForCausalLM`, and a loader that splits the checkpoint's fused `gate_up_proj [E, 2*704, 2816]` into the `SwitchGLU`'s separate stacks |
+| `coreai_models/models/macos/gemma4_moe_pipelined.py` | `Gemma4MoePipelinedForCausalLM` — subclasses the dense pipelined core, overrides one method. The attention half needed nothing: the unified KV pair, the head-count repeat, the head_dim zero-pad and the two-state engine contract are all inherited |
+| `coreai-model-zoo/conversion/export_gemma4_moe_decode_pipelined.py` | the recipe |
+
+## Contents
+
+| Path | Bytes | Files | Manifest context | Functions |
+|---|---:|---|---|---|
+| `gpu-pipelined/gemma4_26b_a4b_qat_decode_int4linsym_moeaff4_msdpa_g8` | 17,580,059,414 | 8 | 16384 | main (decode only) |
+
+| File | Bytes |
+|---|---:|
+| `…aimodel/main.mlirb` | 17,547,866,346 |
+| `…aimodel/main.hash` | 32 |
+| `…aimodel/metadata.json` | 105 |
+| `metadata.json` (bundle manifest) | 689 |
+| `tokenizer/tokenizer.json` | 32,169,626 |
+| `tokenizer/chat_template.jinja` | 18,683 |
+| `tokenizer/tokenizer_config.json` | 3,730 |
+| `tokenizer/generation_config.json` | 203 |
+
+Context was set to 16384 once, from the start, because `--max-ctx` does not change the
+exported graph — it sets `language.max_context_length` in the manifest and nothing else.
+Lower it if the machine cannot afford the KV; that is a metadata edit, not a re-export.
+
+### Decode only — and a prefill variant is not deliverable
+
+There is no `prefill` function, and one cannot be added by a flag. Two independent kernel
+blockers:
+
+1. `--metal-sdpa`'s flash-decode kernel is structurally q=1 with no causal mask. The full
+   attention layers need that kernel for the same scratch-heap reason the dense 12B/31B do,
+   so this model inherits the blocker verbatim.
+2. `MetalSwitchGLU.forward` is itself decode-only — it asserts a token batch of 1 and
+   expands the single activation row across the routed slots. A `BatchedMetalSwitchGLU`
+   (sort-by-expert grouped GEMM) does exist for q>1, so this half is *reachable*, but it has
+   never been exported for any model.
+
+A prefill variant therefore needs a new flash-**prefill** Metal kernel plus the first export
+of the batched MoE kernel, plus a numerics gate for each. That is kernel work, not a
+conversion job. Prompts are prefilled one token at a time.
+
+## Requirements
+
+- **Apple silicon Mac with ≥24–32 GB unified memory**, Core AI runtime. The 16 GB tier is
+  excluded and not narrowly.
+- **Engine contract: 2 inputs** (`input_ids`, `position_ids`) → logits, one growing KV pair,
+  no static inputs and no per-step mask.
+- **States:** `keyCache` / `valueCache` `Float16, 30 × 1 × 8 × ? × 512` — one growing pair,
+  30 slots (no KV sharing), read straight off the exported bundle. The dynamic sequence dim
+  means the runtime resolves `GrowingKVCache`, not a static allocation at the manifest
+  maximum.
+- **KV cost: 491,520 bytes per token of context** (fp16):
+
+  | Context | KV |
+  |---|---|
+  | 4096 | 2.01 GB |
+  | 8192 | 4.03 GB |
+  | **16384** | **8.05 GB** |
+  | 32768 | 16.1 GB |
+
+- **Resident weights, ~17.6 GB:**
+
+  | | bytes |
+  |---|---|
+  | Routed experts, `aff4` (incl. `_kpad256` zero columns) | ~14.7 GB |
+  | Attention + dense MLP + untied head, int4 block-32 | ~1.4 GB |
+  | `embed_tokens`, fp16 in-graph | ~1.5 GB |
+  | Router, fp16 | ~0.02 GB |
+  | **Total** | **~17.6 GB** |
+  | + KV at 4096 | ~19.6 GB |
+  | + KV at 16384 | ~25.7 GB |
+
+  | Tier | Verdict |
+  |---|---|
+  | 16 GB | **no** — weights alone are ~1.6× the entire ~10.7 GB Metal working set |
+  | 24–32 GB | yes at ctx ≤ 8192 (~21.6 GB); 16384 is tight |
+  | 64 GB | yes, at any context |
+
+  ⚠️ **Do not turn this table into a hard memory floor in a model catalog.** It is
+  arithmetic from state shapes and weight formats — a prediction to check, not a measured
+  tier claim. The residency mechanism for MoE bundles on macOS (mmap'd vs wired expert
+  slabs) has **never been measured for any MoE bundle**, and the low end of the range above
+  is deliberately conservative because of that. The first machine that loads this model
+  should produce the measurement before anything depends on it.
+
+  25 of the 30 layers have `head_dim` 256 zero-padded to 512, and all 30 slots grow linearly
+  even though 25 of them only ever attend a 1024-token window. A ring-buffered sliding cache
+  would be roughly 5× cheaper per token here. That is model authoring, not a flag.
+
+## Measured performance
+
+**None. On any machine.** No token has been generated from this bundle. Every figure in
+this card is measured on Linux (export cost, byte counts) or derived from state shapes and
+weight formats; **none is a runtime measurement**, and no Mac-side run is scheduled.
+
+For contrast, every measured number elsewhere in this catalog comes from a 10-row guided
+enrichment harness on a 16 GB M2 Pro Mac. This model cannot load on that machine, so there
+is not a slower number here — there is no number.
+
+The intended argument for this model is speed at comparable size against the dense 31B —
+17.6 GB vs 20 GB, but ~4B active parameters per token read through a gather kernel instead
+of 31B dense — and quality per token against the 12B at ~2.1× the footprint. The nearest
+measured analogue is the LFM2.5 8B-A1B at 140 tok/s through its gather kernel versus 39
+tok/s for the same weights read as a dense over-read. **Neither claim is measured for this
+bundle and neither is scheduled to be.**
+
+## Qualification status
+
+| Artifact | Status |
+|---|---|
+| `gpu-pipelined/gemma4_26b_a4b_qat_decode_int4linsym_moeaff4_msdpa_g8` | **UNQUALIFIED — community tier.** Never measured on target hardware; no Mac-side qualification scheduled. What *is* established: the checkpoint is the QAT source, the block wiring is faithful to `transformers` at 3.5e-7 relative error on real layer-0 weights, the graph exports, and the producer fingerprint passes. That is the whole of it. |
+
+What a qualifying machine would have to run, in priority order:
+
+1. **A decode oracle** — generate and compare against an fp32 HF reference. This is the gate
+   that catches an int4 or kernel-level error which the authoring gate below cannot see.
+2. **The two Metal kernels have never run together.** `gather_qmm` (the `aff4` variant) and
+   the flash-decode SDPA have never been in the same graph, on any model. If decode produces
+   garbage, re-export without `--metal-sdpa` first — it is an optimisation, the MoE kernel is
+   not — and that isolates which kernel is at fault in one run.
+3. **Measure residency properly**, both numbers: `/usr/bin/time -l` (footprint = dirty;
+   maximum resident = including mmap) and `vmmap --summary <pid>` during generation. This is
+   the first MoE bundle Visible has built and the tier map has been waiting on exactly this.
+4. **Compare tok/s against the dense 12B and 31B** on the same machine. The entire case for
+   this model is that ~4B active params through the gather kernel beats 12B dense on
+   quality-per-second. If it does not, the bundle has no slot.
+5. **Router sanity at depth.** 128 experts / top-8 with a learned `per_expert_scale` is far
+   sparser routing than the LFM 8B-A1B's 32/top-4. Check expert-utilisation spread on a real
+   prompt set before trusting long-form output.
+
+## Verification
+
+### The oracle gate — what it proves, and what it does not
+
+`transformers` 4.57.6 (the pinned export environment) has no `gemma4` at all, so there was
+no importable reference on the conversion box. Rather than install `transformers` 5.x beside
+a pinned toolchain, the gate transcribes `Gemma4TextDecoderLayer.forward` (the MoE-block
+half), `Gemma4TextRouter.forward` and `Gemma4TextExperts.forward` literally from upstream
+and runs both implementations on the **real layer-0 weights** in fp32:
+
+```
+T=1: max|dFFN|=7.629e-05  rel=3.453e-07  |  same top-8 ids: True  max|dw|=0.000e+00
+T=4: max|dFFN|=9.537e-05  rel=2.735e-07  |  same top-8 ids: True  max|dw|=0.000e+00
+ORACLE PASS (fp32, real layer-0 weights)
+```
+
+Identical expert selection, bit-identical routing weights, ~3e-7 relative error on the block
+output. **That gates the authoring** — the wiring, the fused-weight split, the activation
+choice, the norm placement, the router maths. It does **not** gate the exported graph, the
+int4 rounding, or either Metal kernel. The attention half was deliberately not re-gated: it
+is the dense `Attention` class already shipped in the 12B and 31B bundles.
+
+### Everything else
+
+- **Producer fingerprint:** the inner `<name>.aimodel/metadata.json` reads
+  `producer: "coreai-core 1.0.0b2"` — PASS. `coreai-torch 0.4.0` / `coreai-core 1.0.0b1`
+  produce ODIE-poisoned bundles that abort **in-process** at load, taking the host
+  application down uncatchably. The field is in the inner asset metadata; the bundle
+  manifest never carries a `producer` field.
+- **Stop token:** `eos_token = "<turn|>"` (id 106). Gemma 4 ends a *turn* with `<turn|>`; the
+  raw upstream `<eos>` ends the whole sequence and a host stopping on it overruns every
+  reply. Unlike the E2B/E4B/12B/31B bundles, where this correction was applied by hand after
+  export, **this recipe applies it in the export script itself.**
+- **`chat_template.jinja` is present** (18,683 B). The E2B/E4B export scripts miss it because
+  their `allow_patterns` only fetch `.json` files; this script fetches and copies it
+  explicitly and *raises* if it is absent.
+- **Determinism:** this bundle was exported twice from the same checkpoint with identical
+  flags. The two runs differ by **28 bytes** (17,580,059,442 vs 17,580,059,414) and their
+  peak RSS by 164 kB. Conversion is not byte-deterministic, so integrity rests on per-file
+  SHA-256 of the artifact actually published, never on rebuilding.
+- **Not wired into the app.** There is no Core AI model-id case for this model, and adding
+  one means an enum case, a descriptor, a unified-id mapping and a catalog variant — four
+  places, where the descriptor lookup force-unwraps, so a half-done addition crashes rather
+  than degrades. It should not be wired in before gates 1–3 above pass; an unqualified entry
+  in an automatic model-selection path is worse than no entry.
+
+### A note on why this repo is public
+
+This bundle was converted before it could be published: the `visible-cx` organisation was at
+its private-storage cap and the LFS push returned
+`403 Private repository storage limit reached`. Of the three ways out — publish public, raise
+the plan, or delete a pinned artifact to make room — publishing publicly under the Gemma
+terms is how Google itself distributes Gemma derivatives, and that decision was taken by the
+project owner rather than by a conversion run. Until it was, the bundle existed on exactly
+one machine, and recreating it costs a 50 GB download plus ~25 minutes on a 247 GB box at
+102 GB peak.
+
+## License
+
+The upstream QAT checkpoint carries Apache-2.0 metadata and is ungated, but it is a Gemma
+model and **the Gemma terms apply downstream**, which is why this repo declares
+`license: gemma`. Use of these weights is subject to the
+[Gemma Terms of Use](https://ai.google.dev/gemma/terms), the
+[Gemma Prohibited Use Policy](https://ai.google.dev/gemma/prohibited_use_policy) and the
+[Gemma 4 license](https://ai.google.dev/gemma/docs/gemma_4_license). Those obligations travel
+with any redistribution of this bundle. Visible's contribution is the port, the recipe and
+the honest statement of what has and has not been verified — not the weights.