Re-export int8 per-block-32: the per-channel bundle had dead LM-head rows from vocab id ~65024 up (never emitted <|im_end|>)

README.md

@@ -20,12 +20,19 @@ Core AI is Apple's on-device ML runtime in iOS 27 / macOS 27 and the successor t
 This model has no row on [DeviceMark](https://devicemark.github.io/), the on-device LLM leaderboard.
 <!-- gen-cards:devicemark end -->
 
-# MiniCPM5-1B — Core AI (int8, runs on iPhone)
+# MiniCPM5-1B — Core AI (int8 block-32, runs on iPhone)
 
 Apple **Core AI** (`.aimodel`) conversion of [openbmb/MiniCPM5-1B](https://huggingface.co/openbmb/MiniCPM5-1B) —
 OpenBMB's 1.08B on-device LLM with **hybrid Think / No-Think reasoning** and **128K** context, reaching
 1B-class open-source SOTA. Runs fully on-device on **iPhone** and Apple Silicon Macs (GPU, pipelined engine).
 
+> **Revision note (2026-09-09).** This revision replaces the per-channel int8 bundle published as
+> `5ad650f`. That bundle's LM head had **dead rows from vocab id ~65024 up** — every token there
+> scored ~0 in the engine, `<|im_end|>` (130073) included — so a chat turn **never halted** (it ran
+> to the token cap) and any answer needing a high-id token lost it. The card's "halts cleanly" line
+> was wrong. Details and the measurements are under *Why per-block-32* below; pin this revision or
+> newer.
+
 Part of the community Core AI model zoo: **https://github.com/john-rocky/coreai-model-zoo**
 
 <!-- gen-cards:use-it begin id=minicpm5-1b (managed by scripts/gen-cards — edit cards.json / QuickStart.swift, not this block) -->
@@ -79,29 +86,62 @@ conversation history; `streamResponse(to:)` yields tokens as they decode.
 - Measure in Release — Debug is ~3× slower on per-token host work
 <!-- gen-cards:use-it end -->
 
-## On-device numbers (iPhone 17 Pro, A19 Pro)
-
-Measured with the zoo's `PipelinedBench` (random 128-token prompt, greedy):
-
-| | decode | prefill | quality | size | engine-ready |
-|---|---:|---:|---|---:|---:|
-| **`int8/`** (ship) | **66.8 tok/s** | 68.0 tok/s | **lossless** (24/24 token-exact vs HF fp32) | **1.0 GB** | 2.0 s |
-
-`int8` is **~2.2× faster than fp16** on iPhone (decode is memory-bandwidth-bound, so halving the
-weight read ≈ doubles throughput) at **no quality cost** — the device greedy output is token-for-token
-identical to the fp32 reference on the benchmark prompts. So int8 strictly dominates fp16 here.
+## Measured
+
+| | decode | prefill | numerics | size |
+|---|---:|---:|---|---:|
+| **iPhone 17 Pro** (A19 Pro, `PipelinedBench`, Release) | **61.7 tok/s** | 65.6 tok/s | **24/24 token-exact** vs HF fp32 on the margin-clean alphabet prompt (min fp32 top-2 margin 0.841) **+ 6/6 including the stop** on the no-think turn `1+1=?` (`1+1=2` then `<|im_end|>`); engine ready 7.3 s cold | **1.1 GB** |
+| **M4 Max** (macOS 27, `llm-benchmark`, 512p/1024g) | **246.6 tok/s** | 6649 tok/s | **16/16 token-exact** vs the fp32 oracle (margin-aware gate, min margin 0.913) + the same 6/6 stop gate | |
+
+Halt check through the engine on the same phone-shaped bundle: the Think-mode turn `1+1=?` stops on
+its own after 171 tokens (cap 400); the no-think turn after 6.
+
+⚠️ **iPhone context cap: prompt + generated tokens must stay under 1024.** The bundle declares a 131072
+dynamic KV, and the shipped `CoreAIPipelinedEngine` caps iOS growing-KV capacity at 1024 (its guard
+against the iOS compiler miscompiling growing-KV specializations at seq ≥ 2048). Chunk or trim the
+history on iOS; macOS has no cap.
+
+## Why per-block-32 (what was wrong with the previous revision)
+
+The previous bundle (`5ad650f`, int8 **per-channel** absmax) passed a 24-token greedy parity check and
+still never ended a chat turn. Teacher-forcing it through the engine against the fp32 reference
+located the defect in the LM head, by vocab id:
+
+| token (id) | fp32 P | per-channel bundle | per-block-32 bundle (this revision) |
+|---|---:|---:|---:|
+| `.lineTo` (65023) | 0.978 | 0.993 | — |
+| `粒子` (65039) | 0.977 | **0.0000** | 0.98 |
+| ` chromosomal` (65528) | 0.383 | **0.0000** | 0.380 |
+| ` OpenAI` (130051) | 0.929 | **0.0000** | 0.923 |
+| `\n\n` after `</think>` (130063) | 0.9999 | **0.0000** | 0.9999 |
+| `<|im_end|>` after `1+1=2` (130073) | 0.873 | **0.0000** (top-1 `.`) | 0.867 |
+
+Every probed row at id ≥ 65039 is dead in the per-channel bundle and every probed row ≤ 65023 is
+healthy (16 probes; the boundary lies in that 16-id window, which contains 65024 = 127 × 512). Rows
+below it match fp32 to ~0.01 in probability, which is why free-running English prose looked fine.
+A fresh per-channel export on the same toolchain (coreai-torch 0.4.1 / coreai-opt 0.2.1 /
+coreai-core 1.0.0b2) reproduces the shipped bundle's logits to four decimals, so this is a property
+of the per-channel int8 path for this 130560-row head, not a one-off; per-block-32 (this bundle),
+the CLI's default int4 preset and fp16 (`--compression none`) are all clean on the same probes.
+Which component owns the row cut-off (quantizer, converter or the runtime's per-channel int8 matmul)
+is not established here.
+
+The gate that catches it is now part of the zoo's `cli/coreai_verify.py`: `--chat no-think --prompt
+"1+1=?"` makes the fp32 oracle's `<|im_end|>` a gated step — a bundle that runs past a stop the oracle
+takes at margin 0.80 fails — and `--must-stop-within N` is the plain halt check. The old bundle is red
+on both, this one green.
 
 ## Quantization
 
-Weight-only **symmetric per-channel int8** (absmax, no clipping — clipping craters the 130k-vocab LM
-head; absmax keeps it lossless), applied as a torch pre-export pass via `coreai-opt`; SDPA / RoPE /
-RMSNorm stay full precision. Same recipe family as the zoo's proven `sym8`.
+Weight-only **symmetric int8, per-block-32** (a scale per 32-wide block along the input dim; no
+clipping), applied as a torch pre-export pass via `coreai-opt`; SDPA / RoPE / RMSNorm stay full
+precision — the same YAML that ships the [2B](https://huggingface.co/mlboydaisuke/MiniCPM5-2B-CoreAI).
 
 ```bash
 uv run coreai.llm.export openbmb/MiniCPM5-1B --experimental --compute-precision float16 \
-  --compression-config minicpm5_int8sym.yaml
-# minicpm5_int8sym.yaml: quantization_config → op_state_spec.weight = {dtype: int8,
-#   qscheme: symmetric, granularity: {type: per_channel, axis: 0}}
+  --compression-config minicpm5_int8sym_b32.yaml
+# minicpm5_int8sym_b32.yaml: quantization_config → op_state_spec.weight = {dtype: int8,
+#   qscheme: symmetric, granularity: {type: per_block, block_size: 32}}
 ```
 
 ## Conversion notes
@@ -110,10 +150,14 @@ uv run coreai.llm.export openbmb/MiniCPM5-1B --experimental --compute-precision
   `llama` graph family, but Mistral's builder is architecturally identical for this config (GQA,
   no qkv bias, no qk-norm, explicit `head_dim` honored). One-line remap in the model registry.
 - **Chat EOS.** Base `eos_token` is `</s>`, but the chat template ends turns with `<|im_end|>`
-  (id 130073). The bundle's tokenizer `eos_token` is set to `<|im_end|>` (as Qwen ships) so
-  generation halts cleanly.
+  (id 130073). The bundle's tokenizer `eos_token` is set to `<|im_end|>` (as Qwen ships). Checked
+  through the engine with the chat template applied: the no-think turn `1+1=?` answers `1+1=2` and
+  stops at step 5, where the fp32 reference stops (margin 0.80).
 - **Dynamic-shape bundle** → the Core AI pipelined engine (the iPhone path); a static iOS export
   routes to the static-shape engine instead, which this FM-format bundle doesn't target.
+- **Thinking.** The model thinks by default (`<think>…</think>` before the answer); pass
+  `enable_thinking=False` through the chat template for a direct answer. Give generation a
+  generous budget (the kit caps at 4096) — the think trace alone can run a few hundred tokens.
 
 ## Run
 
@@ -126,6 +170,20 @@ let session = LanguageModelSession(model: model)
 print(try await session.respond(to: "Explain on-device AI in one sentence."))
 ```
 
+## Reproduce
+
+Exporter, gate, card and port notes live in the
+[Core AI model zoo](https://github.com/john-rocky/coreai-model-zoo):
+[`models/minicpm5-1b/`](https://github.com/john-rocky/coreai-model-zoo/tree/main/models/minicpm5-1b),
+[`conversion/export_minicpm5.py`](https://github.com/john-rocky/coreai-model-zoo/blob/main/conversion/export_minicpm5.py),
+[`knowledge/minicpm5-1b.md`](https://github.com/john-rocky/coreai-model-zoo/blob/main/knowledge/minicpm5-1b.md).
+
+```bash
+python3 conversion/zoo_convert.py show minicpm5-1b
+python3 conversion/zoo_convert.py run  minicpm5-1b
+python3 cli/coreai_verify.py <bundle> --chat no-think --prompt "1+1=?" -n 16 --must-stop-within 16
+```
+
 ## License
 
 Apache-2.0 (upstream MiniCPM5 license). Model © OpenBMB — see

int8/metadata.json

@@ -20,9 +20,9 @@
     "model_definition": "torch",
     "hf_model_id": "openbmb/MiniCPM5-1B"
   },
-  "compression": "_minicpm5_int8sym",
+  "compression": "minicpm5_int8sym_b32",
   "compilation": {
-    "date": "2026-06-26T17:39:31.612088+09:00",
+    "date": "2026-09-09T22:31:10.485907+09:00",
     "targets": []
   }
-}
+}

int8/minicpm5_1b_int8.aimodel/main.hash

@@ -1 +1 @@
-�$�Q���xw+�}��+������INr�6���1K
+c���$��˰�_��� B��JW�,_��$ܯ

int8/minicpm5_1b_int8.aimodel/main.mlirb

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:bd24d951f785c00478772ba07d9bc52ba184f9fd8c9b494e72c136a1bbe3314b
-size 1082313882
+oid sha256:6380a69f24a0b714cbb0fd5f94dfe8204203e6cf054a578b2c5fc097241edcaf
+size 1148614014

int8/minicpm5_1b_int8.aimodel/metadata.json

@@ -1,8 +1,8 @@
 {
+  "license" : "Apache-2.0",
   "author" : "OpenBMB",
-  "creationDate" : "20260720T175201Z",
   "assetVersion" : "2.0",
-  "license" : "Apache-2.0",
-  "producer" : "coreai-core 1.0.0b2",
-  "description" : "MiniCPM5-1B is a 1.08B-parameter on-device causal language model from OpenBMB, with hybrid (Think \/ No-Think) reasoning and 128K context, reaching 1B-class open-source SOTA. Source: https:\/\/huggingface.co\/openbmb\/MiniCPM5-1B"
+  "description" : "MiniCPM5-1B is a 1.08B-parameter on-device causal language model from OpenBMB, with hybrid (Think \/ No-Think) reasoning and 128K context, reaching 1B-class open-source SOTA. Source: https:\/\/huggingface.co\/openbmb\/MiniCPM5-1B",
+  "creationDate" : "20260909T133108Z",
+  "producer" : "coreai-core 1.0.0b2"
 }