Instructions to use mlboydaisuke/MiniCPM5-2B-LiteRT with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- LiteRT-LM
How to use mlboydaisuke/MiniCPM5-2B-LiteRT with LiteRT-LM:
# LiteRT-LM runs on various platforms (Android, iOS, Windows, Linux, macOS, IoT, Web/WASM) # and supports many APIs (C++, Python, Kotlin, Swift, JavaScript, Flutter). # For platform-specific integration guides, please refer to the official developer website: # https://ai.google.dev/edge/litert-lm # To try LiteRT-LM, the easiest way is to use our CLI tool. # 1. Install the LiteRT-LM CLI tool: pip install -U litert-lm # 2. Download and run this model locally: # See: https://ai.google.dev/edge/litert-lm/cli # A single .litertlm file in the repo is picked automatically; otherwise the CLI asks which one to run # (or pass its name right after the repo id). litert-lm run \ --from-huggingface-repo=mlboydaisuke/MiniCPM5-2B-LiteRT \ --prompt="Write me a poem"
- LiteRT
How to use mlboydaisuke/MiniCPM5-2B-LiteRT with LiteRT:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Notebooks
- Google Colab
- Kaggle
Add litertlm_manifest.json (measured rows, verified backends, known issues)
Browse files- litertlm_manifest.json +289 -0
litertlm_manifest.json
ADDED
|
@@ -0,0 +1,289 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"manifest_schema": "0.1.2",
|
| 3 |
+
"repo": "mlboydaisuke/MiniCPM5-2B-LiteRT",
|
| 4 |
+
"generated": "2026-09-08",
|
| 5 |
+
"generator": "make_manifest.py",
|
| 6 |
+
"model": {
|
| 7 |
+
"display_name": "MiniCPM5-2B",
|
| 8 |
+
"base_model": "openbmb/MiniCPM5-2B",
|
| 9 |
+
"architecture": "Dense LlamaForCausalLM (42L, hidden 2048, GQA 16:2, untied 130560-vocab embeddings, rope_theta 5e6, no rope scaling), 2.52B params; hybrid-reasoning chat template (enable_thinking) carried verbatim with a declared thought channel",
|
| 10 |
+
"parameters_b": 2.52,
|
| 11 |
+
"license": "apache-2.0",
|
| 12 |
+
"context_length": 4096,
|
| 13 |
+
"capabilities": {
|
| 14 |
+
"vision": false,
|
| 15 |
+
"audio": false,
|
| 16 |
+
"thinking": {
|
| 17 |
+
"declared": true,
|
| 18 |
+
"channel": {
|
| 19 |
+
"start": "<think>",
|
| 20 |
+
"end": "</think>"
|
| 21 |
+
}
|
| 22 |
+
},
|
| 23 |
+
"channels": [
|
| 24 |
+
{
|
| 25 |
+
"name": "thought",
|
| 26 |
+
"start": "<think>",
|
| 27 |
+
"end": "</think>"
|
| 28 |
+
}
|
| 29 |
+
]
|
| 30 |
+
}
|
| 31 |
+
},
|
| 32 |
+
"variants": [
|
| 33 |
+
{
|
| 34 |
+
"file": "MiniCPM5-2B_int4.litertlm",
|
| 35 |
+
"sha256": "9858563beafbc6d5e0d25fcee3827541515296a9302ed3d088b16a58d4fbe7b8",
|
| 36 |
+
"size_bytes": 1553670064,
|
| 37 |
+
"sections": [
|
| 38 |
+
{
|
| 39 |
+
"type": "LlmMetadataProto",
|
| 40 |
+
"size_bytes": 9346
|
| 41 |
+
},
|
| 42 |
+
{
|
| 43 |
+
"type": "HF_Tokenizer_Zlib",
|
| 44 |
+
"size_bytes": 2048362
|
| 45 |
+
},
|
| 46 |
+
{
|
| 47 |
+
"type": "TFLiteModel",
|
| 48 |
+
"size_bytes": 1281048576,
|
| 49 |
+
"model_type": "tf_lite_prefill_decode"
|
| 50 |
+
},
|
| 51 |
+
{
|
| 52 |
+
"type": "TFLiteModel",
|
| 53 |
+
"size_bytes": 270524336,
|
| 54 |
+
"model_type": "tf_lite_embedder"
|
| 55 |
+
}
|
| 56 |
+
],
|
| 57 |
+
"quantization": "blockwise-32 int4 + OCTAV on linears, int8 embedding (externalized embedder section); 1,664 all-zero-block scales (13 dead layer-0 MLP rows) set to an epsilon in place after export; GPU activations at the runtime default (fp16)",
|
| 58 |
+
"backends": [
|
| 59 |
+
"cpu",
|
| 60 |
+
"gpu"
|
| 61 |
+
],
|
| 62 |
+
"default_backend": "gpu",
|
| 63 |
+
"recommended": [
|
| 64 |
+
{
|
| 65 |
+
"platform": "android",
|
| 66 |
+
"device_class": "flagship",
|
| 67 |
+
"backend": "gpu",
|
| 68 |
+
"reason": "measured on one Galaxy S26 with one binary (2026-09-08, 205-token prompt, 2 runs per backend, cooled below 42 C before each run): gpu prefill 400.7-411.1 tok/s vs cpu 39.0-72.0, TTFT 0.56 s vs 2.9-5.3 s, decode 16.1-18.6 vs 15.6-15.8 tok/s (a modest 1.0-1.2x), peak VmHWM 1.13-1.14 GB vs 1.96-2.12 GB; full LITERT_CL delegation (1873/1873 nodes on every subgraph) and the gate answer verified through the thought channel. The gpu wins on time-to-first-token and memory; decode is close to a wash"
|
| 69 |
+
},
|
| 70 |
+
{
|
| 71 |
+
"platform": "macos",
|
| 72 |
+
"backend": "gpu",
|
| 73 |
+
"reason": "Apple M4 Max, litert-lm 0.17.0 benchmark -p 256 -d 256 --runs 3 --cache no, generation gated per backend first: gpu 1699 tok/s prefill / 92.8 tok/s decode vs cpu 149 / 31.1"
|
| 74 |
+
},
|
| 75 |
+
{
|
| 76 |
+
"platform": "ios",
|
| 77 |
+
"backend": "gpu",
|
| 78 |
+
"reason": "iPhone 17 Pro, 8-question on-device gate at a 2048-token budget: Metal gpu 7/8 (init 5.7 s) and cpu 7/8 (init 2.2 s), both G41_DONE; the one miss on each leg is the rhyme line of the composite prompt, which the bf16 model also misses with thinking off. No speed rows were taken on the phone"
|
| 79 |
+
}
|
| 80 |
+
],
|
| 81 |
+
"requirements": {
|
| 82 |
+
"peak_ram_mb": 2122,
|
| 83 |
+
"platform_notes": [
|
| 84 |
+
"Requires litert-lm >= 0.16 (thought channel + ThinkingConfig); measured on 0.17.0",
|
| 85 |
+
"Hybrid-reasoning model: with no ThinkingConfig it thinks before every answer; budget max output tokens >= 2048 (4096 for math). enable_thinking=false (ThinkingConfig or conversation extra_context) switches to direct answers",
|
| 86 |
+
"Thinking-mode caveat measured on this file: on GSM8K-class questions its reasoning chains run ~4x longer than int8's and do not close inside 3584 tokens on the CPU reference (0/10 closed; int8 9/10, bf16 9/10). Use int8 when the reasoning must complete, or turn thinking off",
|
| 87 |
+
"iPhone 17 Pro: passes the on-device gate on gpu and cpu (init 5.7 s / 2.2 s); the int8 file does not fit default-entitlement iOS (2.33 GB main section)",
|
| 88 |
+
"The 13 all-zero MLP rows in decoder layer 0 make raw blockwise-int4 exports fail to load on XNNPACK (scale 0); this file carries the epsilon fix"
|
| 89 |
+
]
|
| 90 |
+
},
|
| 91 |
+
"measured": [
|
| 92 |
+
{
|
| 93 |
+
"device": "Apple M4 Max",
|
| 94 |
+
"os": "macOS",
|
| 95 |
+
"backend": "cpu",
|
| 96 |
+
"runtime": "litert-lm 0.17.0",
|
| 97 |
+
"prompt_tokens": 256,
|
| 98 |
+
"decode_tokens": 256,
|
| 99 |
+
"prefill_tps": 148.58,
|
| 100 |
+
"decode_tps": 31.12,
|
| 101 |
+
"ttft_s": 1.76,
|
| 102 |
+
"load_s": 4.52,
|
| 103 |
+
"max_num_tokens": 1024,
|
| 104 |
+
"cache": "no",
|
| 105 |
+
"runs": 3,
|
| 106 |
+
"date": "2026-09-08",
|
| 107 |
+
"source": "litert-lm benchmark -p 256 -d 256 --runs 3 --cache no --max-num-tokens 1024, quiet machine, backend gated on a real generation first"
|
| 108 |
+
},
|
| 109 |
+
{
|
| 110 |
+
"device": "Apple M4 Max",
|
| 111 |
+
"os": "macOS",
|
| 112 |
+
"backend": "gpu",
|
| 113 |
+
"runtime": "litert-lm 0.17.0",
|
| 114 |
+
"prompt_tokens": 256,
|
| 115 |
+
"decode_tokens": 256,
|
| 116 |
+
"prefill_tps": 1699.15,
|
| 117 |
+
"decode_tps": 92.79,
|
| 118 |
+
"ttft_s": 0.16,
|
| 119 |
+
"load_s": 3.65,
|
| 120 |
+
"max_num_tokens": 1024,
|
| 121 |
+
"cache": "no",
|
| 122 |
+
"runs": 3,
|
| 123 |
+
"date": "2026-09-08",
|
| 124 |
+
"source": "litert-lm benchmark -p 256 -d 256 --runs 3 --cache no --max-num-tokens 1024, >=300 s rest before the GPU cell, backend gated on a real generation first"
|
| 125 |
+
},
|
| 126 |
+
{
|
| 127 |
+
"device": "Galaxy S26 (SM-S942Q, Qualcomm SM8850, Adreno)",
|
| 128 |
+
"backend": "gpu",
|
| 129 |
+
"runtime": "litert_lm_advanced_main v0.16.0, LiteRT CL delegate",
|
| 130 |
+
"prompt_tokens": 205,
|
| 131 |
+
"decode_tokens": 1021,
|
| 132 |
+
"prefill_tps": "400.7-411.1",
|
| 133 |
+
"decode_tps": "16.14-18.56",
|
| 134 |
+
"ttft_s": 0.56,
|
| 135 |
+
"load_s": 11.2,
|
| 136 |
+
"peak_memory_mb": 1142,
|
| 137 |
+
"cache": "no",
|
| 138 |
+
"runs": 2,
|
| 139 |
+
"date": "2026-09-08",
|
| 140 |
+
"source": "205-token prompt with --benchmark, 2 runs (engine init 11.2 / 13.1 s), device cooled below 42 C before each run; GPU gate full LITERT_CL delegation (1873/1873 nodes on every subgraph, 0 blocking XNNPack), generation verified on the gate prompt; the cpu row of the same date is the same-device control"
|
| 141 |
+
},
|
| 142 |
+
{
|
| 143 |
+
"device": "Galaxy S26 (SM-S942Q, Qualcomm SM8850, Adreno)",
|
| 144 |
+
"backend": "cpu",
|
| 145 |
+
"runtime": "litert_lm_advanced_main v0.16.0 (XNNPACK CPU)",
|
| 146 |
+
"prompt_tokens": 205,
|
| 147 |
+
"decode_tokens": 1021,
|
| 148 |
+
"prefill_tps": "39.0-72.0",
|
| 149 |
+
"decode_tps": "15.56-15.79",
|
| 150 |
+
"ttft_s": "2.9-5.3",
|
| 151 |
+
"load_s": 5.7,
|
| 152 |
+
"peak_memory_mb": 2122,
|
| 153 |
+
"cache": "no",
|
| 154 |
+
"runs": 2,
|
| 155 |
+
"date": "2026-09-08",
|
| 156 |
+
"source": "same-device CPU control: identical binary, prompt file and flags, only --backend differs; 205-token prompt with --benchmark, 2 runs (engine init 5.7 / 3.1 s), cooled before each run"
|
| 157 |
+
}
|
| 158 |
+
],
|
| 159 |
+
"known_issues": [
|
| 160 |
+
"Thinking mode: reasoning chains are ~4x longer than int8/bf16 on GSM8K-class questions and frequently hit a 3584-token budget without emitting the answer (CPU reference 0/10 closed; the Metal GPU's default fp16 activations happen to close 22/30). Quantization damage of int4 on this 42-layer model, not a runtime bug; block-128 int4 was tried and is worse (GSM8K 79 vs 87)"
|
| 161 |
+
]
|
| 162 |
+
},
|
| 163 |
+
{
|
| 164 |
+
"file": "MiniCPM5-2B_int8.litertlm",
|
| 165 |
+
"sha256": "58da116a6eb563185eb5e8d0b1f1e1116b632a516cadc0285b6da35b9f886a86",
|
| 166 |
+
"size_bytes": 2603278256,
|
| 167 |
+
"sections": [
|
| 168 |
+
{
|
| 169 |
+
"type": "LlmMetadataProto",
|
| 170 |
+
"size_bytes": 9346
|
| 171 |
+
},
|
| 172 |
+
{
|
| 173 |
+
"type": "HF_Tokenizer_Zlib",
|
| 174 |
+
"size_bytes": 2048362
|
| 175 |
+
},
|
| 176 |
+
{
|
| 177 |
+
"type": "TFLiteModel",
|
| 178 |
+
"size_bytes": 2330645824,
|
| 179 |
+
"model_type": "tf_lite_prefill_decode"
|
| 180 |
+
},
|
| 181 |
+
{
|
| 182 |
+
"type": "TFLiteModel",
|
| 183 |
+
"size_bytes": 270524336,
|
| 184 |
+
"model_type": "tf_lite_embedder"
|
| 185 |
+
}
|
| 186 |
+
],
|
| 187 |
+
"quantization": "dynamic int8 on linears + embedding (externalized embedder section); prefer_activation_type = fp32 declared in the bundle for the GPU executor",
|
| 188 |
+
"backends": [
|
| 189 |
+
"cpu",
|
| 190 |
+
"gpu"
|
| 191 |
+
],
|
| 192 |
+
"default_backend": "gpu",
|
| 193 |
+
"recommended": [
|
| 194 |
+
{
|
| 195 |
+
"platform": "android",
|
| 196 |
+
"device_class": "flagship",
|
| 197 |
+
"backend": "gpu",
|
| 198 |
+
"reason": "measured on one Galaxy S26 with one binary (2026-09-08, 205-token prompt, 2 runs per backend, cooled before each run): gpu prefill 150.2-159.5 tok/s vs cpu 102.8-157.2, decode 10.86-12.75 vs 11.67-11.75 tok/s (a wash), TTFT 1.4 s vs 1.4-2.1 s, peak VmHWM 0.94-1.10 GB vs 2.90 GB; full LITERT_CL delegation and the gate answer verified through the thought channel. With fp32 activations declared the gpu no longer wins decode; it is recommended for the 2.6x lower process memory"
|
| 199 |
+
},
|
| 200 |
+
{
|
| 201 |
+
"platform": "macos",
|
| 202 |
+
"backend": "gpu",
|
| 203 |
+
"reason": "Apple M4 Max, litert-lm 0.17.0 benchmark -p 256 -d 256 --runs 3 --cache no, generation gated per backend first: gpu 1405 tok/s prefill / 74.7 tok/s decode vs cpu 161 / 30.0"
|
| 204 |
+
}
|
| 205 |
+
],
|
| 206 |
+
"requirements": {
|
| 207 |
+
"peak_ram_mb": 2896,
|
| 208 |
+
"platform_notes": [
|
| 209 |
+
"Requires litert-lm >= 0.16 (thought channel + ThinkingConfig); measured on 0.17.0",
|
| 210 |
+
"Hybrid-reasoning model: with no ThinkingConfig it thinks before every answer; budget max output tokens >= 2048 (4096 for math). enable_thinking=false switches to direct answers",
|
| 211 |
+
"fp32 activations are declared in-bundle: with the runtime's default fp16 GPU activations this model's reasoning on one gate question ran 2000+ tokens without closing </think> (7/8, degenerate); with fp32 it closes in ~450 tokens and passes 8/8, at ~14 % GPU decode cost (Mac). The declaration is a measured trade for the gate, not a general cure",
|
| 212 |
+
"Not for default-entitlement iOS apps: the main weight section is 2.33 GB, above the ~2 GiB single-section mmap limit",
|
| 213 |
+
"GSM8K (thinking off, n=100, greedy): 91 vs bf16 92 - at parity; with thinking on it reproduces the bf16 model question for question on a 10-question subset (9/10 chains close, ~3.2k characters)"
|
| 214 |
+
]
|
| 215 |
+
},
|
| 216 |
+
"measured": [
|
| 217 |
+
{
|
| 218 |
+
"device": "Apple M4 Max",
|
| 219 |
+
"os": "macOS",
|
| 220 |
+
"backend": "cpu",
|
| 221 |
+
"runtime": "litert-lm 0.17.0",
|
| 222 |
+
"prompt_tokens": 256,
|
| 223 |
+
"decode_tokens": 256,
|
| 224 |
+
"prefill_tps": 160.87,
|
| 225 |
+
"decode_tps": 30.02,
|
| 226 |
+
"ttft_s": 1.62,
|
| 227 |
+
"load_s": 15.03,
|
| 228 |
+
"max_num_tokens": 1024,
|
| 229 |
+
"cache": "no",
|
| 230 |
+
"runs": 3,
|
| 231 |
+
"date": "2026-09-08",
|
| 232 |
+
"source": "litert-lm benchmark -p 256 -d 256 --runs 3 --cache no --max-num-tokens 1024, quiet machine, backend gated on a real generation first"
|
| 233 |
+
},
|
| 234 |
+
{
|
| 235 |
+
"device": "Apple M4 Max",
|
| 236 |
+
"os": "macOS",
|
| 237 |
+
"backend": "gpu",
|
| 238 |
+
"runtime": "litert-lm 0.17.0",
|
| 239 |
+
"prompt_tokens": 256,
|
| 240 |
+
"decode_tokens": 256,
|
| 241 |
+
"prefill_tps": 1405.16,
|
| 242 |
+
"decode_tps": 74.65,
|
| 243 |
+
"ttft_s": 0.2,
|
| 244 |
+
"load_s": 3.0,
|
| 245 |
+
"max_num_tokens": 1024,
|
| 246 |
+
"cache": "no",
|
| 247 |
+
"runs": 3,
|
| 248 |
+
"date": "2026-09-08",
|
| 249 |
+
"source": "litert-lm benchmark -p 256 -d 256 --runs 3 --cache no --max-num-tokens 1024, >=300 s rest before the GPU cell, backend gated on a real generation first"
|
| 250 |
+
},
|
| 251 |
+
{
|
| 252 |
+
"device": "Galaxy S26 (SM-S942Q, Qualcomm SM8850, Adreno)",
|
| 253 |
+
"backend": "gpu",
|
| 254 |
+
"runtime": "litert_lm_advanced_main v0.16.0, LiteRT CL delegate",
|
| 255 |
+
"prompt_tokens": 205,
|
| 256 |
+
"decode_tokens": 1021,
|
| 257 |
+
"prefill_tps": "150.2-159.5",
|
| 258 |
+
"decode_tps": "10.86-12.75",
|
| 259 |
+
"ttft_s": "1.35-1.44",
|
| 260 |
+
"load_s": 6.3,
|
| 261 |
+
"peak_memory_mb": 1098,
|
| 262 |
+
"cache": "no",
|
| 263 |
+
"runs": 2,
|
| 264 |
+
"date": "2026-09-08",
|
| 265 |
+
"source": "205-token prompt with --benchmark, 2 runs, device cooled below 42 C before each run; GPU gate full LITERT_CL delegation (1873/1873 nodes on every subgraph), generation verified on the gate prompt through the thought channel (engine init 6.3 / 4.0 s); the cpu row of the same date is the same-device control"
|
| 266 |
+
},
|
| 267 |
+
{
|
| 268 |
+
"device": "Galaxy S26 (SM-S942Q, Qualcomm SM8850, Adreno)",
|
| 269 |
+
"backend": "cpu",
|
| 270 |
+
"runtime": "litert_lm_advanced_main v0.16.0 (XNNPACK CPU)",
|
| 271 |
+
"prompt_tokens": 205,
|
| 272 |
+
"decode_tokens": 1021,
|
| 273 |
+
"prefill_tps": "102.8-157.2",
|
| 274 |
+
"decode_tps": "11.67-11.75",
|
| 275 |
+
"ttft_s": "1.4-2.1",
|
| 276 |
+
"load_s": 0.27,
|
| 277 |
+
"peak_memory_mb": 2896,
|
| 278 |
+
"cache": "no",
|
| 279 |
+
"runs": 2,
|
| 280 |
+
"date": "2026-09-08",
|
| 281 |
+
"source": "same-device CPU control: identical binary, prompt file and flags, only --backend differs; 205-token prompt with --benchmark, 2 runs (engine init 0.27 / 0.30 s), cooled before each run"
|
| 282 |
+
}
|
| 283 |
+
],
|
| 284 |
+
"known_issues": [
|
| 285 |
+
"GPU with the runtime's default fp16 activations (i.e. if the in-bundle fp32 declaration is overridden): one 8-question gate item's reasoning ran 2000+ tokens without closing </think>. Keep the declared fp32 activations for this file"
|
| 286 |
+
]
|
| 287 |
+
}
|
| 288 |
+
]
|
| 289 |
+
}
|