mlboydaisuke commited on
Commit
07df106
·
verified ·
1 Parent(s): 3b8cb71

Card and manifest: the peaks of the 2026-08/09 phone runs in MiB (VmHWM in kB ÷ 1024, rounded down), the values of both runs, wording fixes

Browse files
Files changed (2) hide show
  1. README.md +3 -3
  2. litertlm_manifest.json +10 -12
README.md CHANGED
@@ -86,9 +86,9 @@ Two honest caveats, both measured rather than assumed:
86
 
87
  Init is the `Init time` the CLI prints; that value includes both `Init Total` and the phases inside it, so it counts engine creation twice ([LiteRT-LM #4057](https://github.com/google-ai-edge/LiteRT-LM/issues/4057)).
88
 
89
- Pixel 8a (Tensor G3), `litert_lm_advanced_main` from the v0.16.0 kit, 205-token prompt, `--benchmark`:
90
 
91
- | Backend | Prefill (205) | Decode | TTFT |
92
  |---|---|---|---|
93
  | GPU (OpenCL, LiteRT `CompiledModel` / `LITERT_CL`) | 434 tok/s | 12.3 tok/s | 0.67 s |
94
  | CPU (XNNPACK) | 40 tok/s | 5.8 tok/s | 6.5 s |
@@ -117,7 +117,7 @@ Note that Gallery builds current as of 2026-08 offer **no accelerator choice for
117
  Converted with stock [`litert-torch`](https://github.com/google-ai-edge/litert-torch) 0.9.3 — no patches to the exporter. Reproduction script: [hf-to-litertlm `s1mini_work/`](https://github.com/john-rocky/hf-to-litertlm). Two things were resolved at conversion time, and both matter to anyone converting this model themselves:
118
 
119
  - **The chat template is replaced, because the model requires a flag no runtime can pass.** S1-mini inherits Qwen3's template, whose default renders thinking mode; the upstream card requires `enable_thinking=False` and warns that the thinking render usually produces no usable output at all. A LiteRT-LM engine applies the embedded template with no such argument, so shipping the vendor template verbatim would ship a broken model. The bundled template hardcodes the non-thinking render (the assistant turn opens with an empty `<think>` block) and bakes in the exact system prompt the model was trained with, so clients that cannot send a system message still get the trained format. Before export, the replacement was verified to render byte-identically to the vendor template under `enable_thinking=False` across all twelve card examples, with and without an explicit system message; a second turn's render is a string-extension of the first turn plus its reply, which is what LiteRT-LM's incremental conversation rendering requires.
120
- - **The exporter's composite stop strings are removed, because they eat the final punctuation.** litert-torch expands the template's turn suffix into punctuation-prefixed variants (`".<|im_end|>\n"`, `"?<|im_end|>\n"`, …) to work around SentencePiece greedy merging. Qwen3 is BPE, that merge never happens, but the runtime still matches the multi-token stop and strips the *whole* match — so a reply ending `…Thursday."` comes back as `…Thursday`. Measured cleanly: 7 of 7 test cases whose reference ends in "." lost exactly that period, and rewriting the metadata to keep only the two real stop token ids flipped all 7 back. On a punctuation-restoration model this is fatal, and a keyword-matching quality gate cannot see it.
121
 
122
  Also worth knowing: this checkpoint ties its embedding and `lm_head`, so an int4 recipe that asks for int4 on the linears and int8 on the embedding describes one tensor two ways, and the quantizer resolves that by copying the vocabulary table per prefill signature — an "int4" file that came out *larger* than the int8 built the same way (656 MB against 613 MB) until the embedder was externalized.
123
 
 
86
 
87
  Init is the `Init time` the CLI prints; that value includes both `Init Total` and the phases inside it, so it counts engine creation twice ([LiteRT-LM #4057](https://github.com/google-ai-edge/LiteRT-LM/issues/4057)).
88
 
89
+ Pixel 8a (Tensor G3), `litert_lm_advanced_main` from the v0.16.0 kit, 254-token prompt, `--benchmark`:
90
 
91
+ | Backend | Prefill (254) | Decode | TTFT |
92
  |---|---|---|---|
93
  | GPU (OpenCL, LiteRT `CompiledModel` / `LITERT_CL`) | 434 tok/s | 12.3 tok/s | 0.67 s |
94
  | CPU (XNNPACK) | 40 tok/s | 5.8 tok/s | 6.5 s |
 
117
  Converted with stock [`litert-torch`](https://github.com/google-ai-edge/litert-torch) 0.9.3 — no patches to the exporter. Reproduction script: [hf-to-litertlm `s1mini_work/`](https://github.com/john-rocky/hf-to-litertlm). Two things were resolved at conversion time, and both matter to anyone converting this model themselves:
118
 
119
  - **The chat template is replaced, because the model requires a flag no runtime can pass.** S1-mini inherits Qwen3's template, whose default renders thinking mode; the upstream card requires `enable_thinking=False` and warns that the thinking render usually produces no usable output at all. A LiteRT-LM engine applies the embedded template with no such argument, so shipping the vendor template verbatim would ship a broken model. The bundled template hardcodes the non-thinking render (the assistant turn opens with an empty `<think>` block) and bakes in the exact system prompt the model was trained with, so clients that cannot send a system message still get the trained format. Before export, the replacement was verified to render byte-identically to the vendor template under `enable_thinking=False` across all twelve card examples, with and without an explicit system message; a second turn's render is a string-extension of the first turn plus its reply, which is what LiteRT-LM's incremental conversation rendering requires.
120
+ - **The exporter's composite stop strings are removed, because they eat the final punctuation.** litert-torch expands the template's turn suffix into punctuation-prefixed variants (`".<|im_end|>\n"`, `"?<|im_end|>\n"`, …) to work around SentencePiece greedy merging. Qwen3 is BPE, that merge never happens, but the runtime still matches the multi-token stop and strips the *whole* match — so a reply ending `…Thursday."` comes back as `…Thursday`. Measured cleanly: 7 of 7 test cases whose reference ends in "." lost exactly that period, and rewriting the metadata to keep only the two real stop token ids flipped all 7 back. On a punctuation-restoration model this is fatal, and a keyword-matching quality check cannot see it.
121
 
122
  Also worth knowing: this checkpoint ties its embedding and `lm_head`, so an int4 recipe that asks for int4 on the linears and int8 on the embedding describes one tensor two ways, and the quantizer resolves that by copying the vocabulary table per prefill signature — an "int4" file that came out *larger* than the int8 built the same way (656 MB against 613 MB) until the embedder was externalized.
123
 
litertlm_manifest.json CHANGED
@@ -1,7 +1,7 @@
1
  {
2
  "manifest_schema": "0.1.3",
3
  "repo": "mlboydaisuke/S1-mini-LiteRT",
4
- "generated": "2026-11-14",
5
  "generator": "make_manifest.py",
6
  "model": {
7
  "display_name": "S1-mini (LiteRT-LM)",
@@ -64,13 +64,13 @@
64
  "platform": "android",
65
  "device_class": "midrange",
66
  "backend": "cpu",
67
- "reason": "measured on one Pixel 8a with one binary (2026-08-25, 205-token prompt, single run each): GPU 434 tok/s prefill / 12.3 decode against CPU 40.4 / 5.8 - the GPU is faster, but on this phone it deterministically drops a date separator on number-dense input ('March 3rd, 2026.' -> 'March 32026.') while the CPU output is byte-identical to macOS on all 12 test cases. For a text-normalization model exactness is the job, so CPU is the default here; choose gpu when time-to-first-token matters more (0.67 s vs 6.46 s). Same-device pair taken, so the speed ordering is measured, not assumed"
68
  },
69
  {
70
  "platform": "android",
71
  "device_class": "flagship",
72
  "backend": "gpu",
73
- "reason": "measured on one Galaxy S26 with one binary (2026-09-05, 205-token prompt, 2 cold runs per backend, cooled before each run): gpu decodes 25.13-25.8 tok/s against cpu's 12.29 (2.1x), prefill 940.7-1027.5 vs 166.7-175.3 tok/s, peak 1323 vs 2129 MB, engine init 6.3 vs 2.2 s. Same-device CPU control taken - a measured win for gpu on decode. On the Pixel 8a's Mali GPU this model deterministically dropped a date separator on number-dense input ('March 3rd, 2026.' -> 'March 32026.'); the same date vector run on this Adreno GPU came back intact ('The invoice came to $23,450 and it's due on March 3, 2026.', probe 2026-09-05, same binary), so that Mali defect does not reproduce here. The full 12-case exactness check was not repeated on Adreno; where byte-exactness is the job, cpu remains the conservative choice"
74
  }
75
  ],
76
  "requirements": {
@@ -118,28 +118,28 @@
118
  "os": "Android",
119
  "backend": "gpu",
120
  "runtime": "litert_lm_advanced_main v0.16.0 kit (LiteRT CompiledModel, LITERT_CL)",
121
- "prompt_tokens": 205,
122
  "prefill_tps": 434,
123
  "decode_tps": 12.3,
124
  "ttft_s": 0.67,
125
  "max_num_tokens": 4096,
126
  "runs": 1,
127
  "date": "2026-08-25",
128
- "source": "on-device benchmark run, 205-token prompt"
129
  },
130
  {
131
  "device": "Pixel 8a",
132
  "os": "Android",
133
  "backend": "cpu",
134
  "runtime": "litert_lm_advanced_main v0.16.0 kit (XNNPACK)",
135
- "prompt_tokens": 205,
136
  "prefill_tps": 40.4,
137
  "decode_tps": 5.8,
138
  "ttft_s": 6.46,
139
  "max_num_tokens": 4096,
140
  "runs": 1,
141
  "date": "2026-08-25",
142
- "source": "on-device benchmark run, 205-token prompt"
143
  },
144
  {
145
  "device": "iPhone 17 Pro",
@@ -180,11 +180,10 @@
180
  "decode_tps": "25.13-25.8",
181
  "ttft_s": "0.29-0.31",
182
  "load_s": 6.3,
183
- "peak_memory_mb": 1323,
184
  "cache": "no",
185
  "runs": 2,
186
  "date": "2026-09-05",
187
- "source": "205-token prompt with --benchmark, 2 cold runs (compile/weight caches deleted between runs, device cooled below 42 C before each run); GPU gate full LITERT_CL delegation (15474 ops across 12 subgraphs), generation verified on the gate prompt; the cpu row of the same date is the same-device control"
188
  },
189
  {
190
  "device": "Galaxy S26 (SM-S942Q, Qualcomm SM8850, Adreno)",
@@ -193,14 +192,13 @@
193
  "prompt_tokens": 254,
194
  "decode_tokens": 195,
195
  "prefill_tps": "166.7-175.3",
196
- "decode_tps": 12.29,
197
  "ttft_s": "1.53-1.61",
198
  "load_s": 2.2,
199
- "peak_memory_mb": 2129,
200
  "cache": "no",
201
  "runs": 2,
202
  "date": "2026-09-05",
203
- "source": "same-device CPU control - identical binary, prompt file and flags, only --backend differs; 205-token prompt with --benchmark, 2 cold runs (compile/weight caches deleted between runs, device cooled below 42 C before each run); same-device control for the gpu row of the same date"
204
  }
205
  ],
206
  "known_issues": [
 
1
  {
2
  "manifest_schema": "0.1.3",
3
  "repo": "mlboydaisuke/S1-mini-LiteRT",
4
+ "generated": "2026-11-16",
5
  "generator": "make_manifest.py",
6
  "model": {
7
  "display_name": "S1-mini (LiteRT-LM)",
 
64
  "platform": "android",
65
  "device_class": "midrange",
66
  "backend": "cpu",
67
+ "reason": "measured on one Pixel 8a with one binary (2026-08-25, 254-token prompt, single run each): GPU 434 tok/s prefill / 12.3 decode against CPU 40.4 / 5.8 - the GPU is faster, but on this phone it deterministically drops a date separator on number-dense input ('March 3rd, 2026.' -> 'March 32026.') while the CPU output is byte-identical to macOS on all 12 test cases. For a text-normalization model exactness is the job, so CPU is the default here; choose gpu when time-to-first-token matters more (0.67 s vs 6.46 s). Same-device pair taken, so the speed ordering is measured, not assumed"
68
  },
69
  {
70
  "platform": "android",
71
  "device_class": "flagship",
72
  "backend": "gpu",
73
+ "reason": "measured on one Galaxy S26 with one binary (2026-09-05, 254-token prompt, 2 cold runs per backend, cooled before each run): gpu decodes 25.13-25.8 tok/s against cpu's 12.23-12.34 (2.1x), prefill 940.7-1027.5 vs 166.7-175.3 tok/s, peak 1322-1323 vs 2126-2129 MiB, engine init 6.3-8.2 vs 2.2-2.4 s. Same-device CPU control taken - a measured win for gpu on decode. On the Pixel 8a's Mali GPU this model deterministically dropped a date separator on number-dense input ('March 3rd, 2026.' -> 'March 32026.'); the same input run on this Adreno GPU kept the separator but did not match the Pixel 8a CPU output ('The invoice came to $23,450 and it's due on March 3, 2026.' in a separate run on 2026-09-05 with the same binary, against 'The invoice came to $23,450, and it's due on March 3rd, 2026.' on the Pixel 8a CPU), so the dropped separator does not reproduce here. The full 12-case exactness check was not repeated on Adreno; where byte-exactness is the job, cpu remains the conservative choice"
74
  }
75
  ],
76
  "requirements": {
 
118
  "os": "Android",
119
  "backend": "gpu",
120
  "runtime": "litert_lm_advanced_main v0.16.0 kit (LiteRT CompiledModel, LITERT_CL)",
121
+ "prompt_tokens": 254,
122
  "prefill_tps": 434,
123
  "decode_tps": 12.3,
124
  "ttft_s": 0.67,
125
  "max_num_tokens": 4096,
126
  "runs": 1,
127
  "date": "2026-08-25",
128
+ "source": "on-device benchmark run, 254-token prompt"
129
  },
130
  {
131
  "device": "Pixel 8a",
132
  "os": "Android",
133
  "backend": "cpu",
134
  "runtime": "litert_lm_advanced_main v0.16.0 kit (XNNPACK)",
135
+ "prompt_tokens": 254,
136
  "prefill_tps": 40.4,
137
  "decode_tps": 5.8,
138
  "ttft_s": 6.46,
139
  "max_num_tokens": 4096,
140
  "runs": 1,
141
  "date": "2026-08-25",
142
+ "source": "on-device benchmark run, 254-token prompt"
143
  },
144
  {
145
  "device": "iPhone 17 Pro",
 
180
  "decode_tps": "25.13-25.8",
181
  "ttft_s": "0.29-0.31",
182
  "load_s": 6.3,
 
183
  "cache": "no",
184
  "runs": 2,
185
  "date": "2026-09-05",
186
+ "source": "254-token prompt with --benchmark, 2 cold runs (compile/weight caches deleted between runs, device cooled below 42 C before each run); peaks 1323 and 1322 MiB in the two runs (VmHWM in kB ÷ 1024, rounded down); engine init 6.3 and 8.2 s in the two runs (load_s is the first run's); full delegation (15474 ops across 12 subgraphs); a separate run on the question \"What is the capital of France?\" returned the question unchanged (this model rewrites text and does not answer questions); the cpu row of the same date is the same-device control"
187
  },
188
  {
189
  "device": "Galaxy S26 (SM-S942Q, Qualcomm SM8850, Adreno)",
 
192
  "prompt_tokens": 254,
193
  "decode_tokens": 195,
194
  "prefill_tps": "166.7-175.3",
195
+ "decode_tps": "12.23-12.34",
196
  "ttft_s": "1.53-1.61",
197
  "load_s": 2.2,
 
198
  "cache": "no",
199
  "runs": 2,
200
  "date": "2026-09-05",
201
+ "source": "same-device CPU control - identical binary, prompt file and flags, only --backend differs; 254-token prompt with --benchmark, 2 cold runs (compile/weight caches deleted between runs, device cooled below 42 C before each run); peaks 2129 and 2126 MiB in the two runs (VmHWM in kB ÷ 1024, rounded down); engine init 2.2 and 2.4 s in the two runs (load_s is the first run's); same-device control for the gpu row of the same date"
202
  }
203
  ],
204
  "known_issues": [