mlboydaisuke commited on
Commit
9e0df43
·
verified ·
1 Parent(s): 0d3aca6

Prefill/decode graph in the GPU composite form (weights unchanged); card and manifest updated with the 2026-11-17 Galaxy S26 and Pixel 11 Pro runs

Browse files
README.md CHANGED
@@ -25,12 +25,14 @@ tags:
25
  base_model_relation: quantized
26
  ---
27
 
28
- Measured on device (edge-compat): Galaxy S26 · LiteRT-LM 0.16.1 · GPU · decode 36.1 tok/s · prefill 653 tok/s · TTFT 430 ms · all 1243 ops delegated (2026-09-04); Mac Studio M4 Max · LiteRT-LM 0.16.0 · GPU · decode 138.6 tok/s · prefill 2045 tok/s · TTFT 134 ms (2026-09-04); Galaxy S26 · LiteRT-LM 0.16.1 · CPU · decode 35.4 tok/s · prefill 163 tok/s · TTFT 1.65 s (2026-09-04). Record: https://github.com/john-rocky/edge-compat/blob/main/cards/vibevoice-asr-bitnet/CARD.md
29
 
30
  # VibeVoice-ASR-BitNet — LiteRT-LM
31
 
32
  [microsoft/VibeVoice-ASR-BitNet](https://huggingface.co/microsoft/VibeVoice-ASR-BitNet) converted to the **LiteRT-LM** (`.litertlm`) format for on-device speech recognition with Google's [LiteRT-LM](https://github.com/google-ai-edge/LiteRT-LM) runtime. **Audio in, text out, in one bundle**: the runtime decodes the clip, runs the bundled audio encoder, and the LLM transcribes — no host-side feature extraction.
33
 
 
 
34
  VibeVoice-ASR-BitNet is Microsoft's edge variant of VibeVoice-ASR: the same σ-VAE acoustic + semantic tokenizers (24 kHz → 7.5 latent frames/s) feeding a **Qwen2.5-1.5B-shaped language model trained with BitNet b1.58 ternary weights**, multilingual (en, zh, fr, it, ko, pt, vi), MIT-licensed. This is the first audio-in `.litertlm` in this collection.
35
 
36
  | File | Recipe | Size |
@@ -57,11 +59,13 @@ The Android CPU row differs from the Mac by one clip ("On the general principles
57
 
58
  Sending the clip **without** the duration sentence works (the bundle's template then sends "Please transcribe it.") at 3.57 % on the same set — include the duration for best results.
59
 
 
 
60
  ## Usage
61
 
62
- The audio goes in as an audio content item; any container/sample rate the runtime's decoder reads (wav, mp3, flac — it resamples to 24 kHz). The **audio encoder must run on the CPU** (`audio_backend`), the LM may run on CPU or GPU.
63
 
64
- Python (`pip install litert-lm-api`; this block was run as written on 0.16.1 and 0.17.1):
65
 
66
  ```python
67
  import litert_lm
@@ -93,6 +97,10 @@ conv.close()
93
  --input_prompt='[audio:/path/clip.wav] This is a 5.86 seconds audio, please transcribe it.'
94
  ```
95
 
 
 
 
 
96
  Notes:
97
 
98
  - Greedy decoding (top_k 1) is what the vendor runtime does; the model emits plain text and stops on `<|im_end|>`.
@@ -101,6 +109,8 @@ Notes:
101
 
102
  ## Performance
103
 
 
 
104
  `litert-lm benchmark` (litert-lm 0.16.0), Apple M4 Max, `-p 256 -d 256 --runs 3 --cache no`, text prompt (LM only — the audio encoder is a fixed cost per 30 s window: 1.3–1.7 s on the M4 Max CPU at 8 threads), quiet machine, ≥ 300 s rest before the GPU reading:
105
 
106
  | Backend | Prefill (256) | Decode | TTFT | Init |
@@ -121,6 +131,41 @@ Galaxy S26 (SM-S942Q, Snapdragon SM8850, Adreno), `litert_lm_advanced_main` buil
121
 
122
  On the phone a 5–20 s clip transcribes in roughly 7–10 s of wall-clock from a cold process (CPU), including engine load; peak private footprint 3.2 GB (CPU) / 2.3 GB (LM on GPU).
123
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
124
  ## Conversion notes
125
 
126
  Converted with `litert-torch` 0.9.3/0.9.4, ai-edge-quantizer 0.9.0, litert-lm-builder 0.16.1, transformers 5.14.1 (native `vibevoice_asr`). Scripts and the full recipe: [hf-to-litertlm](https://github.com/john-rocky/hf-to-litertlm) `vibevoice_asr_work/`.
@@ -131,12 +176,26 @@ Converted with `litert-torch` 0.9.3/0.9.4, ai-edge-quantizer 0.9.0, litert-lm-bu
131
  - **Prompt**: the vendor's system prompt is baked into the bundle template; the user turn renders as `<|object_ref_start|>` + audio embeddings + `<|object_ref_end|>` + newline + your text.
132
  - **GPU**: the LM runs on Metal (macOS) and on the Android OpenCL delegate; the audio encoder is CPU-only — on Metal/WebGPU the 30 s window exceeds the delegate's 65 535-workgroup dispatch limit, and on Adreno the fp16 delegate path returns an empty transcript. Keep `audio_backend` on CPU.
133
 
 
 
 
 
 
 
 
 
 
 
134
  ## Known runtime issues
135
 
136
  **External weights (since 2026-11-09).** In `VibeVoice-ASR-BitNet.litertlm`, the weights sit in TFLiteWeights sections at the end of the file (LiteRT's external weights) instead of inside the model graph. On Android, the GPU path needs LiteRT-LM 0.19.0 or later: the GPU engine of the v0.18.0 and v0.18.1 Android builds crashes (SIGSEGV) on such a file ([LiteRT-LM#3986](https://github.com/google-ai-edge/LiteRT-LM/issues/3986)). The earlier file with the weights inside the graph stays at revision [`193b2d5`](https://huggingface.co/litert-community/VibeVoice-ASR-BitNet/tree/193b2d579a786206286ac36e74b6ea513562bdf5).
137
 
138
  From a conversation preface (the system message, initial messages, tools and extra context a conversation is created with) that has tools and no messages, LiteRT-LM renders this file's chat template with an empty message list, and the template reads `messages[0]` without checking that the list is non-empty, so the render fails with `Failed to apply template: undefined value`. This happens in two places: when the preface is prefilled at creation (`prefillPrefaceOnInit = true` in Kotlin, `prefillPrefaceOnInit: true` on the web, `SetPrefillPrefaceOnInit(true)` in C++; the default is false), where creating the conversation fails with that error, and in the preface render (`renderPrefaceIntoString()` in Kotlin and Swift, `RenderPrefaceIntoString()` in C++, `litert_lm_conversation_render_preface_to_string` in the C API). The same happens with extra context (such as `enable_thinking`) and no messages, and in the preface render with an empty preface (run in the runtime on one Qwen3 template; the other templates rendered with minijinja 2.14 and jinja2). Read in the source of LiteRT-LM 0.17.0, 0.18.1 and main; measured with LiteRT-LM 0.18.1 (the litert-lm package from PyPI) through its C API on this file's template text. The default path, where the first message is rendered together with the preface, is not affected: the Python API and the CLI of 0.18.1 expose neither the flag nor the preface render, and Google AI Edge Gallery sets neither (its source as read on 2026-10-25). A system message or any initial message in the conversation config avoids it.
139
 
 
 
 
 
140
  ## License and changes
141
 
142
  Distributed under the **MIT License** (inherited from the base model; see `LICENSE`). **Changes from the original work:** weights ternarized (LM) and converted from safetensors fp32 to LiteRT flatbuffers with the quantization described above; the VAE decoder (synthesis half) is not included; three special tokens added to the tokenizer; chat/prompt template and audio-preprocessing parameters embedded as LiteRT-LM metadata.
 
25
  base_model_relation: quantized
26
  ---
27
 
28
+ Measured on device with an earlier file (edge-compat): Galaxy S26 · LiteRT-LM 0.16.1 · GPU · decode 36.1 tok/s · prefill 653 tok/s · TTFT 430 ms · all 1243 ops delegated (2026-09-04); Mac Studio M4 Max · LiteRT-LM 0.16.0 · GPU · decode 138.6 tok/s · prefill 2045 tok/s · TTFT 134 ms (2026-09-04); Galaxy S26 · LiteRT-LM 0.16.1 · CPU · decode 35.4 tok/s · prefill 163 tok/s · TTFT 1.65 s (2026-09-04). Record: https://github.com/john-rocky/edge-compat/blob/main/cards/vibevoice-asr-bitnet/CARD.md
29
 
30
  # VibeVoice-ASR-BitNet — LiteRT-LM
31
 
32
  [microsoft/VibeVoice-ASR-BitNet](https://huggingface.co/microsoft/VibeVoice-ASR-BitNet) converted to the **LiteRT-LM** (`.litertlm`) format for on-device speech recognition with Google's [LiteRT-LM](https://github.com/google-ai-edge/LiteRT-LM) runtime. **Audio in, text out, in one bundle**: the runtime decodes the clip, runs the bundled audio encoder, and the LLM transcribes — no host-side feature extraction.
33
 
34
+ On 2026-11-17 the file in this repo was replaced with a file whose prefill/decode graph is rewritten for the GPU, with every weight unchanged. On a Galaxy S26 GPU the new file decodes 51.2 tokens/s after a 1,024-token prefill, against 46.7 tokens/s for the file it replaced (LiteRT-LM v0.19.0; see the end of Performance). The new file was run only with LiteRT-LM 0.19.0 to 0.19.2; earlier runtimes were not tried with it.
35
+
36
  VibeVoice-ASR-BitNet is Microsoft's edge variant of VibeVoice-ASR: the same σ-VAE acoustic + semantic tokenizers (24 kHz → 7.5 latent frames/s) feeding a **Qwen2.5-1.5B-shaped language model trained with BitNet b1.58 ternary weights**, multilingual (en, zh, fr, it, ko, pt, vi), MIT-licensed. This is the first audio-in `.litertlm` in this collection.
37
 
38
  | File | Recipe | Size |
 
59
 
60
  Sending the clip **without** the duration sentence works (the bundle's template then sends "Please transcribe it.") at 3.57 % on the same set — include the duration for best results.
61
 
62
+ The table and the paragraphs above were measured on earlier files. The new file was compared with the file it replaced on the same 20 clips on a Mac with litert-lm 0.19.2 (2026-11-16, greedy decoding, the duration sentence in the prompt, audio encoder on the CPU). On the CPU with 1 thread, the new file gave the same transcript as the previous file for all 20 clips, byte for byte (18/448 word errors, 4.02 %, with both files). With the LM on the Mac GPU, the two files also gave the same transcripts for all 20 clips (11/448, 2.46 %, with both files). The CPU count of 18/448 is higher than the 12/448 of the table's Mac CPU row (LiteRT-LM 0.16.1). Both files give it, and no run on this card shows which difference between the two measurements accounts for it.
63
+
64
  ## Usage
65
 
66
+ The audio goes in as an audio content item; any container/sample rate the runtime's decoder reads (wav, mp3, flac — it resamples to 24 kHz). The **audio encoder must run on the CPU** (`audio_backend`), the LM may run on CPU or GPU. On Android the GPU was the faster of the two with the new file on a Galaxy S26 (numbers at the end of Performance); the two blocks below run the LM on the CPU.
67
 
68
+ Python (`pip install litert-lm-api`; this block was run as written on 0.16.1 and 0.17.1 with earlier files):
69
 
70
  ```python
71
  import litert_lm
 
97
  --input_prompt='[audio:/path/clip.wav] This is a 5.86 seconds audio, please transcribe it.'
98
  ```
99
 
100
+ If you download this file with the `litert-lm` command-line tool (PyPI `litert-lm`, not the `litert_lm_advanced_main` program) and it stops right after the download with no error message of its own (a bus error; exit status 138 on a Mac) or with an error log that contains `too large for file size`, the downloaded file may be incomplete. The litert-lm CLI (0.14.0 to 0.19.2, and litert-lm-nightly 0.20.0.dev20261112) can keep an incomplete download in `~/.litert-lm/cache/huggingface/litert-community/VibeVoice-ASR-BitNet/` and uses it again on later runs without checking its size. Delete that file and run the command again (after an `import`, run the `import` again). This was measured with other models' files, not with this one.
101
+
102
+ If you downloaded this file with the litert-lm CLI's `--from-huggingface-repo` before it was replaced on 2026-11-17 (Japan time), delete it from `~/.litert-lm/cache/huggingface/litert-community/VibeVoice-ASR-BitNet/`, then run your command again (after an `import`, run the `import` again). Without deleting it, the litert-lm CLI (0.14.0 to 0.19.2) keeps running the previous file: it uses a file it finds there without checking it against this repo. The previous file stays at revision [`0d3aca6`](https://huggingface.co/litert-community/VibeVoice-ASR-BitNet/tree/0d3aca6fff0167fdce76fbc0c9e89d20421a7f05).
103
+
104
  Notes:
105
 
106
  - Greedy decoding (top_k 1) is what the vendor runtime does; the model emits plain text and stops on `<|im_end|>`.
 
109
 
110
  ## Performance
111
 
112
+ Everything in this section above its last part was measured on earlier files (the tables on 2026-09-04). The new file's speeds were measured on a Galaxy S26 (GPU and CPU) and on a Pixel 11 Pro (GPU); they are in the last part of this section. This card has no Mac speed for the new file.
113
+
114
  `litert-lm benchmark` (litert-lm 0.16.0), Apple M4 Max, `-p 256 -d 256 --runs 3 --cache no`, text prompt (LM only — the audio encoder is a fixed cost per 30 s window: 1.3–1.7 s on the M4 Max CPU at 8 threads), quiet machine, ≥ 300 s rest before the GPU reading:
115
 
116
  | Backend | Prefill (256) | Decode | TTFT | Init |
 
131
 
132
  On the phone a 5–20 s clip transcribes in roughly 7–10 s of wall-clock from a cold process (CPU), including engine load; peak private footprint 3.2 GB (CPU) / 2.3 GB (LM on GPU).
133
 
134
+ ### Galaxy S26 and Pixel 11 Pro: the file of 2026-11-17
135
+
136
+ Each run prefills 1,024 tokens and generates 256 tokens, three times in one process (three iterations), with a text prompt (LM only) and a context of 2,048 tokens. The program is `litert_lm_advanced_main` built from the LiteRT-LM v0.19.0 source (commit `771b06e1`; the tag's own commit after it changes only `Package.swift` and `README.md`), run with `--backend=gpu` or `--backend=cpu` and `--async=false --benchmark --benchmark_prefill_tokens=1024 --benchmark_decode_tokens=256 --num_iterations=3 --max_num_tokens=2048` on 2026-11-17. The CPU runs used the program's default of 4 threads. Each set of four runs (one set per phone on the GPU, two sets on the Galaxy S26 CPU) went in the order previous, new, new, previous. Each run is a new process. Each value is one run's mean of iterations 2–3, so it leaves out iteration 1, the first pass after the engine starts.
137
+
138
+ GPU:
139
+
140
+ | Phone | Prefill, tok/s (previous, new, new, previous) | Decode, tok/s (previous, new, new, previous) | New / previous, runs 2 and 1 | New / previous, runs 3 and 4 |
141
+ |---|---|---|---|---|
142
+ | Galaxy S26 | 689.25, 710.56, 712.12, 686.85 | 46.85, 51.08, 51.34, 46.59 | prefill ×1.031, decode ×1.090 | prefill ×1.037, decode ×1.102 |
143
+ | Pixel 11 Pro | 587.37, 610.56, 605.76, 590.42 | 13.87, 15.50, 16.38, 14.42 | prefill ×1.039, decode ×1.118 | prefill ×1.026, decode ×1.136 |
144
+
145
+ Galaxy S26 CPU (XNNPACK), two sets of four runs:
146
+
147
+ | Set | Prefill, tok/s (previous, new, new, previous) | Decode, tok/s (previous, new, new, previous) | New / previous, runs 2 and 1 | New / previous, runs 3 and 4 |
148
+ |---|---|---|---|---|
149
+ | first | 186.24, 184.84, 189.38, 195.21 | 35.80, 35.55, 36.05, 36.92 | prefill ×0.992, decode ×0.993 | prefill ×0.970, decode ×0.977 |
150
+ | second | 183.99, 185.72, 196.04, 200.15 | 35.99, 36.46, 37.03, 36.87 | prefill ×1.009, decode ×1.013 | prefill ×0.979, decode ×1.004 |
151
+
152
+ The mean of each file's two runs:
153
+
154
+ | Phone and backend | New file: prefill / decode / time to first token | Previous file: prefill / decode / time to first token |
155
+ |---|---|---|
156
+ | Galaxy S26, GPU | 711.34 tok/s / **51.21 tok/s** / 1.46 s | 688.05 tok/s / 46.72 tok/s / 1.51 s |
157
+ | Pixel 11 Pro, GPU | 608.16 tok/s / 15.94 tok/s / 1.75 s | 588.90 tok/s / 14.14 tok/s / 1.81 s |
158
+ | Galaxy S26, CPU, first set | 187.11 tok/s / 35.80 tok/s / 5.54 s | 190.73 tok/s / 36.36 tok/s / 5.42 s |
159
+ | Galaxy S26, CPU, second set | 190.88 tok/s / 36.75 tok/s / 5.41 s | 192.07 tok/s / 36.43 tok/s / 5.38 s |
160
+
161
+ On the GPU the new file was faster in both pairs of runs on both phones. On the CPU the two sets do not agree. In the first set the new file was 0.7 % to 3.0 % slower in all four ratios. Two of them, runs 3 and 4 (3.0 % in prefill, 2.3 % in decode), are larger than the difference between the new file's own two runs of that set (2.4 % in prefill: 184.84 and 189.38 tok/s; 1.4 % in decode: 35.55 and 36.05 tok/s). In the second set it was 0.4 % to 1.3 % faster in three ratios and 2.1 % slower in one (prefill, runs 3 and 4), which is smaller than the 5.3 % by which the new file's own two runs of that set differ in prefill (185.72 and 196.04 tok/s). So this card claims no CPU speed change in either direction.
162
+
163
+ **On the Galaxy S26, run the LM on the GPU.** There the new file ran at 711.34 tok/s prefill and 51.21 tok/s decode on the GPU, against 190.88 and 36.75 tok/s on the CPU in the second set (187.11 and 35.80 in the first), with the same program and workload on the same day. The GPU was also the faster one in iteration 1: 714.37 and 716.12 tok/s prefill and 49.57 and 49.59 tok/s decode in the new file's two GPU runs, against 338.06 to 344.08 and 44.19 to 44.72 tok/s in its four CPU runs in the tables. The Pixel 11 Pro CPU was not run with the new file. The audio encoder stays on the CPU in either case.
164
+
165
+ Every GPU speed run started at Android's thermal status 0. On the Galaxy S26 (SM-S942Q) the GPU driver's thermal level (0 = no limit) reached 1 at most during the iterations of the GPU runs. On the Pixel 11 Pro, Android's thermal status stayed at 0 in every speed run. The CPU runs in the tables started at thermal status 0, except the first run of the first set, which started at 1. Before each CPU set, one more run of the new file was made and is not counted. A system app (`com.google.android.gms`) was in the foreground when each Galaxy S26 run started.
166
+
167
+ On both phones, the new file and the file it replaced gave the same transcripts of two LibriSpeech clips (1272-128104-0000 and 1272-128104-0001), byte for byte and with no word error against the reference texts (17 and 10 words): `Mister quilter is the apostle of the middle classes, and we are glad to welcome his gospel.` and `Nor is mister quilter's manner less interesting than his matter.` Each clip went in with its duration sentence, with the LM on the GPU, the audio encoder on the CPU and greedy decoding (`--sampler_top_k=1`).
168
+
169
  ## Conversion notes
170
 
171
  Converted with `litert-torch` 0.9.3/0.9.4, ai-edge-quantizer 0.9.0, litert-lm-builder 0.16.1, transformers 5.14.1 (native `vibevoice_asr`). Scripts and the full recipe: [hf-to-litertlm](https://github.com/john-rocky/hf-to-litertlm) `vibevoice_asr_work/`.
 
176
  - **Prompt**: the vendor's system prompt is baked into the bundle template; the user turn renders as `<|object_ref_start|>` + audio embeddings + `<|object_ref_end|>` + newline + your text.
177
  - **GPU**: the LM runs on Metal (macOS) and on the Android OpenCL delegate; the audio encoder is CPU-only — on Metal/WebGPU the 30 s window exceeds the delegate's 65 535-workgroup dispatch limit, and on Adreno the fp16 delegate path returns an empty transcript. Keep `audio_backend` on CPU.
178
 
179
+ ### 2026-11-17 — GPU graph (weights unchanged)
180
+
181
+ The prefill/decode graph now has the GPU form: 28 `odml.cache_update` composites for the KV cache and 56 attention composites. The previous file had neither; a check of its graph counted 56 `DYNAMIC_UPDATE_SLICE` and 56 `BATCH_MATMUL` ops instead. The prefill lengths (32, 128 and 512 tokens), the KV-cache length (2,048) and the weight quantization read the same before and after.
182
+
183
+ Only the prefill/decode graph section differs from the previous file: it went from 20,937,360 B to 21,289,304 B. The six other sections (metadata, tokenizer, embedder graph, audio encoder and the two weight sections) have the same size and sha256 as in the previous file. The file went from sha256 `498ad85c…` (1,983,053,824 B) to `7433f1a9…` (1,983,414,272 B).
184
+
185
+ With litert-lm 0.19.0, 0.19.1 and 0.19.2 on a Mac, on the CPU and on the GPU, the new file and the file it replaced both transcribed two test clips, to the same text byte for byte. The Mac comparison on 20 clips is under Correctness and the phone runs are in the last part of Performance.
186
+
187
+ Apart from the parts that name the file of 2026-11-17, this card's measurements of the file were made with its earlier versions.
188
+
189
  ## Known runtime issues
190
 
191
  **External weights (since 2026-11-09).** In `VibeVoice-ASR-BitNet.litertlm`, the weights sit in TFLiteWeights sections at the end of the file (LiteRT's external weights) instead of inside the model graph. On Android, the GPU path needs LiteRT-LM 0.19.0 or later: the GPU engine of the v0.18.0 and v0.18.1 Android builds crashes (SIGSEGV) on such a file ([LiteRT-LM#3986](https://github.com/google-ai-edge/LiteRT-LM/issues/3986)). The earlier file with the weights inside the graph stays at revision [`193b2d5`](https://huggingface.co/litert-community/VibeVoice-ASR-BitNet/tree/193b2d579a786206286ac36e74b6ea513562bdf5).
192
 
193
  From a conversation preface (the system message, initial messages, tools and extra context a conversation is created with) that has tools and no messages, LiteRT-LM renders this file's chat template with an empty message list, and the template reads `messages[0]` without checking that the list is non-empty, so the render fails with `Failed to apply template: undefined value`. This happens in two places: when the preface is prefilled at creation (`prefillPrefaceOnInit = true` in Kotlin, `prefillPrefaceOnInit: true` on the web, `SetPrefillPrefaceOnInit(true)` in C++; the default is false), where creating the conversation fails with that error, and in the preface render (`renderPrefaceIntoString()` in Kotlin and Swift, `RenderPrefaceIntoString()` in C++, `litert_lm_conversation_render_preface_to_string` in the C API). The same happens with extra context (such as `enable_thinking`) and no messages, and in the preface render with an empty preface (run in the runtime on one Qwen3 template; the other templates rendered with minijinja 2.14 and jinja2). Read in the source of LiteRT-LM 0.17.0, 0.18.1 and main; measured with LiteRT-LM 0.18.1 (the litert-lm package from PyPI) through its C API on this file's template text. The default path, where the first message is rendered together with the preface, is not affected: the Python API and the CLI of 0.18.1 expose neither the flag nor the preface render, and Google AI Edge Gallery sets neither (its source as read on 2026-10-25). A system message or any initial message in the conversation config avoids it.
194
 
195
+ From the second message of a conversation on, LiteRT-LM puts the previous answer's stop token into the KV cache with a zero embedding, because this file holds the embedding table in its own section and its prefill/decode graph takes embeddings as input (read in the source of LiteRT-LM 0.16.0, 0.18.1 and 0.19.1; measured with 0.18.1 on WeVisOne-VL-2B and Qwen2.5-Coder-1.5B-Instruct, not on this file). The first message of a conversation is not affected, so send one clip per conversation, as the blocks under Usage do.
196
+
197
+ On LiteRT-LM 0.18.1 and 0.19.1, once a message fails because it does not fit in the context window, every later message in that conversation fails as well (the next one with `Task failed with state: 7`). Start a new conversation after such an error. This was measured with other models' files, not with this one.
198
+
199
  ## License and changes
200
 
201
  Distributed under the **MIT License** (inherited from the base model; see `LICENSE`). **Changes from the original work:** weights ternarized (LM) and converted from safetensors fp32 to LiteRT flatbuffers with the quantization described above; the VAE decoder (synthesis half) is not included; three special tokens added to the tokenizer; chat/prompt template and audio-preprocessing parameters embedded as LiteRT-LM metadata.
VibeVoice-ASR-BitNet.litertlm CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:498ad85c609fb3821989db5f3767a07e38b82d5554bf26f73e091d939d9c1b0b
3
- size 1983053824
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7433f1a9e2294a70544ee79ab7a332d0ce499660cef261a5e156c5458e879e8e
3
+ size 1983414272
litertlm_manifest.json CHANGED
@@ -1,7 +1,7 @@
1
  {
2
  "manifest_schema": "0.1.3",
3
  "repo": "litert-community/VibeVoice-ASR-BitNet",
4
- "generated": "2026-11-09",
5
  "generator": "make_manifest.py",
6
  "model": {
7
  "display_name": "VibeVoice-ASR-BitNet (speech-to-text LLM, 24 kHz audio in)",
@@ -22,8 +22,8 @@
22
  "variants": [
23
  {
24
  "file": "VibeVoice-ASR-BitNet.litertlm",
25
- "sha256": "498ad85c609fb3821989db5f3767a07e38b82d5554bf26f73e091d939d9c1b0b",
26
- "size_bytes": 1983053824,
27
  "sections": [
28
  {
29
  "type": "LlmMetadataProto",
@@ -40,7 +40,7 @@
40
  },
41
  {
42
  "type": "TFLiteModel",
43
- "size_bytes": 20937360,
44
  "model_type": "tf_lite_prefill_decode"
45
  },
46
  {
@@ -75,7 +75,7 @@
75
  {
76
  "platform": "android",
77
  "backend": "gpu",
78
- "reason": "Galaxy S26: LM on OpenCL 652 tok/s prefill / 36 tok/s decode vs 163 / 35 on CPU, transcripts match the Mac GPU run; the audio encoder must stay on CPU (audio_backend cpu) — on the GPU delegate it returns an empty transcript"
79
  }
80
  ],
81
  "requirements": {
@@ -85,7 +85,8 @@
85
  "audio_backend must be cpu (--audio_backend=cpu / audio_backend=CPU()); the LM may run on gpu",
86
  "The encoder window is 30 s; longer clips are chunked by the engine with a context reset at each 30 s boundary",
87
  "Accuracy: 20 LibriSpeech dev-clean clips (448 words): WER 2.68 % on macOS CPU and 2.46 % LM-on-GPU (fp32 PyTorch reference 2.68 %), 3.12 % Galaxy S26 CPU, 2.46 % Galaxy S26 LM-on-GPU; vendor card Libri-clean 2.41 % (full test set, VibeASR.cpp)",
88
- "Peak private footprint on Galaxy S26: 3227 MB CPU/CPU, 2268 MB LM-on-GPU (--report_peak_memory_footprint, 5.9 s clip)"
 
89
  ]
90
  },
91
  "measured": [
@@ -147,6 +148,74 @@
147
  "runs": 1,
148
  "date": "2026-09-04",
149
  "source": "--benchmark with a 264-token text prompt, one reading (LM only; audio encoder on CPU)"
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
150
  }
151
  ],
152
  "known_issues": [
 
1
  {
2
  "manifest_schema": "0.1.3",
3
  "repo": "litert-community/VibeVoice-ASR-BitNet",
4
+ "generated": "2026-11-17",
5
  "generator": "make_manifest.py",
6
  "model": {
7
  "display_name": "VibeVoice-ASR-BitNet (speech-to-text LLM, 24 kHz audio in)",
 
22
  "variants": [
23
  {
24
  "file": "VibeVoice-ASR-BitNet.litertlm",
25
+ "sha256": "7433f1a9e2294a70544ee79ab7a332d0ce499660cef261a5e156c5458e879e8e",
26
+ "size_bytes": 1983414272,
27
  "sections": [
28
  {
29
  "type": "LlmMetadataProto",
 
40
  },
41
  {
42
  "type": "TFLiteModel",
43
+ "size_bytes": 21289304,
44
  "model_type": "tf_lite_prefill_decode"
45
  },
46
  {
 
75
  {
76
  "platform": "android",
77
  "backend": "gpu",
78
+ "reason": "Galaxy S26, this file (2026-11-17, litert_lm_advanced_main v0.19.0, 1,024-token prefill, 256 decode tokens, the mean of iterations 2-3 over 2 runs): LM on the GPU 711.34 tok/s prefill / 51.21 tok/s decode vs 190.88 / 36.75 on the CPU (the second of two CPU sets; the first gave 187.11 / 35.80). An earlier file (2026-09-04, v0.16.1, 264-token prompt): LM on OpenCL 652 tok/s prefill / 36 tok/s decode vs 163 / 35 on CPU, transcripts match the Mac GPU run; the audio encoder must stay on CPU (audio_backend cpu) — on the GPU delegate it returns an empty transcript"
79
  }
80
  ],
81
  "requirements": {
 
85
  "audio_backend must be cpu (--audio_backend=cpu / audio_backend=CPU()); the LM may run on gpu",
86
  "The encoder window is 30 s; longer clips are chunked by the engine with a context reset at each 30 s boundary",
87
  "Accuracy: 20 LibriSpeech dev-clean clips (448 words): WER 2.68 % on macOS CPU and 2.46 % LM-on-GPU (fp32 PyTorch reference 2.68 %), 3.12 % Galaxy S26 CPU, 2.46 % Galaxy S26 LM-on-GPU; vendor card Libri-clean 2.41 % (full test set, VibeASR.cpp)",
88
+ "Peak private footprint on Galaxy S26: 3227 MB CPU/CPU, 2268 MB LM-on-GPU (--report_peak_memory_footprint, 5.9 s clip)",
89
+ "2026-11-17 GPU graph (weights unchanged): the prefill/decode graph has the GPU composite form (28 odml.cache_update composites for the KV cache and 56 attention composites; the previous file had none); every weight and every section outside that graph has the same size and sha256 as in the previous file (Hub revision 0d3aca6f). This file was tested on litert-lm 0.19.0-0.19.2 (macOS) and litert_lm_advanced_main v0.19.0 (Galaxy S26, Pixel 11 Pro). On macOS with litert-lm 0.19.2 this file and the previous file gave the same transcripts of the 20 clips: 18/448 word errors (4.02 %) with both files on the CPU with 1 thread, 11/448 (2.46 %) with both files with the LM on the GPU. The figures in the notes above, peak_ram_mb, the macOS recommendation and the measured rows dated 2026-09-04 were measured on earlier files."
90
  ]
91
  },
92
  "measured": [
 
148
  "runs": 1,
149
  "date": "2026-09-04",
150
  "source": "--benchmark with a 264-token text prompt, one reading (LM only; audio encoder on CPU)"
151
+ },
152
+ {
153
+ "device": "Galaxy S26 (SM-S942Q)",
154
+ "os": "Android",
155
+ "backend": "gpu",
156
+ "runtime": "litert_lm_advanced_main v0.19.0 source (commit 771b06e1, the parent of the v0.19.0 tag commit), LiteRT GPU (LITERT_CL)",
157
+ "prompt_tokens": 1024,
158
+ "decode_tokens": 256,
159
+ "prefill_tps": 711.34,
160
+ "decode_tps": 51.21,
161
+ "ttft_s": 1.46,
162
+ "load_s": 2.7,
163
+ "max_num_tokens": 2048,
164
+ "cache": "no",
165
+ "runs": 2,
166
+ "date": "2026-11-17",
167
+ "source": "--benchmark with 1,024 prefill tokens and 256 decode tokens (LM only, text prompt), 3 iterations per run in one process; the values are the mean of iterations 2-3 of each run, then the mean of this file's 2 runs; four runs went previous, this, this, previous, each a new process started at thermal status 0 (Android's thermal status); per run prefill 689.25, 710.56, 712.12 and 686.85 tok/s and decode 46.85, 51.08, 51.34 and 46.59 tok/s, so this file prefilled 1.031 and 1.037 times as fast as the previous file in the two pairs and decoded 1.090 and 1.102 times as fast; the previous file by the same statistic: 688.05 prefill / 46.72 decode, time to first token 1.51 s; the GPU thermal level (the GPU driver's own level; 0 = no limit) reached 1 at most during the iterations; a system app (com.google.android.gms) was in the foreground when each run started; load_s is the mean engine init of this file's 2 runs; on this phone this file and the previous file gave the same transcripts of two LibriSpeech clips (LM on the GPU, audio encoder on the CPU, greedy decoding), byte for byte"
168
+ },
169
+ {
170
+ "device": "Pixel 11 Pro",
171
+ "os": "Android",
172
+ "backend": "gpu",
173
+ "runtime": "litert_lm_advanced_main v0.19.0 source (commit 771b06e1, the parent of the v0.19.0 tag commit), LiteRT GPU (LITERT_CL)",
174
+ "prompt_tokens": 1024,
175
+ "decode_tokens": 256,
176
+ "prefill_tps": 608.16,
177
+ "decode_tps": 15.94,
178
+ "ttft_s": 1.75,
179
+ "load_s": 5.6,
180
+ "max_num_tokens": 2048,
181
+ "cache": "no",
182
+ "runs": 2,
183
+ "date": "2026-11-17",
184
+ "source": "--benchmark with 1,024 prefill tokens and 256 decode tokens (LM only, text prompt), 3 iterations per run in one process; the values are the mean of iterations 2-3 of each run, then the mean of this file's 2 runs; four runs went previous, this, this, previous, each a new process started at thermal status 0 (Android's thermal status); per run prefill 587.37, 610.56, 605.76 and 590.42 tok/s and decode 13.87, 15.50, 16.38 and 14.42 tok/s, so this file prefilled 1.039 and 1.026 times as fast as the previous file in the two pairs and decoded 1.118 and 1.136 times as fast; the previous file by the same statistic: 588.90 prefill / 14.14 decode, time to first token 1.81 s; the thermal status stayed at 0 in every speed run; load_s is the mean engine init of this file's 2 runs; the Pixel 11 Pro CPU was not measured; on this phone this file and the previous file gave the same transcripts of two LibriSpeech clips (LM on the GPU, audio encoder on the CPU, greedy decoding), byte for byte"
185
+ },
186
+ {
187
+ "device": "Galaxy S26 (SM-S942Q)",
188
+ "os": "Android",
189
+ "backend": "cpu",
190
+ "runtime": "litert_lm_advanced_main v0.19.0 source (commit 771b06e1, the parent of the v0.19.0 tag commit), XNNPACK CPU, 4 threads (the program's default)",
191
+ "prompt_tokens": 1024,
192
+ "decode_tokens": 256,
193
+ "prefill_tps": 187.11,
194
+ "decode_tps": 35.8,
195
+ "ttft_s": 5.54,
196
+ "load_s": 1.53,
197
+ "max_num_tokens": 2048,
198
+ "cache": "no",
199
+ "runs": 2,
200
+ "date": "2026-11-17",
201
+ "source": "--benchmark with 1,024 prefill tokens and 256 decode tokens (LM only, text prompt), 3 iterations per run in one process; the values are the mean of iterations 2-3 of each run, then the mean of this file's 2 runs; the first of two CPU sets on this day; four runs went previous, this, this, previous after one run of this file that is not counted, each a new process; the first run started at thermal status 1 (Android's thermal status), the other three at 0; per run prefill 186.24, 184.84, 189.38 and 195.21 tok/s and decode 35.80, 35.55, 36.05 and 36.92 tok/s, so this file prefilled 0.992 and 0.970 times as fast as the previous file in the two pairs and decoded 0.993 and 0.977 times as fast; this file's own two runs differ by 2.4 % in prefill and 1.4 % in decode; the previous file by the same statistic: 190.73 prefill / 36.36 decode, time to first token 5.42 s; iteration 1 of this file's 2 runs: 338.06 and 341.64 tok/s prefill, 44.21 and 44.19 tok/s decode; a system app (com.google.android.gms) was in the foreground when each run started; load_s is the mean engine init of this file's 2 runs"
202
+ },
203
+ {
204
+ "device": "Galaxy S26 (SM-S942Q)",
205
+ "os": "Android",
206
+ "backend": "cpu",
207
+ "runtime": "litert_lm_advanced_main v0.19.0 source (commit 771b06e1, the parent of the v0.19.0 tag commit), XNNPACK CPU, 4 threads (the program's default)",
208
+ "prompt_tokens": 1024,
209
+ "decode_tokens": 256,
210
+ "prefill_tps": 190.88,
211
+ "decode_tps": 36.75,
212
+ "ttft_s": 5.41,
213
+ "load_s": 1.53,
214
+ "max_num_tokens": 2048,
215
+ "cache": "no",
216
+ "runs": 2,
217
+ "date": "2026-11-17",
218
+ "source": "--benchmark with 1,024 prefill tokens and 256 decode tokens (LM only, text prompt), 3 iterations per run in one process; the values are the mean of iterations 2-3 of each run, then the mean of this file's 2 runs; the second of two CPU sets on this day; four runs went previous, this, this, previous after one run of this file that is not counted, each a new process started at thermal status 0 (Android's thermal status); per run prefill 183.99, 185.72, 196.04 and 200.15 tok/s and decode 35.99, 36.46, 37.03 and 36.87 tok/s, so this file prefilled 1.009 and 0.979 times as fast as the previous file in the two pairs and decoded 1.013 and 1.004 times as fast; this file's own two runs differ by 5.3 % in prefill; the previous file by the same statistic: 192.07 prefill / 36.43 decode, time to first token 5.38 s; iteration 1 of this file's 2 runs: 342.41 and 344.08 tok/s prefill, 44.72 and 44.69 tok/s decode; a system app (com.google.android.gms) was in the foreground when each run started; load_s is the mean engine init of this file's 2 runs"
219
  }
220
  ],
221
  "known_issues": [