mlboydaisuke commited on
Commit
a37547b
·
verified ·
1 Parent(s): 9e0df43

README: the time of the 2026-11-17 file replacement in UTC (11:58 UTC), the Python example run with the new file, and plainer wording; manifest notes reworded the same way

Browse files
Files changed (2) hide show
  1. README.md +5 -5
  2. litertlm_manifest.json +2 -2
README.md CHANGED
@@ -29,7 +29,7 @@ Measured on device with an earlier file (edge-compat): Galaxy S26 · LiteRT-LM 0
29
 
30
  # VibeVoice-ASR-BitNet — LiteRT-LM
31
 
32
- [microsoft/VibeVoice-ASR-BitNet](https://huggingface.co/microsoft/VibeVoice-ASR-BitNet) converted to the **LiteRT-LM** (`.litertlm`) format for on-device speech recognition with Google's [LiteRT-LM](https://github.com/google-ai-edge/LiteRT-LM) runtime. **Audio in, text out, in one bundle**: the runtime decodes the clip, runs the bundled audio encoder, and the LLM transcribes — no host-side feature extraction.
33
 
34
  On 2026-11-17 the file in this repo was replaced with a file whose prefill/decode graph is rewritten for the GPU, with every weight unchanged. On a Galaxy S26 GPU the new file decodes 51.2 tokens/s after a 1,024-token prefill, against 46.7 tokens/s for the file it replaced (LiteRT-LM v0.19.0; see the end of Performance). The new file was run only with LiteRT-LM 0.19.0 to 0.19.2; earlier runtimes were not tried with it.
35
 
@@ -57,7 +57,7 @@ Sections: prefill/decode 793 MB, embedder 237 MB, audio encoder 951 MB, tokenize
57
 
58
  The Android CPU row differs from the Mac by one clip ("On the general principles…" → "Under general principles…"), an int8 dynamic-range numerics flip on Arm (a Pixel 8a produced the identical transcript); the GPU LM row matches the Mac GPU row word for word. For reference, Microsoft reports 2.41 % on the full LibriSpeech test-clean set with their VibeASR.cpp runtime.
59
 
60
- Sending the clip **without** the duration sentence works (the bundle's template then sends "Please transcribe it.") at 3.57 % on the same set — include the duration for best results.
61
 
62
  The table and the paragraphs above were measured on earlier files. The new file was compared with the file it replaced on the same 20 clips on a Mac with litert-lm 0.19.2 (2026-11-16, greedy decoding, the duration sentence in the prompt, audio encoder on the CPU). On the CPU with 1 thread, the new file gave the same transcript as the previous file for all 20 clips, byte for byte (18/448 word errors, 4.02 %, with both files). With the LM on the Mac GPU, the two files also gave the same transcripts for all 20 clips (11/448, 2.46 %, with both files). The CPU count of 18/448 is higher than the 12/448 of the table's Mac CPU row (LiteRT-LM 0.16.1). Both files give it, and no run on this card shows which difference between the two measurements accounts for it.
63
 
@@ -65,7 +65,7 @@ The table and the paragraphs above were measured on earlier files. The new file
65
 
66
  The audio goes in as an audio content item; any container/sample rate the runtime's decoder reads (wav, mp3, flac — it resamples to 24 kHz). The **audio encoder must run on the CPU** (`audio_backend`), the LM may run on CPU or GPU. On Android the GPU was the faster of the two with the new file on a Galaxy S26 (numbers at the end of Performance); the two blocks below run the LM on the CPU.
67
 
68
- Python (`pip install litert-lm-api`; this block was run as written on 0.16.1 and 0.17.1 with earlier files):
69
 
70
  ```python
71
  import litert_lm
@@ -99,7 +99,7 @@ conv.close()
99
 
100
  If you download this file with the `litert-lm` command-line tool (PyPI `litert-lm`, not the `litert_lm_advanced_main` program) and it stops right after the download with no error message of its own (a bus error; exit status 138 on a Mac) or with an error log that contains `too large for file size`, the downloaded file may be incomplete. The litert-lm CLI (0.14.0 to 0.19.2, and litert-lm-nightly 0.20.0.dev20261112) can keep an incomplete download in `~/.litert-lm/cache/huggingface/litert-community/VibeVoice-ASR-BitNet/` and uses it again on later runs without checking its size. Delete that file and run the command again (after an `import`, run the `import` again). This was measured with other models' files, not with this one.
101
 
102
- If you downloaded this file with the litert-lm CLI's `--from-huggingface-repo` before it was replaced on 2026-11-17 (Japan time), delete it from `~/.litert-lm/cache/huggingface/litert-community/VibeVoice-ASR-BitNet/`, then run your command again (after an `import`, run the `import` again). Without deleting it, the litert-lm CLI (0.14.0 to 0.19.2) keeps running the previous file: it uses a file it finds there without checking it against this repo. The previous file stays at revision [`0d3aca6`](https://huggingface.co/litert-community/VibeVoice-ASR-BitNet/tree/0d3aca6fff0167fdce76fbc0c9e89d20421a7f05).
103
 
104
  Notes:
105
 
@@ -173,7 +173,7 @@ Converted with `litert-torch` 0.9.3/0.9.4, ai-edge-quantizer 0.9.0, litert-lm-bu
173
  - **Ternarization is applied at conversion, per tensor, exactly as the vendor runtime does.** The Hub checkpoint stores fp32 latent weights; VibeASR.cpp quantizes the seven projections per LM layer with a per-tensor absmean scale at GGUF-conversion time. The same values are stored here as int4 blockwise-128 min-max, which represents a per-tensor ternary exactly (every block is {−α, 0, +α}); the embedding/head stays int8.
174
  - **Audio front-end folded into the encoder graph.** Input is raw 24 kHz PCM framed by the runtime (3200 samples = one latent frame); the vendor's −25 dBFS RMS normalisation and peak clip run in-graph. Acoustic latents are the mean (no sampling noise), as in the vendor runtime.
175
  - **Tokenizer**: the upstream `tokenizer.json` (Qwen2) has no string for the speech-marker ids the model was trained with; `<|object_ref_start|>` / `<|object_ref_end|>` / `<|box_start|>` are added as special tokens on ids 151646/151647/151648 so the runtime can emit them around the audio embeddings. Ordinary text tokenizes identically.
176
- - **Prompt**: the vendor's system prompt is baked into the bundle template; the user turn renders as `<|object_ref_start|>` + audio embeddings + `<|object_ref_end|>` + newline + your text.
177
  - **GPU**: the LM runs on Metal (macOS) and on the Android OpenCL delegate; the audio encoder is CPU-only — on Metal/WebGPU the 30 s window exceeds the delegate's 65 535-workgroup dispatch limit, and on Adreno the fp16 delegate path returns an empty transcript. Keep `audio_backend` on CPU.
178
 
179
  ### 2026-11-17 — GPU graph (weights unchanged)
 
29
 
30
  # VibeVoice-ASR-BitNet — LiteRT-LM
31
 
32
+ [microsoft/VibeVoice-ASR-BitNet](https://huggingface.co/microsoft/VibeVoice-ASR-BitNet) converted to the **LiteRT-LM** (`.litertlm`) format for on-device speech recognition with Google's [LiteRT-LM](https://github.com/google-ai-edge/LiteRT-LM) runtime. **Audio in, text out, in one file**: the runtime decodes the clip, runs the file's audio encoder, and the LLM transcribes — no host-side feature extraction.
33
 
34
  On 2026-11-17 the file in this repo was replaced with a file whose prefill/decode graph is rewritten for the GPU, with every weight unchanged. On a Galaxy S26 GPU the new file decodes 51.2 tokens/s after a 1,024-token prefill, against 46.7 tokens/s for the file it replaced (LiteRT-LM v0.19.0; see the end of Performance). The new file was run only with LiteRT-LM 0.19.0 to 0.19.2; earlier runtimes were not tried with it.
35
 
 
57
 
58
  The Android CPU row differs from the Mac by one clip ("On the general principles…" → "Under general principles…"), an int8 dynamic-range numerics flip on Arm (a Pixel 8a produced the identical transcript); the GPU LM row matches the Mac GPU row word for word. For reference, Microsoft reports 2.41 % on the full LibriSpeech test-clean set with their VibeASR.cpp runtime.
59
 
60
+ Sending the clip **without** the duration sentence works (the file's template then sends "Please transcribe it.") at 3.57 % on the same set — include the duration for best results.
61
 
62
  The table and the paragraphs above were measured on earlier files. The new file was compared with the file it replaced on the same 20 clips on a Mac with litert-lm 0.19.2 (2026-11-16, greedy decoding, the duration sentence in the prompt, audio encoder on the CPU). On the CPU with 1 thread, the new file gave the same transcript as the previous file for all 20 clips, byte for byte (18/448 word errors, 4.02 %, with both files). With the LM on the Mac GPU, the two files also gave the same transcripts for all 20 clips (11/448, 2.46 %, with both files). The CPU count of 18/448 is higher than the 12/448 of the table's Mac CPU row (LiteRT-LM 0.16.1). Both files give it, and no run on this card shows which difference between the two measurements accounts for it.
63
 
 
65
 
66
  The audio goes in as an audio content item; any container/sample rate the runtime's decoder reads (wav, mp3, flac — it resamples to 24 kHz). The **audio encoder must run on the CPU** (`audio_backend`), the LM may run on CPU or GPU. On Android the GPU was the faster of the two with the new file on a Galaxy S26 (numbers at the end of Performance); the two blocks below run the LM on the CPU.
67
 
68
+ Python (`pip install litert-lm-api`; this block was run as written on 0.16.1 and 0.17.1 with earlier files). On 2026-11-17 it was run as written on 0.19.2 on a Mac, with a clip's path in place of `/abs/path/clip.wav`: once with the new file and once with the file it replaced. Both runs printed the same text.
69
 
70
  ```python
71
  import litert_lm
 
99
 
100
  If you download this file with the `litert-lm` command-line tool (PyPI `litert-lm`, not the `litert_lm_advanced_main` program) and it stops right after the download with no error message of its own (a bus error; exit status 138 on a Mac) or with an error log that contains `too large for file size`, the downloaded file may be incomplete. The litert-lm CLI (0.14.0 to 0.19.2, and litert-lm-nightly 0.20.0.dev20261112) can keep an incomplete download in `~/.litert-lm/cache/huggingface/litert-community/VibeVoice-ASR-BitNet/` and uses it again on later runs without checking its size. Delete that file and run the command again (after an `import`, run the `import` again). This was measured with other models' files, not with this one.
101
 
102
+ If you downloaded this file with the litert-lm CLI's `--from-huggingface-repo` before it was replaced on 2026-11-17 (11:58 UTC), delete it from `~/.litert-lm/cache/huggingface/litert-community/VibeVoice-ASR-BitNet/`, then run your command again (after an `import`, run the `import` again). Without deleting it, the litert-lm CLI (0.14.0 to 0.19.2) keeps running the previous file: it uses a file it finds there without checking it against this repo. The previous file stays at revision [`0d3aca6`](https://huggingface.co/litert-community/VibeVoice-ASR-BitNet/tree/0d3aca6fff0167fdce76fbc0c9e89d20421a7f05).
103
 
104
  Notes:
105
 
 
173
  - **Ternarization is applied at conversion, per tensor, exactly as the vendor runtime does.** The Hub checkpoint stores fp32 latent weights; VibeASR.cpp quantizes the seven projections per LM layer with a per-tensor absmean scale at GGUF-conversion time. The same values are stored here as int4 blockwise-128 min-max, which represents a per-tensor ternary exactly (every block is {−α, 0, +α}); the embedding/head stays int8.
174
  - **Audio front-end folded into the encoder graph.** Input is raw 24 kHz PCM framed by the runtime (3200 samples = one latent frame); the vendor's −25 dBFS RMS normalisation and peak clip run in-graph. Acoustic latents are the mean (no sampling noise), as in the vendor runtime.
175
  - **Tokenizer**: the upstream `tokenizer.json` (Qwen2) has no string for the speech-marker ids the model was trained with; `<|object_ref_start|>` / `<|object_ref_end|>` / `<|box_start|>` are added as special tokens on ids 151646/151647/151648 so the runtime can emit them around the audio embeddings. Ordinary text tokenizes identically.
176
+ - **Prompt**: the vendor's system prompt is baked into the file's template; the user turn renders as `<|object_ref_start|>` + audio embeddings + `<|object_ref_end|>` + newline + your text.
177
  - **GPU**: the LM runs on Metal (macOS) and on the Android OpenCL delegate; the audio encoder is CPU-only — on Metal/WebGPU the 30 s window exceeds the delegate's 65 535-workgroup dispatch limit, and on Adreno the fp16 delegate path returns an empty transcript. Keep `audio_backend` on CPU.
178
 
179
  ### 2026-11-17 — GPU graph (weights unchanged)
litertlm_manifest.json CHANGED
@@ -81,7 +81,7 @@
81
  "requirements": {
82
  "peak_ram_mb": 3227,
83
  "platform_notes": [
84
- "Audio only through the runtime's generic audio path: send the clip as an audio content item (wav/mp3/flac; any sample rate — the engine resamples to 24 kHz) plus the text 'This is a X.XX seconds audio, please transcribe it.' (X.XX = clip length); with no text the bundle's template sends 'Please transcribe it.' (+0.9 pp WER on the fixture set)",
85
  "audio_backend must be cpu (--audio_backend=cpu / audio_backend=CPU()); the LM may run on gpu",
86
  "The encoder window is 30 s; longer clips are chunked by the engine with a context reset at each 30 s boundary",
87
  "Accuracy: 20 LibriSpeech dev-clean clips (448 words): WER 2.68 % on macOS CPU and 2.46 % LM-on-GPU (fp32 PyTorch reference 2.68 %), 3.12 % Galaxy S26 CPU, 2.46 % Galaxy S26 LM-on-GPU; vendor card Libri-clean 2.41 % (full test set, VibeASR.cpp)",
@@ -221,7 +221,7 @@
221
  "known_issues": [
222
  "Android GPU needs LiteRT-LM 0.19.0 or later: since 2026-11-09 this file's weights sit in TFLiteWeights sections at the end of the file (LiteRT's external weights), and the GPU engine of the v0.18.0 and v0.18.1 Android builds crashes (SIGSEGV) on such a file (https://github.com/google-ai-edge/LiteRT-LM/issues/3986); the earlier file with the weights inside the graph stays at revision 193b2d579a786206286ac36e74b6ea513562bdf5 (https://huggingface.co/litert-community/VibeVoice-ASR-BitNet/tree/193b2d579a786206286ac36e74b6ea513562bdf5)",
223
  "Audio encoder does not run on the GPU delegates: on Metal/WebGPU the 30 s window dispatches 90001+ workgroups per stem kernel (limit 65535); on Adreno (OpenCL, all 1857 nodes delegated) the fp16 path returns an empty transcript; keep audio_backend=cpu",
224
- "Audio placeholder tokens <|object_ref_start|>/<|object_ref_end|>/<|box_start|> are added to the bundled tokenizer.json (ids 151646/151647/151648, the ids the model was trained with); the upstream tokenizer.json has no string for them"
225
  ],
226
  "context_length": 2048
227
  }
 
81
  "requirements": {
82
  "peak_ram_mb": 3227,
83
  "platform_notes": [
84
+ "Audio only through the runtime's generic audio path: send the clip as an audio content item (wav/mp3/flac; any sample rate — the runtime resamples to 24 kHz) plus the text 'This is a X.XX seconds audio, please transcribe it.' (X.XX = clip length); with no text the file's template sends 'Please transcribe it.' (+0.9 pp WER on the 20 LibriSpeech clips)",
85
  "audio_backend must be cpu (--audio_backend=cpu / audio_backend=CPU()); the LM may run on gpu",
86
  "The encoder window is 30 s; longer clips are chunked by the engine with a context reset at each 30 s boundary",
87
  "Accuracy: 20 LibriSpeech dev-clean clips (448 words): WER 2.68 % on macOS CPU and 2.46 % LM-on-GPU (fp32 PyTorch reference 2.68 %), 3.12 % Galaxy S26 CPU, 2.46 % Galaxy S26 LM-on-GPU; vendor card Libri-clean 2.41 % (full test set, VibeASR.cpp)",
 
221
  "known_issues": [
222
  "Android GPU needs LiteRT-LM 0.19.0 or later: since 2026-11-09 this file's weights sit in TFLiteWeights sections at the end of the file (LiteRT's external weights), and the GPU engine of the v0.18.0 and v0.18.1 Android builds crashes (SIGSEGV) on such a file (https://github.com/google-ai-edge/LiteRT-LM/issues/3986); the earlier file with the weights inside the graph stays at revision 193b2d579a786206286ac36e74b6ea513562bdf5 (https://huggingface.co/litert-community/VibeVoice-ASR-BitNet/tree/193b2d579a786206286ac36e74b6ea513562bdf5)",
223
  "Audio encoder does not run on the GPU delegates: on Metal/WebGPU the 30 s window dispatches 90001+ workgroups per stem kernel (limit 65535); on Adreno (OpenCL, all 1857 nodes delegated) the fp16 path returns an empty transcript; keep audio_backend=cpu",
224
+ "Audio placeholder tokens <|object_ref_start|>/<|object_ref_end|>/<|box_start|> are added to the tokenizer.json inside the file (ids 151646/151647/151648, the ids the model was trained with); the upstream tokenizer.json has no string for them"
225
  ],
226
  "context_length": 2048
227
  }