spybyscript commited on
Commit
39d57af
·
verified ·
1 Parent(s): c528ceb

Publish INT8 T1500 release and preserve previous FP16 bundle under legacy

Browse files

Archive every prior release file including its ZIP byte-for-byte. Publish the validated INT8 encoder, unchanged decoder/joint/tokenizer, updated ZIP, measured memory/quality evidence and native integration instructions. Android memory and delegate validation remain pending.

INTEGRATION.md ADDED
@@ -0,0 +1,55 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Integrating the INT8 Parakeet bundle
2
+
3
+ ## Migrating from the FP16 release
4
+
5
+ Use the root INT8 encoder with root decoder, joint and tokenizer. Their filenames remain stable. Decoder/joint/tokenizer bytes match the previous release. Check `release_id` and SHA256 in the manifest when invalidating downloaded-file caches. The previous complete release is in `legacy/`; its documentation describes that historical release, not the current INT8 memory measurements.
6
+
7
+ The 15-second shape and TDT token/duration contracts are retained. Use the actual tensor names and signature metadata rather than relying on positional ordering across exports. Keep all three interpreters alive and reuse buffers; do not load all precision variants concurrently.
8
+
9
+ ## Encoder
10
+
11
+ Signature: `serving_default`.
12
+
13
+ | Role | Actual tensor name | Type | Shape |
14
+ |---|---|---|---|
15
+ | Normalized log-mel input | `serving_default_args_0` | float32 | `[1,128,1500]` |
16
+ | Actual valid mel length | `serving_default_args_1` | int32 | `[1]` |
17
+ | Encoded sequence | `serving_default_output_0_output` | float32 | `[1,1024,188]` |
18
+ | Valid encoder length | `serving_default_output_1_output` | int32 | `[1]` |
19
+
20
+ Anchor actual audio/features at position zero and zero-pad only at the tail. Pass the true valid length. Use the returned encoded length rather than treating all 188 output positions as valid. Handle empty/no-speech input in the host; this bundle does not include VAD.
21
+
22
+ ## Frontend
23
+
24
+ Match the evaluated NeMo frontend: 16 kHz mono, preemphasis 0.97, FFT 512, nonperiodic Hann window 400 samples, hop 160, 128 Slaney-normalized mel filters, power spectrum, and natural log with additive guard `2**-24`. Inference dither is disabled. Normalize each mel channel over valid time frames using the mean and sample standard deviation plus `1e-5`; mask invalid frames to zero afterward. Valid mel length is the reference frontend's floor of sample count / 160, not simply every STFT output column.
25
+
26
+ For the long-recording validation, frontend features were computed independently for each overlapping audio window. Do not normalize across future, unavailable audio for a live implementation. Match centering/padding, mel filter construction and lengths against NeMo before attributing differences to quantization. The legacy README's `log(mel + 1e-5)` description was not the exact evaluated NeMo log guard; the values above record the actual reference. No frontend graph or new frontend implementation is included here.
27
+
28
+ ## Prediction network and joint
29
+
30
+ Decoder inputs: token int64 `[1,1]`, hidden and cell float32 `[2,1,640]`. Outputs: prediction float32 `[1,1,640]`, updated hidden/cell of the same state shape.
31
+
32
+ Joint inputs: encoded frame float32 `[1,1024,1]`, transposed prediction float32 `[1,640,1]`. Output: float32 `[1,1,1,8198]`. Split logits into `[0:8193]` for tokens and `[8193:8198]` for durations `[0,1,2,3,4]`. Vocabulary contains 8192 pieces; blank ID is **8192**.
33
+
34
+ Initialize hidden/cell to zero and prime the decoder with blank. On non-blank emission, append the token, update decoder state, and advance by the predicted duration. On blank, advance by at least one frame without updating prediction state. Guard repeated zero-duration emissions, as in the validated host loop. Reset independent-window decoder state; do not pretend this offline encoder has a cache. Do not blindly collapse repeated text or token pieces.
35
+
36
+ ## Continuous input
37
+
38
+ Tested profile: 496 new mel frames per commit, 496 left-context frames, 200 right-context frames. Each mel frame is 10 ms; encoder steps are 80 ms. Re-encode the overlapping window and commit only tokens in its target interval. No left zero-padding at the beginning of the recording. The profile is approximately 5-second updates after approximately 7 seconds of initial buffering, plus compute. Audio/frame timestamps are emission estimates, not verified word alignments. Boundary stitching remains imperfect.
39
+
40
+ ## Memory and optional native mapping hint
41
+
42
+ The default INT8 runtime measured about 1.60 GB process peak on Linux. The optional verified source-page hint measured about 1.46 GB initialization peak and 0.954 GB retained RSS after the complete evaluation. The latter includes all three model objects; it is not a post-teardown measurement. Full English and long-recording token sequences were identical with and without the hint.
43
+
44
+ `mapped_model_file.h` is a Linux/Android native example, compiled and tested on Linux only. It owns an immutable file-backed mapping. If the chosen model API explicitly supports an externally owned read-only model buffer:
45
+
46
+ 1. Construct the mapping owner before the model/interpreter and keep it alive until after they are destroyed.
47
+ 2. Prepare the CPU delegate and run representative encoder warmup.
48
+ 3. Call `DropVerifiedSourcePages()` once. It verifies mapped bytes against the file before `MADV_DONTNEED`.
49
+ 4. Keep the mapping and file valid. Required pages can fault back in; this does not unmap the model.
50
+
51
+ This discards redundant resident source pages after weights have been packed. It does not shrink the file, guarantee immediate removal from the global OS file cache, or eliminate startup allocations. Some runtimes may continue reading source weights and refault more pages.
52
+
53
+ Never apply the hint to an anonymous or writable model ByteBuffer. Do not advise an unrelated second mapping and assume it affects the interpreter. Do not call it while inference is running. A Java file-path-only `CompiledModel` API may not expose the necessary mapping, so this is not a drop-in Java option. Ordinary INT8 remains usable without this optimization.
54
+
55
+ On Android, measure total application PSS/RSS, initialization overlap, sustained latency, page faults and thermals. The local ASR figures exclude the UI, microphone frontend, text/audio emotion models and a second ASR model. Keeping Nemotron and Parakeet resident together is not covered by this memory result. GPU/NPU delegates and INT8 coverage must be evaluated separately; retain CPU decoder/joint initially.
NOTICE.md ADDED
@@ -0,0 +1 @@
 
 
1
+ Derived from NVIDIA Parakeet-TDT-0.6B-v3, licensed CC BY 4.0: https://creativecommons.org/licenses/by/4.0/ . Original model and attribution: https://huggingface.co/nvidia/parakeet-tdt-0.6b-v3 . Modified by community LiteRT export and post-training quantization; no additional training.
README.md CHANGED
@@ -9,198 +9,88 @@ language:
9
  - pt
10
  library_name: litert
11
  base_model: nvidia/parakeet-tdt-0.6b-v3
 
12
  tags:
13
- - automatic-speech-recognition
14
  - speech
15
  - audio
16
  - parakeet
17
  - tdt
18
- - litert
19
  - tflite
20
- - on-device
21
- - mobile
22
  - android
23
- - streaming
24
- pipeline_tag: automatic-speech-recognition
25
  ---
 
26
 
27
- # Parakeet-TDT-0.6B-v3 — LiteRT (TFLite) port
28
-
29
- LiteRT (TFLite) port of
30
- [`nvidia/parakeet-tdt-0.6b-v3`](https://huggingface.co/nvidia/parakeet-tdt-0.6b-v3),
31
- packaged for on-device inference (Android / Mac / embedded) without a Python
32
- or NeMo runtime dependency.
33
-
34
- For **model capabilities, languages, training data, license, and benchmarks**,
35
- see the upstream model card. This card only documents what's specific to the
36
- LiteRT port.
37
-
38
- ## What's in this bundle
39
-
40
- | File | Size | Purpose |
41
- |---|---|---|
42
- | `encoder_T1500.tflite` | 1.15 GB | FP16 encoder, fixed `T_mel = 1500` (15 s window) |
43
- | `decoder_step.tflite` | 23 MB | Single-step LSTM prediction network |
44
- | `joint_step.tflite` | 12 MB | TDT joint network (token + duration logits) |
45
- | `tokenizer.model` | 353 KB | SentencePiece BPE tokenizer (vocab=8192) |
46
- | `manifest.json` | — | All metadata the runtime needs |
47
-
48
- Total: **~1.18 GB** (FP16). FP32 reference is ~2.37 GB.
49
-
50
- ## Encoder I/O contract
51
-
52
- ```
53
- inputs:
54
- audio_signal : float32 [1, 128, 1500] # log-mel features (NeMo preproc)
55
- length : int32 [1] # actual mel frames used (≤ 1500)
56
- outputs:
57
- encoded : float32 [1, 1024, 188] # 188 = (1500 - 4) // 8
58
- encoded_lengths : int32 [1]
59
- ```
60
-
61
- Pad shorter inputs with zeros at the **tail** (the encoder was trained with
62
- audio anchored at position 0; left-padding causes hallucinations) and pass
63
- the true length.
64
 
65
- The 1500-mel bucket covers ≤ 15 s of audio. For long-form input, run the
66
- encoder in a sliding-window streaming loop — see "Streaming usage" below.
67
 
68
- **Why int32, not int64.** LiteRT's GPU/NPU delegates (LiteRT-CL / OpenCL,
69
- NPU accelerator) reject int64 tensors entirely. With int64 length, every
70
- internal CAST node touching it falls back to CPU, and `CompiledModel.create()`
71
- fails outright on Android with the GPU backend. This bundle is exported with
72
- int32 length end-to-end (input → internal mask arange/comparisons → output
73
- `encoded_lengths`). int32 covers > 2 billion mel frames (~5 hours of audio),
74
- so no practical range loss.
75
 
76
- ## Why a single bucket and not multi-signature
 
 
 
 
 
 
 
 
77
 
78
- An earlier revision shipped a multi-signature encoder with 4 buckets
79
- (300/500/700/1500) sharing weights inside one `.tflite`. The disk savings
80
- were real (~1.2 GB instead of 4.8 GB for 4 separate files), but on Android
81
- the LiteRT `CompiledModel.create()` API prepares **every** signature's
82
- subgraph at load time — each one going through the full delegate-partition
83
- pass. With 4 signatures × ~7 s of XNNPACK / GPU partition prep, app cold
84
- start was ~28 s.
85
 
86
- A single-bucket file is one subgraph: ~7 s init, then ready. If you need
87
- multiple bucket sizes for latency reasons, ship them as separate `.tflite`
88
- files (TFLite has no cross-file weight sharing) and load on demand.
89
 
90
- ## Decoder + joint contract
 
91
 
 
 
 
 
 
 
 
 
 
92
  ```
93
- decoder_step:
94
- inputs: token int64 [1,1], h float32 [2,1,640], c float32 [2,1,640]
95
- outputs: g float32 [1,1,640], h float32 [2,1,640], c float32 [2,1,640]
96
-
97
- joint_step:
98
- inputs: enc_frame float32 [1,1024,1], pred_frame float32 [1,640,1]
99
- outputs: logits float32 [1,1,1,8198]
100
- # logits[..., 0:8193] → token logits (8192 BPE + 1 blank)
101
- # logits[..., 8193:8198] → duration logits over [0,1,2,3,4]
102
- ```
103
-
104
- `decoder_step.token` is `int64` because it's an embedding lookup; that op
105
- runs on CPU regardless of delegate, so int64 there is harmless.
106
-
107
- Greedy TDT decoding (per encoder frame):
108
 
109
- 1. Run joint with current `enc_frame` and last predicted `pred_frame`.
110
- 2. `token = argmax(token_logits)`; `dur = durations[argmax(duration_logits)] ∈ {0,1,2,3,4}`.
111
- 3. If `token != blank_id (8192)`: emit token, advance `dur` encoder frames,
112
- re-prime decoder with the emitted token (h, c update).
113
- 4. Else: advance `max(dur, 1)` encoder frames; do not advance the decoder.
114
- 5. Repeat until `enc_lengths` is exhausted.
115
 
116
- Cap at ~10 non-blank emissions per encoder frame to guard against the
117
- pathological `dur=0` decode loop.
118
 
119
- ## Audio preprocessing
 
 
 
 
120
 
121
- LiteRT itself does not produce mel features — your runtime must compute
122
- them. Match NeMo's preprocessor exactly:
123
-
124
- ```
125
- sample_rate : 16000 Hz (resample if needed)
126
- n_fft : 512
127
- hop_length : 160 → 100 mel frames / second
128
- win_length : 400
129
- n_mels : 128
130
- preemph : 0.97
131
- log : log(mel + 1e-5), per-feature normalized
132
- mel_scale : slaney
133
- ```
134
 
135
- Encoder frame rate after the 8× subsampler: **12.5 fps** (1 enc frame = 80 ms).
136
 
137
- ## Streaming usage
138
 
139
- This bundle supports chunked streaming inference using a left+chunk+right
140
- context window that fits inside 15 s. A reference Python implementation is
141
- in the upstream repo (`transcribe_litert_streaming.py`). Recommended config
142
- for Android UX:
143
 
144
- | Knob | Value | Reason |
145
- |---|---|---|
146
- | `chunk_seconds` | 5 | committed per step |
147
- | `left_context_seconds` | 5 | encoder bilateral context |
148
- | `right_context_seconds` | 2 | end-to-end latency ≈ 7 s |
149
- | `window total` | 12 s | (5 + 5 + 2) × 100 = 1200 mel ≤ 1500 |
150
- | `carry_state` | false | offline-trained model; carrying LSTM state across chunks tends to hurt |
151
 
152
- We measured ~27 % WER on multilingual long-form audio (EN/ES/IT
153
- code-switching) with this config, ~22 % on clean offline ≤15 s English.
154
 
155
- ## Quantization
156
 
157
- - All `.tflite` weights are FP16. Activations remain FP32.
158
- - Bit-identical token output vs the upstream FP32 model on a 99-clip eval
159
- set.
160
 
161
- ## Conversion provenance
162
 
163
- Built from upstream `nvidia/parakeet-tdt-0.6b-v3.nemo` via:
164
 
165
- 1. **NeMo → torch.export ExportedProgram** (per encoder/decoder/joint module).
166
- 2. **ExportedProgram → TFLite** via
167
- [`litert-torch`](https://github.com/google-ai-edge/LiteRT) 0.8.0.
168
- 3. **FP32 → FP16** via `ai_edge_quantizer` `FLOAT_CASTING` algorithm on
169
- FC / Conv / DepthwiseConv / ConvTranspose / EmbeddingLookup ops.
170
 
171
- Several NeMo internals required export-time monkey-patches:
172
 
173
- - `MaskedConvSequential.{forward,_create_mask}` and `apply_channel_mask` — to
174
- remove `.expand(...)` patterns rejected by the TFLite broadcast checker.
175
- - `RelPositionMultiHeadAttentionLongformer._get_invalid_locations_mask` — to
176
- build masks in `bool` instead of `uint8` (litert-torch has no uint8
177
- lowering).
178
- - `ConformerEncoder.{forward_internal,_create_masks}` and
179
- `MaskedConvSequential.{forward,_create_mask}` — to keep the entire length
180
- pipeline in `int32` instead of NeMo's default `int64`, so LiteRT's
181
- GPU/NPU delegates can compile the graph without falling back to CPU.
182
 
183
- ## Limitations
184
-
185
- 1. **Audio at position 0.** The encoder expects audio anchored at the start
186
- of its input window. Padding before the audio causes hallucinations.
187
- 2. **15 s max per call.** Use the streaming chunker for longer clips.
188
- 3. **No VAD or diarization.** Pair with an external VAD or a diarizer
189
- (e.g. Sortformer) for speaker-attributed transcripts.
190
- 4. **Multilingual but no language token.** Code-switching works, but the
191
- model doesn't emit a language ID. Run a separate classifier if you need it.
192
-
193
- ## License
194
-
195
- Inherits the upstream `nvidia/parakeet-tdt-0.6b-v3` license (CC-BY-4.0).
196
-
197
- ## Citation
198
-
199
- ```bibtex
200
- @misc{nvidia_parakeet_tdt_0_6b_v3,
201
- title = {Parakeet-TDT-0.6B-v3},
202
- author = {NVIDIA},
203
- year = {2025},
204
- url = {https://huggingface.co/nvidia/parakeet-tdt-0.6b-v3},
205
- }
206
- ```
 
9
  - pt
10
  library_name: litert
11
  base_model: nvidia/parakeet-tdt-0.6b-v3
12
+ pipeline_tag: automatic-speech-recognition
13
  tags:
 
14
  - speech
15
  - audio
16
  - parakeet
17
  - tdt
 
18
  - tflite
19
+ - int8
 
20
  - android
21
+ - on-device
22
+ - buffered-streaming
23
  ---
24
+ # Parakeet-TDT-0.6B-v3 — INT8 LiteRT
25
 
26
+ Community conversion of [NVIDIA Parakeet-TDT-0.6B-v3](https://huggingface.co/nvidia/parakeet-tdt-0.6b-v3), with a dynamic INT8 encoder and the original FP16 decoder/joint. No additional training. This is not an official NVIDIA or Google release.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
27
 
28
+ **The repository root now contains the INT8 release.** The previous FP16 release, including its original ZIP and documentation, is preserved byte-for-byte in [legacy/](https://huggingface.co/spybyscript/parakeet-tdt-litert/tree/main/legacy). See [migration and integration instructions](INTEGRATION.md).
 
29
 
30
+ ## Files and download
 
 
 
 
 
 
31
 
32
+ | File | Size, decimal MB | Purpose |
33
+ |---|---:|---|
34
+ | [encoder_T1500.tflite](encoder_T1500.tflite) | 628.6 | Dynamic INT8 encoder; one fixed 15-second input bucket |
35
+ | [decoder_step.tflite](decoder_step.tflite) | 23.7 | Original FP16 LSTM prediction network |
36
+ | [joint_step.tflite](joint_step.tflite) | 12.7 | Original FP16 token/duration joint network |
37
+ | [tokenizer.model](tokenizer.model) | 0.4 | Unchanged SentencePiece tokenizer |
38
+ | [manifest.json](manifest.json) | — | Tensor contracts, profile, component hashes and provenance |
39
+ | [validation.json](validation.json) | — | Measured quality and memory results, with limitations |
40
+ | [mapped_model_file.h](mapped_model_file.h) | — | Optional native source-page memory hint; Linux-tested example |
41
 
42
+ The four runtime assets total approximately **665 MB on disk**. Download them individually, or use the updated [ZIP bundle](https://huggingface.co/spybyscript/parakeet-tdt-litert/resolve/main/parakeet-tdt-0.6b-v3-litert.zip). Root [SHA256SUMS](SHA256SUMS) includes the ZIP checksum; the ZIP contains component checksums in its manifest.
 
 
 
 
 
 
43
 
44
+ To avoid downloading the legacy release and duplicate ZIPs, select root files explicitly:
 
 
45
 
46
+ ```python
47
+ from huggingface_hub import snapshot_download
48
 
49
+ snapshot_download(
50
+ "spybyscript/parakeet-tdt-litert",
51
+ local_dir="parakeet-int8",
52
+ allow_patterns=[
53
+ "encoder_T1500.tflite", "decoder_step.tflite", "joint_step.tflite",
54
+ "tokenizer.model", "manifest.json", "validation.json",
55
+ "README.md", "INTEGRATION.md", "NOTICE.md", "mapped_model_file.h",
56
+ ],
57
+ )
58
  ```
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
59
 
60
+ Pin a repository revision in production. The root encoder filename is unchanged, but its content and precision have changed; invalidate caches using the manifest hashes or release ID `20260914-int8-t1500-v1`.
 
 
 
 
 
61
 
62
+ ## Measured memory
 
63
 
64
+ | Linux CPU configuration | Process peak | Resident memory with all models retained |
65
+ |---|---:|---:|
66
+ | Previous FP16 encoder | 4.30 GB | — |
67
+ | INT8, ordinary runtime | 1.60 GB | — |
68
+ | INT8, verified native source-page hint after warmup | 1.46 GB | 0.954 GB RSS / 0.947 GB PSS |
69
 
70
+ All figures use decimal GB and include encoder, decoder and joint in an isolated ASR process. They exclude the Android UI, capture frontend and other ML models. Runtime: LiteRT 2.1.2/XNNPACK, four encoder threads and one decoder/joint thread. **Android, GPU and NPU memory/performance have not been validated for this release.**
 
 
 
 
 
 
 
 
 
 
 
 
71
 
72
+ The approximately 954 MB result requires the optional file-mapping optimization described in [INTEGRATION.md](INTEGRATION.md). It is not automatic when downloading this model and is not a Java `CompiledModel` setting. Startup memory remains separately reported; clean source pages can remain in the OS file cache until reclaimed.
73
 
74
+ ## Recognition validation
75
 
76
+ On 33 English synthetic clips with 745 source-text reference words, INT8 scored **4.30% word error**, versus **4.03%** for the prior FP16 baseline. These references were not independently transcribed by humans. INT8 matched 29/33 complete token sequences and 31/33 normalized word sequences; it is not an exact-parity replacement for FP16.
 
 
 
77
 
78
+ Four additional human long recordings totaled 159.758 seconds. They lack complete reference transcripts, so no human-reference WER is claimed. The native source-page hint preserved **all 37/37 token sequences versus ordinary INT8**. That parity claim concerns the memory hint, not quantization versus FP16.
 
 
 
 
 
 
79
 
80
+ Those four long recordings took about 12.18 seconds of ordinary INT8 compute, or 11.42 seconds with the hint, versus 31.94 seconds for FP16. Frontend calculation, loading and warmup are excluded; other work was running on the machine. These timings are observations, not a controlled speed ranking or phone benchmark. The base model is multilingual; the current quantization evaluation was English only. Private audio, transcripts and token sequences are not distributed.
 
81
 
82
+ ## Buffered live use
83
 
84
+ This is a component bundle with a host-side TDT decoder, not a single audio-to-text graph. It can process a continuous audio source using overlapping windows. The tested profile commits 4.96 seconds, with 4.96 seconds of left context and 2 seconds of right context, padded to the 15-second capacity.
 
 
85
 
86
+ Expect approximately **7 seconds of initial buffering and updates every 5 seconds**, plus compute. The 15-second input capacity does not mean waiting 15 seconds for every result. This is buffered processing of an offline-trained encoder; it does not have Nemotron's cache-aware encoder or frequent immediate partials. Boundary repetitions and recognition errors remain possible. Neither VAD nor speaker diarization is included.
87
 
88
+ ## Precision and provenance
89
 
90
+ The encoder has 193 fully connected, 50 Conv2D and 27 depthwise-convolution operations with INT8 weights. Dynamic integer kernels consume floating-point activations/interfaces; this is not a fully integer end-to-end graph. The unchanged decoder/joint use FP16 weight storage and floating-point interfaces. Start with CPU/XNNPACK and test other backends independently.
 
 
 
 
91
 
92
+ Conversion used the existing staged Parakeet program, PyTorch 2.11.0, litert-torch 0.9.1 and AI Edge Quantizer 0.7.0. The staged source-program SHA256 and runtime component hashes are in the manifest. INT4 alternatives were investigated locally but are not included in this default release because they changed more long-form words and omitted some phrases.
93
 
94
+ ## License and attribution
 
 
 
 
 
 
 
 
95
 
96
+ Model weights derive from NVIDIA Parakeet-TDT-0.6B-v3 under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/). Conversion and quantization by spybyscript; see [NOTICE.md](NOTICE.md). Consult the upstream card for base-model capabilities and training information. Upstream accuracy benchmarks are not measurements of this conversion.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
SHA256SUMS ADDED
@@ -0,0 +1,14 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ 11ad7efa24975ee4b0c3c3a38ed18737f0658a5f75a0a96787b576a78a023361 .gitattributes
2
+ 510b2919183633eb53418a6c0ef65cf6b72ac6e04845893e3a67f70d840a2ccb INTEGRATION.md
3
+ 5bf6b396ccf5b900a009f166a33d2464ba0fdce0ec117ad7f002cf5dc66ee14e NOTICE.md
4
+ c0dac492e179384912d4a623522d349d7f212a0012d5bc1db54c52adfea3cef4 README.md
5
+ eb0bf3559a0b4cbdc3ca05b7e8ff948ee5ef158ce424667b62a85f6c769a9ce1 decoder_step.tflite
6
+ b26804686f98d68f7394baf3b13bdcea8e846ebf4ecf2903e3fc30fe3a3882a8 encoder_T1500.tflite
7
+ 0e28c22fc426df9900ef4a1bd15760ec757e44f0fd1818e0afb51c4fe79031be joint_step.tflite
8
+ 2c6b2684937eaa67b2b6e68a0ad4aff96c0a114e8a135435de554822b258f3d0 legacy/archive.json
9
+ 12bc7c5c126db8f7f97b617eb151d8e5b14d0ee3d57244f952693faa2a8fde98 manifest.json
10
+ 4226d273861baa0328a5dcf8ef5d9eba604074137111ba53116f709ba63399c0 mapped_model_file.h
11
+ cd49232e6756b62ba14049bb31bab799838e30170b235a6993888e57662fdc95 parakeet-tdt-0.6b-v3-litert.zip
12
+ 8efdaeebf2ead197dc5a265e07be3b184cce31056d53a2315a6a3e535f4dbcd3 publication.json
13
+ eacec2b0a77f336d4a2ca4a25a7047575d3c2b74de47e997f4c205126ed3135e tokenizer.model
14
+ f869d60f54a5d1cfbe2f3c73e2a1f3845fccf64109decc6cdd12345a89862b16 validation.json
encoder_T1500.tflite CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:8f826f40ab8e50c0e05ff1532d1cedcaa9c79c34909f1c14d9ac86c0643db45b
3
- size 1206789364
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b26804686f98d68f7394baf3b13bdcea8e846ebf4ecf2903e3fc30fe3a3882a8
3
+ size 628579360
legacy/.gitattributes ADDED
@@ -0,0 +1,35 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ *.7z filter=lfs diff=lfs merge=lfs -text
2
+ *.arrow filter=lfs diff=lfs merge=lfs -text
3
+ *.bin filter=lfs diff=lfs merge=lfs -text
4
+ *.bz2 filter=lfs diff=lfs merge=lfs -text
5
+ *.ckpt filter=lfs diff=lfs merge=lfs -text
6
+ *.ftz filter=lfs diff=lfs merge=lfs -text
7
+ *.gz filter=lfs diff=lfs merge=lfs -text
8
+ *.h5 filter=lfs diff=lfs merge=lfs -text
9
+ *.joblib filter=lfs diff=lfs merge=lfs -text
10
+ *.lfs.* filter=lfs diff=lfs merge=lfs -text
11
+ *.mlmodel filter=lfs diff=lfs merge=lfs -text
12
+ *.model filter=lfs diff=lfs merge=lfs -text
13
+ *.msgpack filter=lfs diff=lfs merge=lfs -text
14
+ *.npy filter=lfs diff=lfs merge=lfs -text
15
+ *.npz filter=lfs diff=lfs merge=lfs -text
16
+ *.onnx filter=lfs diff=lfs merge=lfs -text
17
+ *.ot filter=lfs diff=lfs merge=lfs -text
18
+ *.parquet filter=lfs diff=lfs merge=lfs -text
19
+ *.pb filter=lfs diff=lfs merge=lfs -text
20
+ *.pickle filter=lfs diff=lfs merge=lfs -text
21
+ *.pkl filter=lfs diff=lfs merge=lfs -text
22
+ *.pt filter=lfs diff=lfs merge=lfs -text
23
+ *.pth filter=lfs diff=lfs merge=lfs -text
24
+ *.rar filter=lfs diff=lfs merge=lfs -text
25
+ *.safetensors filter=lfs diff=lfs merge=lfs -text
26
+ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
27
+ *.tar.* filter=lfs diff=lfs merge=lfs -text
28
+ *.tar filter=lfs diff=lfs merge=lfs -text
29
+ *.tflite filter=lfs diff=lfs merge=lfs -text
30
+ *.tgz filter=lfs diff=lfs merge=lfs -text
31
+ *.wasm filter=lfs diff=lfs merge=lfs -text
32
+ *.xz filter=lfs diff=lfs merge=lfs -text
33
+ *.zip filter=lfs diff=lfs merge=lfs -text
34
+ *.zst filter=lfs diff=lfs merge=lfs -text
35
+ *tfevents* filter=lfs diff=lfs merge=lfs -text
legacy/README.md ADDED
@@ -0,0 +1,206 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: cc-by-4.0
3
+ language:
4
+ - en
5
+ - es
6
+ - it
7
+ - de
8
+ - fr
9
+ - pt
10
+ library_name: litert
11
+ base_model: nvidia/parakeet-tdt-0.6b-v3
12
+ tags:
13
+ - automatic-speech-recognition
14
+ - speech
15
+ - audio
16
+ - parakeet
17
+ - tdt
18
+ - litert
19
+ - tflite
20
+ - on-device
21
+ - mobile
22
+ - android
23
+ - streaming
24
+ pipeline_tag: automatic-speech-recognition
25
+ ---
26
+
27
+ # Parakeet-TDT-0.6B-v3 — LiteRT (TFLite) port
28
+
29
+ LiteRT (TFLite) port of
30
+ [`nvidia/parakeet-tdt-0.6b-v3`](https://huggingface.co/nvidia/parakeet-tdt-0.6b-v3),
31
+ packaged for on-device inference (Android / Mac / embedded) without a Python
32
+ or NeMo runtime dependency.
33
+
34
+ For **model capabilities, languages, training data, license, and benchmarks**,
35
+ see the upstream model card. This card only documents what's specific to the
36
+ LiteRT port.
37
+
38
+ ## What's in this bundle
39
+
40
+ | File | Size | Purpose |
41
+ |---|---|---|
42
+ | `encoder_T1500.tflite` | 1.15 GB | FP16 encoder, fixed `T_mel = 1500` (15 s window) |
43
+ | `decoder_step.tflite` | 23 MB | Single-step LSTM prediction network |
44
+ | `joint_step.tflite` | 12 MB | TDT joint network (token + duration logits) |
45
+ | `tokenizer.model` | 353 KB | SentencePiece BPE tokenizer (vocab=8192) |
46
+ | `manifest.json` | — | All metadata the runtime needs |
47
+
48
+ Total: **~1.18 GB** (FP16). FP32 reference is ~2.37 GB.
49
+
50
+ ## Encoder I/O contract
51
+
52
+ ```
53
+ inputs:
54
+ audio_signal : float32 [1, 128, 1500] # log-mel features (NeMo preproc)
55
+ length : int32 [1] # actual mel frames used (≤ 1500)
56
+ outputs:
57
+ encoded : float32 [1, 1024, 188] # 188 = (1500 - 4) // 8
58
+ encoded_lengths : int32 [1]
59
+ ```
60
+
61
+ Pad shorter inputs with zeros at the **tail** (the encoder was trained with
62
+ audio anchored at position 0; left-padding causes hallucinations) and pass
63
+ the true length.
64
+
65
+ The 1500-mel bucket covers ≤ 15 s of audio. For long-form input, run the
66
+ encoder in a sliding-window streaming loop — see "Streaming usage" below.
67
+
68
+ **Why int32, not int64.** LiteRT's GPU/NPU delegates (LiteRT-CL / OpenCL,
69
+ NPU accelerator) reject int64 tensors entirely. With int64 length, every
70
+ internal CAST node touching it falls back to CPU, and `CompiledModel.create()`
71
+ fails outright on Android with the GPU backend. This bundle is exported with
72
+ int32 length end-to-end (input → internal mask arange/comparisons → output
73
+ `encoded_lengths`). int32 covers > 2 billion mel frames (~5 hours of audio),
74
+ so no practical range loss.
75
+
76
+ ## Why a single bucket and not multi-signature
77
+
78
+ An earlier revision shipped a multi-signature encoder with 4 buckets
79
+ (300/500/700/1500) sharing weights inside one `.tflite`. The disk savings
80
+ were real (~1.2 GB instead of 4.8 GB for 4 separate files), but on Android
81
+ the LiteRT `CompiledModel.create()` API prepares **every** signature's
82
+ subgraph at load time — each one going through the full delegate-partition
83
+ pass. With 4 signatures × ~7 s of XNNPACK / GPU partition prep, app cold
84
+ start was ~28 s.
85
+
86
+ A single-bucket file is one subgraph: ~7 s init, then ready. If you need
87
+ multiple bucket sizes for latency reasons, ship them as separate `.tflite`
88
+ files (TFLite has no cross-file weight sharing) and load on demand.
89
+
90
+ ## Decoder + joint contract
91
+
92
+ ```
93
+ decoder_step:
94
+ inputs: token int64 [1,1], h float32 [2,1,640], c float32 [2,1,640]
95
+ outputs: g float32 [1,1,640], h float32 [2,1,640], c float32 [2,1,640]
96
+
97
+ joint_step:
98
+ inputs: enc_frame float32 [1,1024,1], pred_frame float32 [1,640,1]
99
+ outputs: logits float32 [1,1,1,8198]
100
+ # logits[..., 0:8193] → token logits (8192 BPE + 1 blank)
101
+ # logits[..., 8193:8198] → duration logits over [0,1,2,3,4]
102
+ ```
103
+
104
+ `decoder_step.token` is `int64` because it's an embedding lookup; that op
105
+ runs on CPU regardless of delegate, so int64 there is harmless.
106
+
107
+ Greedy TDT decoding (per encoder frame):
108
+
109
+ 1. Run joint with current `enc_frame` and last predicted `pred_frame`.
110
+ 2. `token = argmax(token_logits)`; `dur = durations[argmax(duration_logits)] ∈ {0,1,2,3,4}`.
111
+ 3. If `token != blank_id (8192)`: emit token, advance `dur` encoder frames,
112
+ re-prime decoder with the emitted token (h, c update).
113
+ 4. Else: advance `max(dur, 1)` encoder frames; do not advance the decoder.
114
+ 5. Repeat until `enc_lengths` is exhausted.
115
+
116
+ Cap at ~10 non-blank emissions per encoder frame to guard against the
117
+ pathological `dur=0` decode loop.
118
+
119
+ ## Audio preprocessing
120
+
121
+ LiteRT itself does not produce mel features — your runtime must compute
122
+ them. Match NeMo's preprocessor exactly:
123
+
124
+ ```
125
+ sample_rate : 16000 Hz (resample if needed)
126
+ n_fft : 512
127
+ hop_length : 160 → 100 mel frames / second
128
+ win_length : 400
129
+ n_mels : 128
130
+ preemph : 0.97
131
+ log : log(mel + 1e-5), per-feature normalized
132
+ mel_scale : slaney
133
+ ```
134
+
135
+ Encoder frame rate after the 8× subsampler: **12.5 fps** (1 enc frame = 80 ms).
136
+
137
+ ## Streaming usage
138
+
139
+ This bundle supports chunked streaming inference using a left+chunk+right
140
+ context window that fits inside 15 s. A reference Python implementation is
141
+ in the upstream repo (`transcribe_litert_streaming.py`). Recommended config
142
+ for Android UX:
143
+
144
+ | Knob | Value | Reason |
145
+ |---|---|---|
146
+ | `chunk_seconds` | 5 | committed per step |
147
+ | `left_context_seconds` | 5 | encoder bilateral context |
148
+ | `right_context_seconds` | 2 | end-to-end latency ≈ 7 s |
149
+ | `window total` | 12 s | (5 + 5 + 2) × 100 = 1200 mel ≤ 1500 |
150
+ | `carry_state` | false | offline-trained model; carrying LSTM state across chunks tends to hurt |
151
+
152
+ We measured ~27 % WER on multilingual long-form audio (EN/ES/IT
153
+ code-switching) with this config, ~22 % on clean offline ≤15 s English.
154
+
155
+ ## Quantization
156
+
157
+ - All `.tflite` weights are FP16. Activations remain FP32.
158
+ - Bit-identical token output vs the upstream FP32 model on a 99-clip eval
159
+ set.
160
+
161
+ ## Conversion provenance
162
+
163
+ Built from upstream `nvidia/parakeet-tdt-0.6b-v3.nemo` via:
164
+
165
+ 1. **NeMo → torch.export ExportedProgram** (per encoder/decoder/joint module).
166
+ 2. **ExportedProgram → TFLite** via
167
+ [`litert-torch`](https://github.com/google-ai-edge/LiteRT) 0.8.0.
168
+ 3. **FP32 → FP16** via `ai_edge_quantizer` `FLOAT_CASTING` algorithm on
169
+ FC / Conv / DepthwiseConv / ConvTranspose / EmbeddingLookup ops.
170
+
171
+ Several NeMo internals required export-time monkey-patches:
172
+
173
+ - `MaskedConvSequential.{forward,_create_mask}` and `apply_channel_mask` — to
174
+ remove `.expand(...)` patterns rejected by the TFLite broadcast checker.
175
+ - `RelPositionMultiHeadAttentionLongformer._get_invalid_locations_mask` — to
176
+ build masks in `bool` instead of `uint8` (litert-torch has no uint8
177
+ lowering).
178
+ - `ConformerEncoder.{forward_internal,_create_masks}` and
179
+ `MaskedConvSequential.{forward,_create_mask}` — to keep the entire length
180
+ pipeline in `int32` instead of NeMo's default `int64`, so LiteRT's
181
+ GPU/NPU delegates can compile the graph without falling back to CPU.
182
+
183
+ ## Limitations
184
+
185
+ 1. **Audio at position 0.** The encoder expects audio anchored at the start
186
+ of its input window. Padding before the audio causes hallucinations.
187
+ 2. **15 s max per call.** Use the streaming chunker for longer clips.
188
+ 3. **No VAD or diarization.** Pair with an external VAD or a diarizer
189
+ (e.g. Sortformer) for speaker-attributed transcripts.
190
+ 4. **Multilingual but no language token.** Code-switching works, but the
191
+ model doesn't emit a language ID. Run a separate classifier if you need it.
192
+
193
+ ## License
194
+
195
+ Inherits the upstream `nvidia/parakeet-tdt-0.6b-v3` license (CC-BY-4.0).
196
+
197
+ ## Citation
198
+
199
+ ```bibtex
200
+ @misc{nvidia_parakeet_tdt_0_6b_v3,
201
+ title = {Parakeet-TDT-0.6B-v3},
202
+ author = {NVIDIA},
203
+ year = {2025},
204
+ url = {https://huggingface.co/nvidia/parakeet-tdt-0.6b-v3},
205
+ }
206
+ ```
legacy/archive.json ADDED
@@ -0,0 +1,55 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "source_repo": "spybyscript/parakeet-tdt-litert",
3
+ "source_revision": "c528ceb57bbc4e366993e50c79b22e0417a68484",
4
+ "layout": "Original release files preserved byte-for-byte under legacy/",
5
+ "files": [
6
+ {
7
+ "path": ".gitattributes",
8
+ "bytes": 1519,
9
+ "blob_id": "a6344aac8c09253b3b630fb776ae94478aa0275b",
10
+ "lfs_sha256": null
11
+ },
12
+ {
13
+ "path": "README.md",
14
+ "bytes": 7545,
15
+ "blob_id": "99bedadd0d487c3777086d131e2f4f06294fe1eb",
16
+ "lfs_sha256": null
17
+ },
18
+ {
19
+ "path": "decoder_step.tflite",
20
+ "bytes": 23650084,
21
+ "blob_id": "ef4d5b35b1e08b89be5a45d16632197369b95dd2",
22
+ "lfs_sha256": "eb0bf3559a0b4cbdc3ca05b7e8ff948ee5ef158ce424667b62a85f6c769a9ce1"
23
+ },
24
+ {
25
+ "path": "encoder_T1500.tflite",
26
+ "bytes": 1206789364,
27
+ "blob_id": "f97b8f24f9759ef4963a1d08dc057a2b31ebb032",
28
+ "lfs_sha256": "8f826f40ab8e50c0e05ff1532d1cedcaa9c79c34909f1c14d9ac86c0643db45b"
29
+ },
30
+ {
31
+ "path": "joint_step.tflite",
32
+ "bytes": 12664976,
33
+ "blob_id": "83121aa7742953278fe8d41a5319153725644348",
34
+ "lfs_sha256": "0e28c22fc426df9900ef4a1bd15760ec757e44f0fd1818e0afb51c4fe79031be"
35
+ },
36
+ {
37
+ "path": "manifest.json",
38
+ "bytes": 6294,
39
+ "blob_id": "6fe030289b5aad6912c07247c5e479e055c37150",
40
+ "lfs_sha256": null
41
+ },
42
+ {
43
+ "path": "parakeet-tdt-0.6b-v3-litert.zip",
44
+ "bytes": 1243526688,
45
+ "blob_id": "8a122cdd57d101647c43d375eb8fcd02af8fbb47",
46
+ "lfs_sha256": "da306f15d3835e939fe8585933709646c01621f4570f3186121cb435d50fd1ac"
47
+ },
48
+ {
49
+ "path": "tokenizer.model",
50
+ "bytes": 360916,
51
+ "blob_id": "60c11e1e1ac25a6514cb2d27cf8c1a7590ed2833",
52
+ "lfs_sha256": "eacec2b0a77f336d4a2ca4a25a7047575d3c2b74de47e997f4c205126ed3135e"
53
+ }
54
+ ]
55
+ }
legacy/decoder_step.tflite ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:eb0bf3559a0b4cbdc3ca05b7e8ff948ee5ef158ce424667b62a85f6c769a9ce1
3
+ size 23650084
legacy/encoder_T1500.tflite ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:8f826f40ab8e50c0e05ff1532d1cedcaa9c79c34909f1c14d9ac86c0643db45b
3
+ size 1206789364
legacy/joint_step.tflite ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0e28c22fc426df9900ef4a1bd15760ec757e44f0fd1818e0afb51c4fe79031be
3
+ size 12664976
legacy/manifest.json ADDED
@@ -0,0 +1,345 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "model": "nvidia/parakeet-tdt-0.6b-v3",
3
+ "torch_version": "2.11.0+cu130",
4
+ "model_class": "EncDecRNNTBPEModel",
5
+ "vocab_size": 8192,
6
+ "blank_id": 8192,
7
+ "durations": [
8
+ 0,
9
+ 1,
10
+ 2,
11
+ 3,
12
+ 4
13
+ ],
14
+ "num_durations": 5,
15
+ "joint_output_dim": 8198,
16
+ "joint_token_logits_slice": [
17
+ 0,
18
+ 8193
19
+ ],
20
+ "joint_duration_logits_slice": [
21
+ 8193,
22
+ 8198
23
+ ],
24
+ "encoder": {
25
+ "d_model": 1024,
26
+ "subsampling_factor": 8,
27
+ "n_layers": 24,
28
+ "n_heads": 8,
29
+ "feat_in": 128,
30
+ "attention_mode": "rel_pos",
31
+ "att_context_size": null,
32
+ "buckets": [
33
+ {
34
+ "n_mel_frames": 1500,
35
+ "n_encoder_frames": 187,
36
+ "input_shape": [
37
+ 1,
38
+ 128,
39
+ 1500
40
+ ],
41
+ "output_shape": [
42
+ 1,
43
+ 1024,
44
+ 188
45
+ ],
46
+ "artifact": "encoder_T1500.pt2",
47
+ "size_mb": 2366.85
48
+ }
49
+ ],
50
+ "multisig": false
51
+ },
52
+ "decoder": {
53
+ "num_layers": 2,
54
+ "hidden": 640,
55
+ "embed_dim": 640
56
+ },
57
+ "joint": {
58
+ "d_enc": 1024,
59
+ "d_pred": 640,
60
+ "joint_dim": 640
61
+ },
62
+ "preprocessor": {
63
+ "sample_rate": 16000,
64
+ "n_fft": 512,
65
+ "win_length": 400,
66
+ "hop_length": 160,
67
+ "n_mels": 128,
68
+ "preemph": 0.97,
69
+ "log": true,
70
+ "frame_rate_hz_post_subsample": 12.5
71
+ },
72
+ "artifacts": {
73
+ "decoder_step": {
74
+ "filename": "decoder_step.pt2",
75
+ "size_mb": 45.07,
76
+ "input_shapes": {
77
+ "token": [
78
+ 1,
79
+ 1
80
+ ],
81
+ "h": [
82
+ 2,
83
+ 1,
84
+ 640
85
+ ],
86
+ "c": [
87
+ 2,
88
+ 1,
89
+ 640
90
+ ]
91
+ },
92
+ "input_dtypes": {
93
+ "token": "int64",
94
+ "h": "float32",
95
+ "c": "float32"
96
+ },
97
+ "output_shapes": {
98
+ "g": [
99
+ 1,
100
+ 1,
101
+ 640
102
+ ],
103
+ "h": [
104
+ 2,
105
+ 1,
106
+ 640
107
+ ],
108
+ "c": [
109
+ 2,
110
+ 1,
111
+ 640
112
+ ]
113
+ }
114
+ },
115
+ "joint_step": {
116
+ "filename": "joint_step.pt2",
117
+ "size_mb": 24.14,
118
+ "input_shapes": {
119
+ "enc_frame": [
120
+ 1,
121
+ 1024,
122
+ 1
123
+ ],
124
+ "pred_frame": [
125
+ 1,
126
+ 640,
127
+ 1
128
+ ]
129
+ },
130
+ "output_shape": [
131
+ 1,
132
+ 1,
133
+ 1,
134
+ 8198
135
+ ]
136
+ }
137
+ },
138
+ "tokenizer": {
139
+ "saved": true,
140
+ "method": "serialized_model_proto",
141
+ "vocab_size": 8192
142
+ },
143
+ "litert": {
144
+ "quant": "fp16",
145
+ "results": [
146
+ {
147
+ "graph": "encoder",
148
+ "source_artifact": "encoder_T1500.pt2",
149
+ "output_artifact": "encoder_T1500.tflite",
150
+ "size_mb": 1150.88,
151
+ "convert_seconds": 158.59,
152
+ "quant": "fp16",
153
+ "torch_output_shapes": [
154
+ [
155
+ 1,
156
+ 1024,
157
+ 188
158
+ ],
159
+ [
160
+ 1
161
+ ]
162
+ ],
163
+ "parity": {
164
+ "ok": true,
165
+ "max_abs_diff": 0.0,
166
+ "per_output_diffs": [
167
+ [
168
+ "shape mismatch",
169
+ [
170
+ 1
171
+ ],
172
+ [
173
+ 1,
174
+ 1024,
175
+ 188
176
+ ]
177
+ ],
178
+ [
179
+ "shape mismatch",
180
+ [
181
+ 1,
182
+ 1024,
183
+ 188
184
+ ],
185
+ [
186
+ 1
187
+ ]
188
+ ]
189
+ ],
190
+ "tflite_output_shapes": [
191
+ [
192
+ 1
193
+ ],
194
+ [
195
+ 1,
196
+ 1024,
197
+ 188
198
+ ]
199
+ ],
200
+ "torch_output_shapes": [
201
+ [
202
+ 1,
203
+ 1024,
204
+ 188
205
+ ],
206
+ [
207
+ 1
208
+ ]
209
+ ]
210
+ }
211
+ },
212
+ {
213
+ "graph": "decoder_step",
214
+ "source_artifact": "decoder_step.pt2",
215
+ "output_artifact": "decoder_step.tflite",
216
+ "size_mb": 22.55,
217
+ "convert_seconds": 1.92,
218
+ "quant": "fp16",
219
+ "torch_output_shapes": [
220
+ [
221
+ 1,
222
+ 1,
223
+ 640
224
+ ],
225
+ [
226
+ 2,
227
+ 1,
228
+ 640
229
+ ],
230
+ [
231
+ 2,
232
+ 1,
233
+ 640
234
+ ]
235
+ ],
236
+ "parity": {
237
+ "ok": true,
238
+ "max_abs_diff": 0.0044100284576416016,
239
+ "per_output_diffs": [
240
+ [
241
+ "shape mismatch",
242
+ [
243
+ 2,
244
+ 1,
245
+ 640
246
+ ],
247
+ [
248
+ 1,
249
+ 1,
250
+ 640
251
+ ]
252
+ ],
253
+ [
254
+ "shape mismatch",
255
+ [
256
+ 1,
257
+ 1,
258
+ 640
259
+ ],
260
+ [
261
+ 2,
262
+ 1,
263
+ 640
264
+ ]
265
+ ],
266
+ 0.0044100284576416016
267
+ ],
268
+ "tflite_output_shapes": [
269
+ [
270
+ 2,
271
+ 1,
272
+ 640
273
+ ],
274
+ [
275
+ 1,
276
+ 1,
277
+ 640
278
+ ],
279
+ [
280
+ 2,
281
+ 1,
282
+ 640
283
+ ]
284
+ ],
285
+ "torch_output_shapes": [
286
+ [
287
+ 1,
288
+ 1,
289
+ 640
290
+ ],
291
+ [
292
+ 2,
293
+ 1,
294
+ 640
295
+ ],
296
+ [
297
+ 2,
298
+ 1,
299
+ 640
300
+ ]
301
+ ]
302
+ }
303
+ },
304
+ {
305
+ "graph": "joint_step",
306
+ "source_artifact": "joint_step.pt2",
307
+ "output_artifact": "joint_step.tflite",
308
+ "size_mb": 12.08,
309
+ "convert_seconds": 1.61,
310
+ "quant": "fp16",
311
+ "torch_output_shapes": [
312
+ [
313
+ 1,
314
+ 1,
315
+ 1,
316
+ 8198
317
+ ]
318
+ ],
319
+ "parity": {
320
+ "ok": true,
321
+ "max_abs_diff": 0.408447265625,
322
+ "per_output_diffs": [
323
+ 0.408447265625
324
+ ],
325
+ "tflite_output_shapes": [
326
+ [
327
+ 1,
328
+ 1,
329
+ 1,
330
+ 8198
331
+ ]
332
+ ],
333
+ "torch_output_shapes": [
334
+ [
335
+ 1,
336
+ 1,
337
+ 1,
338
+ 8198
339
+ ]
340
+ ]
341
+ }
342
+ }
343
+ ]
344
+ }
345
+ }
legacy/parakeet-tdt-0.6b-v3-litert.zip ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:da306f15d3835e939fe8585933709646c01621f4570f3186121cb435d50fd1ac
3
+ size 1243526688
legacy/tokenizer.model ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:eacec2b0a77f336d4a2ca4a25a7047575d3c2b74de47e997f4c205126ed3135e
3
+ size 360916
manifest.json CHANGED
@@ -1,7 +1,5 @@
1
  {
2
  "model": "nvidia/parakeet-tdt-0.6b-v3",
3
- "torch_version": "2.11.0+cu130",
4
- "model_class": "EncDecRNNTBPEModel",
5
  "vocab_size": 8192,
6
  "blank_id": 8192,
7
  "durations": [
@@ -32,7 +30,7 @@
32
  "buckets": [
33
  {
34
  "n_mel_frames": 1500,
35
- "n_encoder_frames": 187,
36
  "input_shape": [
37
  1,
38
  128,
@@ -43,8 +41,8 @@
43
  1024,
44
  188
45
  ],
46
- "artifact": "encoder_T1500.pt2",
47
- "size_mb": 2366.85
48
  }
49
  ],
50
  "multisig": false
@@ -67,73 +65,15 @@
67
  "n_mels": 128,
68
  "preemph": 0.97,
69
  "log": true,
70
- "frame_rate_hz_post_subsample": 12.5
71
- },
72
- "artifacts": {
73
- "decoder_step": {
74
- "filename": "decoder_step.pt2",
75
- "size_mb": 45.07,
76
- "input_shapes": {
77
- "token": [
78
- 1,
79
- 1
80
- ],
81
- "h": [
82
- 2,
83
- 1,
84
- 640
85
- ],
86
- "c": [
87
- 2,
88
- 1,
89
- 640
90
- ]
91
- },
92
- "input_dtypes": {
93
- "token": "int64",
94
- "h": "float32",
95
- "c": "float32"
96
- },
97
- "output_shapes": {
98
- "g": [
99
- 1,
100
- 1,
101
- 640
102
- ],
103
- "h": [
104
- 2,
105
- 1,
106
- 640
107
- ],
108
- "c": [
109
- 2,
110
- 1,
111
- 640
112
- ]
113
- }
114
- },
115
- "joint_step": {
116
- "filename": "joint_step.pt2",
117
- "size_mb": 24.14,
118
- "input_shapes": {
119
- "enc_frame": [
120
- 1,
121
- 1024,
122
- 1
123
- ],
124
- "pred_frame": [
125
- 1,
126
- 640,
127
- 1
128
- ]
129
- },
130
- "output_shape": [
131
- 1,
132
- 1,
133
- 1,
134
- 8198
135
- ]
136
- }
137
  },
138
  "tokenizer": {
139
  "saved": true,
@@ -141,205 +81,93 @@
141
  "vocab_size": 8192
142
  },
143
  "litert": {
144
- "quant": "fp16",
145
  "results": [
146
  {
147
  "graph": "encoder",
148
- "source_artifact": "encoder_T1500.pt2",
149
- "output_artifact": "encoder_T1500.tflite",
150
- "size_mb": 1150.88,
151
- "convert_seconds": 158.59,
152
- "quant": "fp16",
153
- "torch_output_shapes": [
154
- [
155
- 1,
156
- 1024,
157
- 188
158
- ],
159
- [
160
- 1
161
- ]
162
- ],
163
- "parity": {
164
- "ok": true,
165
- "max_abs_diff": 0.0,
166
- "per_output_diffs": [
167
- [
168
- "shape mismatch",
169
- [
170
- 1
171
- ],
172
- [
173
- 1,
174
- 1024,
175
- 188
176
- ]
177
- ],
178
- [
179
- "shape mismatch",
180
- [
181
- 1,
182
- 1024,
183
- 188
184
- ],
185
- [
186
- 1
187
- ]
188
- ]
189
- ],
190
- "tflite_output_shapes": [
191
- [
192
- 1
193
- ],
194
- [
195
- 1,
196
- 1024,
197
- 188
198
- ]
199
- ],
200
- "torch_output_shapes": [
201
- [
202
- 1,
203
- 1024,
204
- 188
205
- ],
206
- [
207
- 1
208
- ]
209
- ]
210
- }
211
  },
212
  {
213
  "graph": "decoder_step",
214
- "source_artifact": "decoder_step.pt2",
215
- "output_artifact": "decoder_step.tflite",
216
- "size_mb": 22.55,
217
- "convert_seconds": 1.92,
218
- "quant": "fp16",
219
- "torch_output_shapes": [
220
- [
221
- 1,
222
- 1,
223
- 640
224
- ],
225
- [
226
- 2,
227
- 1,
228
- 640
229
- ],
230
- [
231
- 2,
232
- 1,
233
- 640
234
- ]
235
- ],
236
- "parity": {
237
- "ok": true,
238
- "max_abs_diff": 0.0044100284576416016,
239
- "per_output_diffs": [
240
- [
241
- "shape mismatch",
242
- [
243
- 2,
244
- 1,
245
- 640
246
- ],
247
- [
248
- 1,
249
- 1,
250
- 640
251
- ]
252
- ],
253
- [
254
- "shape mismatch",
255
- [
256
- 1,
257
- 1,
258
- 640
259
- ],
260
- [
261
- 2,
262
- 1,
263
- 640
264
- ]
265
- ],
266
- 0.0044100284576416016
267
- ],
268
- "tflite_output_shapes": [
269
- [
270
- 2,
271
- 1,
272
- 640
273
- ],
274
- [
275
- 1,
276
- 1,
277
- 640
278
- ],
279
- [
280
- 2,
281
- 1,
282
- 640
283
- ]
284
- ],
285
- "torch_output_shapes": [
286
- [
287
- 1,
288
- 1,
289
- 640
290
- ],
291
- [
292
- 2,
293
- 1,
294
- 640
295
- ],
296
- [
297
- 2,
298
- 1,
299
- 640
300
- ]
301
- ]
302
- }
303
  },
304
  {
305
  "graph": "joint_step",
306
- "source_artifact": "joint_step.pt2",
307
- "output_artifact": "joint_step.tflite",
308
- "size_mb": 12.08,
309
- "convert_seconds": 1.61,
310
- "quant": "fp16",
311
- "torch_output_shapes": [
312
- [
313
- 1,
314
- 1,
315
- 1,
316
- 8198
317
- ]
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
318
  ],
319
- "parity": {
320
- "ok": true,
321
- "max_abs_diff": 0.408447265625,
322
- "per_output_diffs": [
323
- 0.408447265625
324
- ],
325
- "tflite_output_shapes": [
326
- [
327
- 1,
328
- 1,
329
- 1,
330
- 8198
331
- ]
332
- ],
333
- "torch_output_shapes": [
334
- [
335
- 1,
336
- 1,
337
- 1,
338
- 8198
339
- ]
340
- ]
341
- }
342
  }
343
  ]
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
344
  }
345
  }
 
1
  {
2
  "model": "nvidia/parakeet-tdt-0.6b-v3",
 
 
3
  "vocab_size": 8192,
4
  "blank_id": 8192,
5
  "durations": [
 
30
  "buckets": [
31
  {
32
  "n_mel_frames": 1500,
33
+ "n_encoder_frames": 188,
34
  "input_shape": [
35
  1,
36
  128,
 
41
  1024,
42
  188
43
  ],
44
+ "artifact": "encoder_T1500.tflite",
45
+ "signature": "serving_default"
46
  }
47
  ],
48
  "multisig": false
 
65
  "n_mels": 128,
66
  "preemph": 0.97,
67
  "log": true,
68
+ "frame_rate_hz_post_subsample": 12.5,
69
+ "normalize": "per_feature",
70
+ "window": "hann",
71
+ "window_periodic": false,
72
+ "log_zero_guard_type": "add",
73
+ "log_zero_guard_value": 5.960464477539063e-08,
74
+ "mag_power": 2.0,
75
+ "mel_norm": "slaney",
76
+ "inference_dither": 0.0
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
77
  },
78
  "tokenizer": {
79
  "saved": true,
 
81
  "vocab_size": 8192
82
  },
83
  "litert": {
84
+ "quant": "dynamic_INT8",
85
  "results": [
86
  {
87
  "graph": "encoder",
88
+ "output_artifact": "encoder_T1500.tflite"
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
89
  },
90
  {
91
  "graph": "decoder_step",
92
+ "output_artifact": "decoder_step.tflite"
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
93
  },
94
  {
95
  "graph": "joint_step",
96
+ "output_artifact": "joint_step.tflite"
97
+ }
98
+ ]
99
+ },
100
+ "experimental": true,
101
+ "android_validated": false,
102
+ "inference_profile": {
103
+ "commit_seconds": 4.96,
104
+ "left_context_seconds": 4.96,
105
+ "right_context_seconds": 2.0,
106
+ "carry_decoder_state": false,
107
+ "tail_padding_only": true
108
+ },
109
+ "actual_encoder_tensors": {
110
+ "inputs": [
111
+ {
112
+ "name": "serving_default_args_0",
113
+ "shape": [
114
+ 1,
115
+ 128,
116
+ 1500
117
+ ],
118
+ "type": 0,
119
+ "dtype": "float32"
120
+ },
121
+ {
122
+ "name": "serving_default_args_1",
123
+ "shape": [
124
+ 1
125
+ ],
126
+ "type": 2,
127
+ "dtype": "int32"
128
+ }
129
+ ],
130
+ "outputs": [
131
+ {
132
+ "name": "serving_default_output_0_output",
133
+ "shape": [
134
+ 1,
135
+ 1024,
136
+ 188
137
  ],
138
+ "type": 0,
139
+ "dtype": "float32"
140
+ },
141
+ {
142
+ "name": "serving_default_output_1_output",
143
+ "shape": [
144
+ 1
145
+ ],
146
+ "type": 2,
147
+ "dtype": "int32"
 
 
 
 
 
 
 
 
 
 
 
 
 
148
  }
149
  ]
150
+ },
151
+ "tested_runtime": {
152
+ "ai_edge_litert": "2.1.2",
153
+ "numpy": "2.4.4",
154
+ "sentencepiece": "0.2.1",
155
+ "encoder_threads": 4,
156
+ "decoder_joint_threads": 1
157
+ },
158
+ "source_program": {
159
+ "artifact": "encoder_dynamicT.pt2",
160
+ "sha256": "d16113f726f854f57a616387d1cbb88d4f2f8f80e84732a72f9bbcb788e38009"
161
+ },
162
+ "sha256": {
163
+ "encoder_T1500.tflite": "b26804686f98d68f7394baf3b13bdcea8e846ebf4ecf2903e3fc30fe3a3882a8",
164
+ "decoder_step.tflite": "eb0bf3559a0b4cbdc3ca05b7e8ff948ee5ef158ce424667b62a85f6c769a9ce1",
165
+ "joint_step.tflite": "0e28c22fc426df9900ef4a1bd15760ec757e44f0fd1818e0afb51c4fe79031be",
166
+ "tokenizer.model": "eacec2b0a77f336d4a2ca4a25a7047575d3c2b74de47e997f4c205126ed3135e"
167
+ },
168
+ "release_id": "20260914-int8-t1500-v1",
169
+ "legacy_release": {
170
+ "directory": "legacy/",
171
+ "revision": "c528ceb57bbc4e366993e50c79b22e0417a68484"
172
  }
173
  }
mapped_model_file.h ADDED
@@ -0,0 +1,64 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #pragma once
2
+ // Linux/Android native integration example. Keep this owner alive longer than
3
+ // the model/interpreter. Only pass data() to an API documented to accept an
4
+ // external, immutable model buffer. This is NOT a Java CompiledModel option.
5
+ #include <algorithm>
6
+ #include <array>
7
+ #include <cerrno>
8
+ #include <cstring>
9
+ #include <stdexcept>
10
+ #include <string>
11
+ #include <fcntl.h>
12
+ #include <sys/mman.h>
13
+ #include <sys/stat.h>
14
+ #include <unistd.h>
15
+
16
+ class MappedModelFile {
17
+ public:
18
+ explicit MappedModelFile(const char* path) {
19
+ fd_ = open(path, O_RDONLY | O_CLOEXEC);
20
+ if (fd_ < 0) throw std::runtime_error("Cannot open model");
21
+ struct stat st {};
22
+ if (fstat(fd_, &st) != 0 || st.st_size <= 0) {
23
+ close(fd_);
24
+ throw std::runtime_error("Invalid model file");
25
+ }
26
+ size_ = static_cast<size_t>(st.st_size);
27
+ address_ = mmap(nullptr, size_, PROT_READ, MAP_PRIVATE, fd_, 0);
28
+ if (address_ == MAP_FAILED) {
29
+ close(fd_);
30
+ throw std::runtime_error("Cannot map model");
31
+ }
32
+ }
33
+ MappedModelFile(const MappedModelFile&) = delete;
34
+ MappedModelFile& operator=(const MappedModelFile&) = delete;
35
+ ~MappedModelFile() { munmap(address_, size_); close(fd_); }
36
+ const void* data() const { return address_; }
37
+ size_t size() const { return size_; }
38
+
39
+ // Call once after encoder delegate preparation and representative warmup.
40
+ // The map remains valid; kernels can fault required source pages back in.
41
+ // Never use MADV_DONTNEED on an anonymous ByteBuffer or writable model buffer.
42
+ void DropVerifiedSourcePages() {
43
+ struct stat st {};
44
+ if (fstat(fd_, &st) != 0 || static_cast<size_t>(st.st_size) != size_)
45
+ throw std::runtime_error("Model file changed size");
46
+ std::array<unsigned char, 65536> bytes {};
47
+ for (size_t offset = 0; offset < size_;) {
48
+ size_t count = std::min(bytes.size(), size_ - offset);
49
+ ssize_t got;
50
+ do { got = pread(fd_, bytes.data(), count, static_cast<off_t>(offset)); }
51
+ while (got < 0 && errno == EINTR);
52
+ if (got <= 0 || std::memcmp(bytes.data(),
53
+ static_cast<const unsigned char*>(address_) + offset, got) != 0)
54
+ throw std::runtime_error("Model mapping differs from file");
55
+ offset += static_cast<size_t>(got);
56
+ }
57
+ if (madvise(address_, size_, MADV_DONTNEED) != 0)
58
+ throw std::runtime_error(std::string("madvise failed: ") + std::strerror(errno));
59
+ }
60
+ private:
61
+ int fd_;
62
+ void* address_;
63
+ size_t size_;
64
+ };
parakeet-tdt-0.6b-v3-litert.zip CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:da306f15d3835e939fe8585933709646c01621f4570f3186121cb435d50fd1ac
3
- size 1243526688
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:cd49232e6756b62ba14049bb31bab799838e30170b235a6993888e57662fdc95
3
+ size 665277590
publication.json ADDED
@@ -0,0 +1,11 @@
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "repo_id": "spybyscript/parakeet-tdt-litert",
3
+ "release_id": "20260914-int8-t1500-v1",
4
+ "source_model": "nvidia/parakeet-tdt-0.6b-v3",
5
+ "prior_revision": "c528ceb57bbc4e366993e50c79b22e0417a68484",
6
+ "legacy_directory": "legacy/",
7
+ "encoder_precision": "dynamic_INT8",
8
+ "decoder_joint_precision": "FP16 weights, FP32 interfaces",
9
+ "android_validated": false,
10
+ "private_evaluation_material_included": false
11
+ }
validation.json ADDED
@@ -0,0 +1,51 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "candidate": {
3
+ "bucket": 1500,
4
+ "variant": "all8",
5
+ "peak_rss_gb": 1.600212992,
6
+ "english_wer": 0.042953020134228186,
7
+ "english_seconds": 11.591583443339914,
8
+ "longform_seconds": 12.17721762927249,
9
+ "comparisons": {
10
+ "english": {
11
+ "clips": 33,
12
+ "exact_token_sequences": 29,
13
+ "exact_normalized_words": 31,
14
+ "token_edits": 10,
15
+ "word_edits_vs_fp16": 2
16
+ },
17
+ "longform": {
18
+ "clips": 4,
19
+ "exact_token_sequences": 1,
20
+ "exact_normalized_words": 2,
21
+ "token_edits": 73,
22
+ "word_edits_vs_fp16": 31
23
+ }
24
+ }
25
+ },
26
+ "reference": "Original T1500 FP16 LiteRT",
27
+ "english_reference_words": 745,
28
+ "english_reference_type": "Synthetic source text, not independently human-verified transcripts",
29
+ "longform_reference_type": "No full human transcripts; no WER claimed",
30
+ "android_validated": false,
31
+ "linux_page_advice": [
32
+ {
33
+ "corpus": "english",
34
+ "clips": 33,
35
+ "exact_tokens_vs_unhinted_int8": true,
36
+ "peak_rss_gb": 1.4610432,
37
+ "resident_rss_gb": 0.954400768,
38
+ "resident_pss_gb": 0.947043328,
39
+ "seconds": 10.858530042227358
40
+ },
41
+ {
42
+ "corpus": "longform",
43
+ "clips": 4,
44
+ "exact_tokens_vs_unhinted_int8": true,
45
+ "peak_rss_gb": 1.462030336,
46
+ "resident_rss_gb": 0.954167296,
47
+ "resident_pss_gb": 0.946869248,
48
+ "seconds": 11.424920583609492
49
+ }
50
+ ]
51
+ }