File size: 4,334 Bytes
9256e4d e4c7702 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 | ---
license: apache-2.0
tags:
- executorch
- xnnpack
- pte
- on-device
- automatic-speech-recognition
base_model:
- openai/whisper-tiny
---
# Whisper-tiny — ExecuTorch XNNPACK (encoder + decoder)
Speech recognition in two `.pte` files: the encoder runs once per 30-second window,
the decoder once per generated token.
| graph | precision | file | size (MB) | corr vs fp32 eager |
|-------|-----------|------|-----------|--------------------|
| encoder | fp32 | `whisper_tiny_encoder_xnnpack_fp32.pte` | 32.9 | 1.000000 |
| encoder | fp16 | `whisper_tiny_encoder_xnnpack_fp16.pte` | 17.6 | 0.999999 |
| encoder | int8 | `whisper_tiny_encoder_xnnpack_int8.pte` | 11.7 | 0.999454 |
| decoder | fp32 | `whisper_tiny_decoder_xnnpack_fp32.pte` | 198.0 | 1.000000 |
| decoder | fp16 | `whisper_tiny_decoder_xnnpack_fp16.pte` | 99.1 | 0.999988 |
Every file takes and returns fp32 tensors (token ids stay int64), so any encoder
pairs with any decoder. The lightest working pair is 110.8 MB.
- **Source**: [openai/whisper-tiny](https://huggingface.co/openai/whisper-tiny)
- **License**: Apache-2.0
- **Encoder input**: log-mel spectrogram `[1,80,3000]` — 30 s at 16 kHz, 80 mel bins,
hop 160, window 400. This is exactly what `WhisperFeatureExtractor` produces; pad
or trim audio to 30 s as it does.
- **Encoder output**: `encoder_hidden_states [1,1500,384]`
- **Decoder input**: the encoder output plus `decoder_input_ids [1,128]` int64,
left-aligned and padded. Start the sequence with
`<|startoftranscript|>`, a language token, `<|transcribe|>`, `<|notimestamps|>`.
- **Decoder output**: `logits [1,128,51865]`
## Decoding
There is no KV cache. The decoder is a static graph over a fixed 128-token window,
so a greedy step is: take `argmax` of row `len-1`, append it, run again. Stop at
`<|endoftext|>` (50257). 128 tokens covers a 30-second window of ordinary speech
with room to spare; for longer audio, start a new window.
That costs a full 128-position forward pass per token. On a 37M-parameter model
this is cheap enough to be practical, and it keeps the graph static — which is what
lets the same file run unchanged across runtimes and precisions.
## Verification (Mac arm64, executorch 1.4.0, torch 2.13.0)
The two wrappers compose back to `WhisperForConditionalGeneration` exactly
(max_abs_diff 0.000e+00), and every graph matches torch fp32 eager at the
correlations in the table above.
Median over 5 runs, Mac arm64 single process — a relative reference, not a device
number: encoder 54.5 ms (torch eager 20.5 ms), decoder 18.1 ms (eager 12.1 ms).
## Two things worth knowing about the sizes
**The decoder .pte is larger than the decoder's weights.** Its parameters come to
118 MB, and the file is 198 MB. Whisper ties `proj_out.weight` to
`decoder.embed_tokens.weight` — one 19.9M-parameter tensor — but the two uses need
different representations in the `.pte`: an embedding table the portable kernels
index into, and the same values packed into the XNNPACK delegate's blob for the
output matmul. Tying them in PyTorch does not tie them here, and referencing the
embedding weight directly through `F.linear` does not either.
**The decoder has no int8 build.** PT2E puts an observer on the int64
`decoder_input_ids` feeding the token embedding, and the lookup then refuses a float
index (`tensors used as indices must be long, int, byte or bool`). The encoder takes
float mel input and quantizes without complaint, which is where the size is worth
taking anyway.
## Conversion
torch.export → to_edge_transform_and_lower(XnnpackPartitioner) → .pte
(conversion script: [executorch-models](https://github.com/john-rocky/executorch-models))
The ExecuTorch tree ships a single-graph Whisper example under
`examples/models/whisper`. This is that model with the halves separated, because a
combined graph re-encodes the audio on every decoded token.
<!-- funnel:v1 -->
---
**More models in this format:** [ExecuTorch Model Zoo](https://huggingface.co/collections/mlboydaisuke/executorch-model-zoo-6a7ff328390b63075ffeae5e) — 31 models, each with the recipe that produced it.
**Want a different model on-device?** [Open a request](https://github.com/john-rocky/on-device-requests) — free, open weights only; the export and its measured numbers get published publicly.
<!-- /funnel:v1 -->
|