whisper-it-multi-hailo

Whisper encoder compiled for Hailo-8 (int8 PTQ) plus an incremental KV-cache decoder that runs on the CPU, derived from the Italian-multilingual fine-tune LocalAI-io/whisper-base-it-multi (openai/whisper-base, 72.6M params, d_model 512, 6 encoder + 6 decoder layers, full 51865-token vocabulary).

Two encoder windows are published, 5 s and 10 s, each with its matching pair of decoder graphs. Use the 10 s encoder: it is the most accurate and the fastest of the three pipelines measured here, because a 10–12 s utterance no longer has to be split into 2–3 windows.

Results

Word error rate and character error rate on 500 filtered Italian Common Voice v23 clips. Lower is better. Best value in each column in bold.

Pipeline	WER ↓	CER ↓	Speed-up ↑	Exact match ↑	s/clip ↓
10 s encoder + KV decoder	24.96	8.39	6.78×	93 / 500	1.03
5 s encoder + KV decoder	43.32	21.63	6.30×	46 / 500	1.18
Official Hailo 5 s (encoder + decoder HEF)	55.86	23.03	4.56×	8 / 500	1.67

error rates, 1 block = 2 percentage points

                     WER                         CER
10 s encoder + KV    ████████████                  24.96    ████           8.39
5 s encoder + KV     ██████████████████████        43.32    ███████████   21.63
Official Hailo 5 s   ████████████████████████████  55.86    ████████████  23.03

Against the 5 s encoder the 10 s one is 1.7× lower WER and 2.6× lower CER; against Hailo's own 5 s encoder/decoder pair it is 2.2× lower WER and 2.9× lower CER.

Results by clip duration

The same 500 clips, split by length. The gap widens with duration, because a clip longer than one window has to be transcribed in several independent passes.

Bucket	Pipeline	WER ↓	CER ↓	Speed-up ↑	Exact ↑	s/clip ↓
< 5 s (167)	10 s encoder + KV	27.66	9.25	5.48×	47 / 167	0.76
	5 s encoder + KV	28.73	10.30	6.82×	41 / 167	0.62
	Official Hailo 5 s	53.87	20.77	4.85×	8 / 167	0.94
5–10 s (166)	10 s encoder + KV	20.12	5.39	6.60×	30 / 166	1.04
	5 s encoder + KV	48.21	24.72	5.74×	4 / 166	1.24
	Official Hailo 5 s	52.41	22.38	4.13×	0 / 166	1.71
> 10 s (167)	10 s encoder + KV	27.06	10.50	8.26×	16 / 167	1.30
	5 s encoder + KV	53.05	29.89	6.34×	1 / 167	1.68
	Official Hailo 5 s	61.27	25.92	4.70×	0 / 167	2.35
all 500	10 s encoder + KV	24.96	8.39	6.78×	93 / 500	1.03
	5 s encoder + KV	43.32	21.63	6.30×	46 / 500	1.18
	Official Hailo 5 s	55.86	23.03	4.56×	8 / 500	1.67

Speed-up is audio seconds processed per wall second. The 5 s pipeline is only faster on clips that already fit in a single window; from 5 s upwards the 10 s encoder wins on accuracy and latency together. WER/CER are arithmetic means over clips, exact match is the number of clips whose transcript matches the reference exactly.

Download size and download tracking

Sizes of what a client actually pulls, tokenizer included:

Selection	Files	Bytes
10 s pipeline (recommended)	itmulti_enc10s_int8_hailo8.hef + itmulti_dec_cross500_kv_fp32.onnx + itmulti_dec_step500_kv_fp32.onnx + tokenizer/	361,091,710	344.4 MiB
5 s pipeline	itmulti_enc5s_int8_hailo8.hef + itmulti_dec_cross250_kv_fp32.onnx + itmulti_dec_step250_kv_fp32.onnx + tokenizer/	363,461,809	346.6 MiB
everything	both pipelines	720,739,827	687.4 MiB

The 301.7 MB step graph dominates: it is the fp32 decoder, and the 250- and 500-frame variants are the same size because the weight count does not depend on the audio window. Only the encoder differs in size (43.0 MB vs 45.3 MB).

All six model files are stored with Git LFS, so they are served from the Hub CDN rather than from git.

Why this repo ships a config.json

The Hub does not count every file request as a download. To avoid counting a single model fetch several times, it counts requests to one query file per repo, which by default is config.json (with config.yaml, hyperparams.yaml, params.json and meta.yaml as alternatives). A repo with no query file therefore reports 0 downloads no matter how many .hef and .onnx files are pulled, and so did this one until config.json was added.

config.json here is hand-written metadata, not a Transformers config: it records which encoder pairs with which decoder graph, the mel geometry, the token IDs to force, the measured WER/CER and the download sizes. It deliberately has no model_type or architectures key, because the repo ships no Transformers weights.

Practical consequences:

• snapshot_download fetches config.json as part of the snapshot, so a full-repo download is counted.
• Fetching only the .hef/.onnx files with hf_hub_download is not counted, because those are not query files.
• The snippet below fetches config.json first, which counts the download and also tells a client which encoder and decoder to pair.

from huggingface_hub import hf_hub_download

repo = "mpeex/whisper-it-multi-hailo"
hf_hub_download(repo, "config.json")                                    # counted query file
encoder = hf_hub_download(repo, "itmulti_enc10s_int8_hailo8.hef")
cross   = hf_hub_download(repo, "itmulti_dec_cross500_kv_fp32.onnx")
step    = hf_hub_download(repo, "itmulti_dec_step500_kv_fp32.onnx")

Repository contents

File	Size	sha256	Role
itmulti_enc10s_int8_hailo8.hef	42,973,700	49c378877683275c1fd865fcf501037c81006a52109b41065a84f1aeb047973c	Encoder, 10 s window, int8, recommended
itmulti_enc5s_int8_hailo8.hef	45,343,799	6a1eb2394d8c9668caa01fbb1a8d689eab72ed35d3439614c9a58c92eac3edfb	Encoder, 5 s window, int8
itmulti_dec_cross500_kv_fp32.onnx	12,604,861	72efd16e908f433b225d9ee612e9d9f81f1dc67cfc4cdcb8274cc09b1dfabd04	Cross-attention projections, 500 frames
itmulti_dec_cross250_kv_fp32.onnx	12,604,861	64149b18e413a3d40c944da1150a16108a6f915a132310bc1072133cea427200	Cross-attention projections, 250 frames
itmulti_dec_step500_kv_fp32.onnx	301,699,457	3c662637c5abeb471bfdb823ed44490bc1992b8b1e94eac6792e4328ade707cc	Incremental decoder step, pairs with the 10 s encoder
itmulti_dec_step250_kv_fp32.onnx	301,699,457	aec4646ee15ed25a32aa129c6408c272fb7e3ddc3ca5a9f4b4e110c2a7aa90ad	Incremental decoder step, pairs with the 5 s encoder
tokenizer/	–	–	HuggingFace fast tokenizer (51865 tokens)

Both encoder HEFs are self-contained: weights included, no external data files.

Model and I/O signatures

Mel front-end: 16 kHz mono, 80 log-mel bins, n_fft 400, hop 160 (100 frames/s), log10 with Whisper's (x + 4.0) / 4.0 normalisation.

Component	Input	Output
itmulti_enc10s_int8_hailo8.hef	mel (80, 1000) → FCR (1, 1000, 80)	features (1, 500, 512)
itmulti_enc5s_int8_hailo8.hef	mel (80, 500) → FCR (1, 500, 80)	features (1, 250, 512)
itmulti_dec_cross500_kv_fp32.onnx	features (1, 500, 512)	ck, cv (6, 500, 8, 64)
itmulti_dec_cross250_kv_fp32.onnx	features (1, 250, 512)	ck, cv (6, 250, 8, 64)
itmulti_dec_step{500,250}_kv_fp32.onnx	token (1,1) int64, pos (1,1) int64, ck, cv, kvk/kvv (6,1,8,32,64)	logits (1,1,51865), kvk_out, kvv_out

The encoder window and the decoder cross-graph must match: 500-frame encoder with the 500 graphs, 250-frame encoder with the 250 graphs.

How the KV-cache decoder works

The standard Whisper decoder recomputes the whole prefix at every step. Here the decoder is split into two ONNX graphs:

1. cross graph, run once per window: takes the encoder features and returns the per-layer cross-attention keys and values. They depend only on the audio, so they never change during a window (11.7 MiB for 500 frames, 5.9 MiB for 250).
2. step graph, run once per emitted token: takes a single token plus its absolute position, the precomputed ck/cv, and the self-attention cache kvk/kvv, and returns the logits for that position plus the updated cache, which is fed back in on the next step. The cache has a fixed length of 32 positions and is updated in place inside the graph with a write mask, so no dynamic shapes are needed.

Consequence: step n costs the same as step 1. Measured on this hardware, a token step takes 31.9 ms (10 s model) and 27.7 ms (5 s model), against ~4.8 s and ~5.0 s per clip for the non-cached decoder graph:

	non-cached	KV-cache	speed-up	transcripts identical
10 s	4.80 s/clip	1.56 s/clip	3.07×	20/20
5 s	5.00 s/clip	1.91 s/clip	2.62×	20/20

WER/CER were unchanged by the split on that set, so the gain is purely in latency.

Memory footprint

Measured as process RSS in isolated processes, one per decoder, after loading and after a warm-up run. Baseline for Python + NumPy + ONNX Runtime is 33.6 MiB.

Decoder	after load	after warm-up	note
KV ONNX, 500 frames	412.5 MiB	467.1 MiB	11.7 MiB cross KV cache, 0.75 MiB self KV cache
KV ONNX, 250 frames	413.5 MiB	448.5 MiB	5.9 MiB cross KV cache, 0.75 MiB self KV cache
Official Hailo 5 s decoder	221.2 MiB	227.7 MiB	101.3 MiB is the host-side wte embedding table; decoder weights live in device memory

The two KV decoders carry identical weights (same 301.7 MB step graph), so the ~19 MiB difference is entirely the cross-attention cache, which scales with the encoder window. Host RSS is not directly comparable with a HEF decoder, whose weights are offloaded to the NPU.

Running the full 500-clip, 3-model benchmark in one process peaked at 1.26 GiB RSS (mean 1.00 GiB) on a Raspberry Pi 5 with the Hailo-8, leaving over 5.1 GiB available and no swap traffic.

How to run

Requirements: HailoRT 4.20 on Hailo-8, Python 3.11, numpy<2 (NumPy 2 breaks the HailoRT bindings with an Input buffer size 0 error), onnxruntime, librosa.

pip install "numpy==1.26.4" onnxruntime librosa
# HailoRT 4.20 from the Hailo software package

import sys
sys.path.insert(0, "/path/to/whisper-base-it-hailo")

from runtime.transcribe_hailo import EncoderHef, DecoderKv, encoder_features, log_mel_whisper

encoder = EncoderHef("itmulti_enc10s_int8_hailo8.hef")
with encoder:
    decoder = DecoderKv("itmulti_dec_step500_kv_fp32.onnx", "itmulti_dec_cross500_kv_fp32.onnx")
    mel = log_mel_whisper(wave_16k_mono, time_frames=1000)   # (80, 1000)
    feats = encoder_features(encoder.run(mel), mel.shape[1])  # (1, 500, 512)
    # greedy decoding:
    #   decoder.prime(ids, feats, len(forced_ids) - 1)
    #   token = int(decoder.run(ids, feats)[0, i].argmax())

Decoding used for every number in this card:

• forced prefix <|startoftranscript|> <|it|> <|transcribe|> <|notimestamps|> = [50258, 50274, 50359, 50363], EOS 50257, greedy argmax, no sampling and no prompt conditioning
• at most 32 tokens per window, self-attention cache reset at every window boundary. All three models in the benchmark above ran with a 0.05 s minimum segment size, so short tail segments are kept; note that the 5 s recipe's own default is a 1.2 s minimum segment
• audio longer than the window is split on silence, with a 5 s window using a 1.2 s minimum segment and a 10 s window a 0.05 s minimum

Benchmark methodology

• 500 Italian clips from Common Voice v23 test, sampled deterministically (seed 2610) and stratified: 167 < 5 s, 166 5–10 s, 167 > 10 s, with observed ranges 2016–4968 ms, 5016–9960 ms and 10008–11988 ms.
• Offline input filter applied identically to all models: 16 kHz mono, high-pass 70 Hz, low-pass 7.6 kHz, DC offset removal, RMS normalisation to 0.1 with a gain cap of 60 and a peak cap of 0.98.
• WER/CER are per-clip arithmetic means; timings exclude offline filtering, WAV reading and model initialisation, and include segmentation, mel computation, encoder, decoder and stitching. One warm-up clip per model and bucket.
• The official baseline is Hailo's published whisper_base 5 s encoder + decoder pair, both as HEF running on the NPU, driven through the same front-end and the same forced prefix. Its decoder is compiled for a 24-token sequence, against 32 for the KV decoders here, which is a handicap on longer windows.

Notes and caveats

• The encoders are int8 post-training quantizations produced with the Hailo Dataflow Compiler, without a post-quantization finetune pass. The 10 s calibration set is 2073 ten-second windows built by concatenating adjacent windows of the 5 s LibriSpeech dev-clean calibration set (50% overlap).
• The 5 s encoder and its 250-frame decoder graphs are kept for latency-sensitive or short-utterance workloads, but expect a large accuracy gap on utterances longer than one window.
• The decoder is fp32 ONNX on the CPU. Quantising it to int8 would cut its memory and possibly its latency, at the usual accuracy cost.
• Only the encoder is offloaded to the NPU, so end-to-end speed-up is bounded by the CPU decoder: roughly 30 ms per emitted token.

Credits

Derivation chain: openai/whisper-base → fine-tuned into LocalAI-io/whisper-base-it-multi → exported to ONNX and quantised for Hailo-8 here. No training was performed for this repository: the encoder was quantised with int8 post-training quantization and the decoder graphs are a KV-cache re-export of the same fp32 weights, so the base model is declared with the quantized relation.

Export, compilation and runtime tooling: mpeex/whisper-base-it-hailo. Code and documentation released under the MIT license; the Whisper weights remain under their original license.