madjinn/kb-whisper-small-coreml

KB-Whisper Small for Core ML (WhisperKit format)

This is KBLab/kb-whisper-small, the Swedish Whisper model from KBLab at the National Library of Sweden, converted to Core ML for the Apple Neural Engine. It's in the format of WhisperKit (Argmax). It runs on iPhone and Mac, and it's made for the dictation keyboard WhispAway.

Why another conversion

An existing Core ML conversion of KB-Whisper Small never stays compiled for the Neural Engine on an A14 iPhone (iPhone 12 mini, iOS 26):

• every load spends 6–7 minutes compiling again;
• the encoder then runs on the CPU, at about 4–5 s per 30-second window.

That conversion was made with 2025 versions of the tools. Its graph is about a third larger: the attention is split into ragged chunks, and the LayerNorm is fused along the channel axis.

Argmax's own openai_whisper-small (2024) runs well on the same phone: a reload takes 0.8 s and the encoder about 90 ms. This conversion reproduces that 2024 recipe with KB-Whisper's weights. Its operation graph is identical, op for op, to Argmax's openai_whisper-small for the mel spectrogram, the encoder and the decoder.

Files

kb-whisper-small/:

• MelSpectrogram.mlmodelc: input audio Float16 [480000]; output melspectrogram_features [1,80,1,3000].
• AudioEncoder.mlmodelc: output encoder_output_embeds [1,768,1,1500].
• TextDecoder.mlmodelc:
  • inputs: input_ids, cache_length, key_cache/value_cache [1,9216,1,224], kv_cache_update_mask, encoder_output_embeds and decoder_key_padding_mask;
  • outputs: logits [1,1,51865], key_cache_updates, value_cache_updates and alignment_heads_weights.
• config.json, generation_config.json and tokenizer.json: KBLab's files, unchanged.

All weights are fp16, with no quantisation. The three models take about 490 MB in all.

How it was made

• Source: KBLab/kb-whisper-small at revision 3564d61a42fc210ceaa55a22a96dd64478959c78.
• Converter: whisperkittools at b5c1e50eb611240025c158784ec34bbfce26de09, with argmaxtools 0.1.7, coremltools 7.1, torch 2.2.1 and transformers 4.39.1. Those are the versions recorded in Argmax's openai_whisper-small metadata.
• Command: whisperkit-generate-model --model-version KBLab/kb-whisper-small --text-decoder-max-sequence-length 224. The defaults are otherwise unchanged: encoder attention SplitHeadsQ, decoder Cat.
• Compiled with xcrun coremlcompiler (coremlc 3520.5.1).
• Checked:
  • the interface matches Argmax's openai_whisper-small exactly;
  • in PyTorch, the converted modules match Hugging Face's model (encoder 140 dB PSNR);
  • greedy Swedish transcripts of the test clips are identical to transformers' generate().
• Not yet measured: speed on devices.

Use with WhisperKit

The folder follows WhisperKit's layout, so pointing WhisperKit's modelRepo at this repository and its model at kb-whisper-small should work. We haven't tested that: WhispAway runs these files with its own Core ML pipeline.

Licence and credits

• Licence: Apache-2.0, as KB-Whisper is (see LICENSE).
• KB-Whisper: © KBLab, National Library of Sweden, fine-tuned from OpenAI Whisper (MIT).
• Converter: whisperkittools and WhisperKit by Argmax (MIT).
• This repository: a format conversion by Madjinn. The weights are KBLab's, converted to Core ML fp16; nothing was retrained.