HeartTranscriptor-oss for Core ML (WhisperKit)
HeartMuLa/HeartTranscriptor-oss, a
Whisper-medium fine-tune for sung lyrics by the HeartMuLa team, converted to
WhisperKit's Core ML format for macOS and iOS.
SwiftHeardLyrics (HeardLyricsHeart) downloads and runs it, on the GPU or the
Neural Engine, and DemucsStudio uses it for lyrics of separated vocals.
| Folder | Weights | Size | Words different from PyTorch (GPU / Neural Engine) | WER, JamendoLyrics |
|---|---|---|---|---|
HeartTranscriptor-whisperkit |
float16 | 1.4 GB | 0.8 % / 2.3 % | 25.3 % |
HeartTranscriptor-whisperkit-6bit |
6-bit palettized | 627 MB | 5.8 % / 7.9 % | 24.4 % |
"Words different" compares with the float32 PyTorch model on 50 MUSDB18-HQ vocal excerpts; the word error rate is against the lyrics of 10 JamendoLyrics songs (2,792 words), with words aligned by a CTC aligner. The 6-bit model needs 1.0 GB of memory on an iPhone's Neural Engine instead of 1.8 GB, runs 1.5x slower on the GPU, and is what SwiftHeardLyrics downloads on iOS.
Changes from the original
- Converted with whisperkittools (audio encoder, text decoder, mel spectrogram, tokenizer).
- The audio encoder's residual stream is multiplied by 8 (with the attention output projections, fc2 and LayerNorm eps scaled to match). LayerNorm is scale invariant, so the encoder computes the same function; in float16 on the Neural Engine its output improves from 48 to 56 dB PSNR.
- The checkpoint's
use_cacheis turned on for conversion; the weights are unchanged. HeartTranscriptor-whisperkit-6bit: post-training 6-bit k-means palettization with coremltools (per grouped channel, 16 channels per lookup table).
SwiftHeardLyrics computes the log-mel input in float32 itself, as the reference does, rather than
with the included float16 MelSpectrogram model. Each folder's files and their SHA-256 are in
manifest.json. Conversion scripts: scripts/export_heart_whisperkit.sh and
scripts/quantize_heart_whisperkit.py in SwiftHeardLyrics.
License
Apache-2.0, as the original model (LICENSE).
- Downloads last month
- 36
Model tree for ssmall256/HeartTranscriptor-oss-coreml
Base model
HeartMuLa/HeartTranscriptor-oss
# Install CLI with Homebrew on macOS device brew install whisperkit-cli # View all available inference options whisperkit-cli transcribe --help # Download and run inference using whisper base model whisperkit-cli transcribe --audio-path /path/to/audio.mp3 # Or use your preferred model variant whisperkit-cli transcribe --model "large-v3" --model-prefix "distil" --audio-path /path/to/audio.mp3 --verbose