| --- |
| license: apache-2.0 |
| base_model: HeartMuLa/HeartTranscriptor-oss |
| pipeline_tag: automatic-speech-recognition |
| library_name: whisperkit |
| tags: |
| - coreml |
| - whisperkit |
| - lyrics |
| - singing |
| - music |
| --- |
| |
| # HeartTranscriptor-oss for Core ML (WhisperKit) |
|
|
| [HeartMuLa/HeartTranscriptor-oss](https://huggingface.co/HeartMuLa/HeartTranscriptor-oss), a |
| Whisper-medium fine-tune for sung lyrics by the HeartMuLa team, converted to |
| [WhisperKit](https://github.com/argmaxinc/WhisperKit)'s Core ML format for macOS and iOS. |
| [SwiftHeardLyrics](https://github.com/ssmall256/SwiftHeardLyrics) (`HeardLyricsHeart`) downloads and runs it, on the GPU or the |
| Neural Engine, and DemucsStudio uses it for lyrics of separated vocals. |
|
|
| | Folder | Weights | Size | Words different from PyTorch (GPU / Neural Engine) | WER, JamendoLyrics | |
| |---|---|---:|---:|---:| |
| | `HeartTranscriptor-whisperkit` | float16 | 1.4 GB | 0.8 % / 2.3 % | 25.3 % | |
| | `HeartTranscriptor-whisperkit-6bit` | 6-bit palettized | 627 MB | 5.8 % / 7.9 % | 24.4 % | |
|
|
| "Words different" compares with the float32 PyTorch model on 50 MUSDB18-HQ vocal excerpts; the word |
| error rate is against the lyrics of 10 JamendoLyrics songs (2,792 words), with words aligned by a |
| CTC aligner. The 6-bit model needs 1.0 GB of memory on an iPhone's Neural Engine instead of 1.8 GB, |
| runs 1.5x slower on the GPU, and is what SwiftHeardLyrics downloads on iOS. |
|
|
| ## Changes from the original |
|
|
| - Converted with whisperkittools (audio encoder, text decoder, mel spectrogram, tokenizer). |
| - The audio encoder's residual stream is multiplied by 8 (with the attention output projections, |
| fc2 and LayerNorm eps scaled to match). LayerNorm is scale invariant, so the encoder computes the |
| same function; in float16 on the Neural Engine its output improves from 48 to 56 dB PSNR. |
| - The checkpoint's `use_cache` is turned on for conversion; the weights are unchanged. |
| - `HeartTranscriptor-whisperkit-6bit`: post-training 6-bit k-means palettization with |
| coremltools (per grouped channel, 16 channels per lookup table). |
|
|
| SwiftHeardLyrics computes the log-mel input in float32 itself, as the reference does, rather than |
| with the included float16 `MelSpectrogram` model. Each folder's files and their SHA-256 are in |
| `manifest.json`. Conversion scripts: `scripts/export_heart_whisperkit.sh` and |
| `scripts/quantize_heart_whisperkit.py` in SwiftHeardLyrics. |
|
|
| ## License |
|
|
| Apache-2.0, as the original model ([LICENSE](LICENSE)). |
|
|
