---
license: apache-2.0
base_model: HeartMuLa/HeartTranscriptor-oss
pipeline_tag: automatic-speech-recognition
library_name: whisperkit
tags:
- coreml
- whisperkit
- lyrics
- singing
- music
---

# HeartTranscriptor-oss for Core ML (WhisperKit)

[HeartMuLa/HeartTranscriptor-oss](https://huggingface.co/HeartMuLa/HeartTranscriptor-oss), a
Whisper-medium fine-tune for sung lyrics by the HeartMuLa team, converted to
[WhisperKit](https://github.com/argmaxinc/WhisperKit)'s Core ML format for macOS and iOS.
[SwiftHeardLyrics](https://github.com/ssmall256/SwiftHeardLyrics) (`HeardLyricsHeart`) downloads and runs it, on the GPU or the
Neural Engine, and DemucsStudio uses it for lyrics of separated vocals.

| Folder | Weights | Size | Words different from PyTorch (GPU / Neural Engine) | WER, JamendoLyrics |
|---|---|---:|---:|---:|
| `HeartTranscriptor-whisperkit` | float16 | 1.4 GB | 0.8 % / 2.3 % | 25.3 % |
| `HeartTranscriptor-whisperkit-6bit` | 6-bit palettized | 627 MB | 5.8 % / 7.9 % | 24.4 % |

"Words different" compares with the float32 PyTorch model on 50 MUSDB18-HQ vocal excerpts; the word
error rate is against the lyrics of 10 JamendoLyrics songs (2,792 words), with words aligned by a
CTC aligner. The 6-bit model needs 1.0 GB of memory on an iPhone's Neural Engine instead of 1.8 GB,
runs 1.5x slower on the GPU, and is what SwiftHeardLyrics downloads on iOS.

## Changes from the original

- Converted with whisperkittools (audio encoder, text decoder, mel spectrogram, tokenizer).
- The audio encoder's residual stream is multiplied by 8 (with the attention output projections,
  fc2 and LayerNorm eps scaled to match). LayerNorm is scale invariant, so the encoder computes the
  same function; in float16 on the Neural Engine its output improves from 48 to 56 dB PSNR.
- The checkpoint's `use_cache` is turned on for conversion; the weights are unchanged.
- `HeartTranscriptor-whisperkit-6bit`: post-training 6-bit k-means palettization with
  coremltools (per grouped channel, 16 channels per lookup table).

SwiftHeardLyrics computes the log-mel input in float32 itself, as the reference does, rather than
with the included float16 `MelSpectrogram` model. Each folder's files and their SHA-256 are in
`manifest.json`. Conversion scripts: `scripts/export_heart_whisperkit.sh` and
`scripts/quantize_heart_whisperkit.py` in SwiftHeardLyrics.

## License

Apache-2.0, as the original model ([LICENSE](LICENSE)).