File size: 2,434 Bytes
181aa17 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 | ---
license: apache-2.0
base_model: HeartMuLa/HeartTranscriptor-oss
pipeline_tag: automatic-speech-recognition
library_name: whisperkit
tags:
- coreml
- whisperkit
- lyrics
- singing
- music
---
# HeartTranscriptor-oss for Core ML (WhisperKit)
[HeartMuLa/HeartTranscriptor-oss](https://huggingface.co/HeartMuLa/HeartTranscriptor-oss), a
Whisper-medium fine-tune for sung lyrics by the HeartMuLa team, converted to
[WhisperKit](https://github.com/argmaxinc/WhisperKit)'s Core ML format for macOS and iOS.
[SwiftHeardLyrics](https://github.com/ssmall256/SwiftHeardLyrics) (`HeardLyricsHeart`) downloads and runs it, on the GPU or the
Neural Engine, and DemucsStudio uses it for lyrics of separated vocals.
| Folder | Weights | Size | Words different from PyTorch (GPU / Neural Engine) | WER, JamendoLyrics |
|---|---|---:|---:|---:|
| `HeartTranscriptor-whisperkit` | float16 | 1.4 GB | 0.8 % / 2.3 % | 25.3 % |
| `HeartTranscriptor-whisperkit-6bit` | 6-bit palettized | 627 MB | 5.8 % / 7.9 % | 24.4 % |
"Words different" compares with the float32 PyTorch model on 50 MUSDB18-HQ vocal excerpts; the word
error rate is against the lyrics of 10 JamendoLyrics songs (2,792 words), with words aligned by a
CTC aligner. The 6-bit model needs 1.0 GB of memory on an iPhone's Neural Engine instead of 1.8 GB,
runs 1.5x slower on the GPU, and is what SwiftHeardLyrics downloads on iOS.
## Changes from the original
- Converted with whisperkittools (audio encoder, text decoder, mel spectrogram, tokenizer).
- The audio encoder's residual stream is multiplied by 8 (with the attention output projections,
fc2 and LayerNorm eps scaled to match). LayerNorm is scale invariant, so the encoder computes the
same function; in float16 on the Neural Engine its output improves from 48 to 56 dB PSNR.
- The checkpoint's `use_cache` is turned on for conversion; the weights are unchanged.
- `HeartTranscriptor-whisperkit-6bit`: post-training 6-bit k-means palettization with
coremltools (per grouped channel, 16 channels per lookup table).
SwiftHeardLyrics computes the log-mel input in float32 itself, as the reference does, rather than
with the included float16 `MelSpectrogram` model. Each folder's files and their SHA-256 are in
`manifest.json`. Conversion scripts: `scripts/export_heart_whisperkit.sh` and
`scripts/quantize_heart_whisperkit.py` in SwiftHeardLyrics.
## License
Apache-2.0, as the original model ([LICENSE](LICENSE)).
|