File size: 4,260 Bytes
b34c2a8 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 | ---
license: mit
language:
- ko
base_model: openai/whisper-large-v2
pipeline_tag: automatic-speech-recognition
tags:
- whisper
- lyrics-transcription
- lyrics-alignment
- korean-folk-song
- music
- music-information-retrieval
---
# RefWhisper
Model weights for [RefWhisper](https://github.com/MALerLab/RefWhisper), from the DLfM 2023 paper
"[Aligning Incomplete Lyrics of Korean Folk Song Dataset using Whisper](https://doi.org/10.1145/3625135.3625154)".
RefWhisper is Whisper large-v2 with an extra reference encoder and a reference cross-attention layer in the decoder.
It transcribes sung lyrics while consulting a pre-transcribed lyric that may be incomplete or partly wrong, and the
reference cross-attention also aligns that lyric with the transcription and the audio.
## File
| File | Contents |
|---|---|
| `ref_whisper_weight.pt` | Training checkpoint with `model` (state dict), `optim` (optimizer state) and `dims` (model dimensions). fp32, 11.3 GB. The same file that was previously shared on OneDrive. |
## Model
- Initialized from `whisper-large-v2`. The audio encoder is frozen.
- Reference encoder: token embedding initialized from the Whisper decoder, a learnable positional encoding with a 256-token context, and two residual blocks of 1D convolutions (kernel size 3, 1280 channels).
- Decoder: the last 16 of the 32 blocks get a reference cross-attention layer and are fine-tuned. The earlier blocks and the token embedding are frozen.
- Training data: synthesized speech (Google Speech API) of two consecutive lines from the lyrics of the *Anthology of Korean Traditional Folksongs*. The reference input is the annotated lyric of the song, with its line order shuffled.
- AdamW, learning rate 1e-4 for new parameters and 3e-6 for pre-trained ones, batch size 8. This checkpoint was saved at the end of epoch 12 (70,776 steps).
## Usage
Install the code from [GitHub](https://github.com/MALerLab/RefWhisper) and run it from the repository root.
```
git clone https://github.com/MALerLab/RefWhisper.git && cd RefWhisper
pip install pipenv && pipenv install && pipenv shell
wget https://huggingface.co/MALer-Lab/refwhisper/resolve/main/ref_whisper_weight.pt
```
The reference lyric is a plain text file with one lyric line per line.
```
python -m ref_whisper.inference_timestamped song.wav song.txt --model ref_whisper_weight.pt --language ko --output_dir results
```
```python
from ref_whisper.inference_timestamped import load_model, transcribe_timestamped
model = load_model("ref_whisper_weight.pt", device="cuda")
# transcription guided by the reference lyric
result = model.transcribe(audio="song.wav", reference_text="song.txt", temperature=0, language="ko")
# transcription with word timestamps and line-level alignment to the reference lyric
result = transcribe_timestamped(model, "song.wav", "song.txt", language="ko")
print(result["ref_align_segments"])
```
## Results
Transcription accuracy on 201 manually transcribed segments (under 30 s) from folk songs that were not used for fine-tuning (Table 2 of the paper):
| Model | WER | CER |
|---|---|---|
| Whisper large-v2 | 0.722 | 0.384 |
| Whisper fine-tuned on the synthesized data | 0.804 | 0.478 |
| RefWhisper | 0.319 | 0.169 |
A later re-evaluation of this checkpoint on 352 surviving test segments gave WER 0.331 / CER 0.162 (Whisper large-v2: 0.744 / 0.380).
The transcriptions and alignments for 13,801 songs of the anthology, produced with this model, are in
[`transcription_result_csvs.tar.gz`](https://github.com/MALerLab/RefWhisper/blob/main/transcription_result_csvs.tar.gz).
## License
MIT, the same as the code. The audio and lyrics of the *Anthology of Korean Traditional Folksongs* are not included; see the [official website](http://urisori.co.kr/urisori-en/doku.php?id=start).
## Citation
```bibtex
@inproceedings{han2023aligning,
title = {Aligning Incomplete Lyrics of Korean Folk Song Dataset using Whisper},
author = {Han, Danbinaerin and Kim, Daewoong and Jeong, Dasaem},
booktitle = {Proceedings of the 10th International Conference on Digital Libraries for Musicology (DLfM)},
pages = {7--11},
year = {2023},
address = {Milan, Italy},
publisher = {ACM},
doi = {10.1145/3625135.3625154}
}
```
|