---
license: mit
language:
- ko
base_model: openai/whisper-large-v2
pipeline_tag: automatic-speech-recognition
tags:
- whisper
- lyrics-transcription
- lyrics-alignment
- korean-folk-song
- music
- music-information-retrieval
---

# RefWhisper

Model weights for [RefWhisper](https://github.com/MALerLab/RefWhisper), from the DLfM 2023 paper
"[Aligning Incomplete Lyrics of Korean Folk Song Dataset using Whisper](https://doi.org/10.1145/3625135.3625154)".

RefWhisper is Whisper large-v2 with an extra reference encoder and a reference cross-attention layer in the decoder.
It transcribes sung lyrics while consulting a pre-transcribed lyric that may be incomplete or partly wrong, and the
reference cross-attention also aligns that lyric with the transcription and the audio.

## File

| File | Contents |
|---|---|
| `ref_whisper_weight.pt` | Training checkpoint with `model` (state dict), `optim` (optimizer state) and `dims` (model dimensions). fp32, 11.3 GB. The same file that was previously shared on OneDrive. |

## Model

- Initialized from `whisper-large-v2`. The audio encoder is frozen.
- Reference encoder: token embedding initialized from the Whisper decoder, a learnable positional encoding with a 256-token context, and two residual blocks of 1D convolutions (kernel size 3, 1280 channels).
- Decoder: the last 16 of the 32 blocks get a reference cross-attention layer and are fine-tuned. The earlier blocks and the token embedding are frozen.
- Training data: synthesized speech (Google Speech API) of two consecutive lines from the lyrics of the *Anthology of Korean Traditional Folksongs*. The reference input is the annotated lyric of the song, with its line order shuffled.
- AdamW, learning rate 1e-4 for new parameters and 3e-6 for pre-trained ones, batch size 8. This checkpoint was saved at the end of epoch 12 (70,776 steps).

## Usage

Install the code from [GitHub](https://github.com/MALerLab/RefWhisper) and run it from the repository root.

```
git clone https://github.com/MALerLab/RefWhisper.git && cd RefWhisper
pip install pipenv && pipenv install && pipenv shell
wget https://huggingface.co/MALer-Lab/refwhisper/resolve/main/ref_whisper_weight.pt
```

The reference lyric is a plain text file with one lyric line per line.

```
python -m ref_whisper.inference_timestamped song.wav song.txt --model ref_whisper_weight.pt --language ko --output_dir results
```

```python
from ref_whisper.inference_timestamped import load_model, transcribe_timestamped

model = load_model("ref_whisper_weight.pt", device="cuda")

# transcription guided by the reference lyric
result = model.transcribe(audio="song.wav", reference_text="song.txt", temperature=0, language="ko")

# transcription with word timestamps and line-level alignment to the reference lyric
result = transcribe_timestamped(model, "song.wav", "song.txt", language="ko")
print(result["ref_align_segments"])
```

## Results

Transcription accuracy on 201 manually transcribed segments (under 30 s) from folk songs that were not used for fine-tuning (Table 2 of the paper):

| Model | WER | CER |
|---|---|---|
| Whisper large-v2 | 0.722 | 0.384 |
| Whisper fine-tuned on the synthesized data | 0.804 | 0.478 |
| RefWhisper | 0.319 | 0.169 |

A later re-evaluation of this checkpoint on 352 surviving test segments gave WER 0.331 / CER 0.162 (Whisper large-v2: 0.744 / 0.380).

The transcriptions and alignments for 13,801 songs of the anthology, produced with this model, are in
[`transcription_result_csvs.tar.gz`](https://github.com/MALerLab/RefWhisper/blob/main/transcription_result_csvs.tar.gz).

## License

MIT, the same as the code. The audio and lyrics of the *Anthology of Korean Traditional Folksongs* are not included; see the [official website](http://urisori.co.kr/urisori-en/doku.php?id=start).

## Citation

```bibtex
@inproceedings{han2023aligning,
  title     = {Aligning Incomplete Lyrics of Korean Folk Song Dataset using Whisper},
  author    = {Han, Danbinaerin and Kim, Daewoong and Jeong, Dasaem},
  booktitle = {Proceedings of the 10th International Conference on Digital Libraries for Musicology (DLfM)},
  pages     = {7--11},
  year      = {2023},
  address   = {Milan, Italy},
  publisher = {ACM},
  doi       = {10.1145/3625135.3625154}
}
```