| --- |
| license: mit |
| language: |
| - ko |
| base_model: openai/whisper-large-v2 |
| pipeline_tag: automatic-speech-recognition |
| tags: |
| - whisper |
| - lyrics-transcription |
| - lyrics-alignment |
| - korean-folk-song |
| - music |
| - music-information-retrieval |
| --- |
| |
| # RefWhisper |
|
|
| Model weights for [RefWhisper](https://github.com/MALerLab/RefWhisper), from the DLfM 2023 paper |
| "[Aligning Incomplete Lyrics of Korean Folk Song Dataset using Whisper](https://doi.org/10.1145/3625135.3625154)". |
|
|
| RefWhisper is Whisper large-v2 with an extra reference encoder and a reference cross-attention layer in the decoder. |
| It transcribes sung lyrics while consulting a pre-transcribed lyric that may be incomplete or partly wrong, and the |
| reference cross-attention also aligns that lyric with the transcription and the audio. |
|
|
| ## File |
|
|
| | File | Contents | |
| |---|---| |
| | `ref_whisper_weight.pt` | Training checkpoint with `model` (state dict), `optim` (optimizer state) and `dims` (model dimensions). fp32, 11.3 GB. The same file that was previously shared on OneDrive. | |
|
|
| ## Model |
|
|
| - Initialized from `whisper-large-v2`. The audio encoder is frozen. |
| - Reference encoder: token embedding initialized from the Whisper decoder, a learnable positional encoding with a 256-token context, and two residual blocks of 1D convolutions (kernel size 3, 1280 channels). |
| - Decoder: the last 16 of the 32 blocks get a reference cross-attention layer and are fine-tuned. The earlier blocks and the token embedding are frozen. |
| - Training data: synthesized speech (Google Speech API) of two consecutive lines from the lyrics of the *Anthology of Korean Traditional Folksongs*. The reference input is the annotated lyric of the song, with its line order shuffled. |
| - AdamW, learning rate 1e-4 for new parameters and 3e-6 for pre-trained ones, batch size 8. This checkpoint was saved at the end of epoch 12 (70,776 steps). |
|
|
| ## Usage |
|
|
| Install the code from [GitHub](https://github.com/MALerLab/RefWhisper) and run it from the repository root. |
|
|
| ``` |
| git clone https://github.com/MALerLab/RefWhisper.git && cd RefWhisper |
| pip install pipenv && pipenv install && pipenv shell |
| wget https://huggingface.co/MALer-Lab/refwhisper/resolve/main/ref_whisper_weight.pt |
| ``` |
|
|
| The reference lyric is a plain text file with one lyric line per line. |
|
|
| ``` |
| python -m ref_whisper.inference_timestamped song.wav song.txt --model ref_whisper_weight.pt --language ko --output_dir results |
| ``` |
|
|
| ```python |
| from ref_whisper.inference_timestamped import load_model, transcribe_timestamped |
| |
| model = load_model("ref_whisper_weight.pt", device="cuda") |
| |
| # transcription guided by the reference lyric |
| result = model.transcribe(audio="song.wav", reference_text="song.txt", temperature=0, language="ko") |
| |
| # transcription with word timestamps and line-level alignment to the reference lyric |
| result = transcribe_timestamped(model, "song.wav", "song.txt", language="ko") |
| print(result["ref_align_segments"]) |
| ``` |
|
|
| ## Results |
|
|
| Transcription accuracy on 201 manually transcribed segments (under 30 s) from folk songs that were not used for fine-tuning (Table 2 of the paper): |
|
|
| | Model | WER | CER | |
| |---|---|---| |
| | Whisper large-v2 | 0.722 | 0.384 | |
| | Whisper fine-tuned on the synthesized data | 0.804 | 0.478 | |
| | RefWhisper | 0.319 | 0.169 | |
|
|
| A later re-evaluation of this checkpoint on 352 surviving test segments gave WER 0.331 / CER 0.162 (Whisper large-v2: 0.744 / 0.380). |
|
|
| The transcriptions and alignments for 13,801 songs of the anthology, produced with this model, are in |
| [`transcription_result_csvs.tar.gz`](https://github.com/MALerLab/RefWhisper/blob/main/transcription_result_csvs.tar.gz). |
|
|
| ## License |
|
|
| MIT, the same as the code. The audio and lyrics of the *Anthology of Korean Traditional Folksongs* are not included; see the [official website](http://urisori.co.kr/urisori-en/doku.php?id=start). |
|
|
| ## Citation |
|
|
| ```bibtex |
| @inproceedings{han2023aligning, |
| title = {Aligning Incomplete Lyrics of Korean Folk Song Dataset using Whisper}, |
| author = {Han, Danbinaerin and Kim, Daewoong and Jeong, Dasaem}, |
| booktitle = {Proceedings of the 10th International Conference on Digital Libraries for Musicology (DLfM)}, |
| pages = {7--11}, |
| year = {2023}, |
| address = {Milan, Italy}, |
| publisher = {ACM}, |
| doi = {10.1145/3625135.3625154} |
| } |
| ``` |
|
|
