Add model card

README.md

@@ -0,0 +1,100 @@
+---
+license: mit
+language:
+- ko
+base_model: openai/whisper-large-v2
+pipeline_tag: automatic-speech-recognition
+tags:
+- whisper
+- lyrics-transcription
+- lyrics-alignment
+- korean-folk-song
+- music
+- music-information-retrieval
+---
+
+# RefWhisper
+
+Model weights for [RefWhisper](https://github.com/MALerLab/RefWhisper), from the DLfM 2023 paper
+"[Aligning Incomplete Lyrics of Korean Folk Song Dataset using Whisper](https://doi.org/10.1145/3625135.3625154)".
+
+RefWhisper is Whisper large-v2 with an extra reference encoder and a reference cross-attention layer in the decoder.
+It transcribes sung lyrics while consulting a pre-transcribed lyric that may be incomplete or partly wrong, and the
+reference cross-attention also aligns that lyric with the transcription and the audio.
+
+## File
+
+| File | Contents |
+|---|---|
+| `ref_whisper_weight.pt` | Training checkpoint with `model` (state dict), `optim` (optimizer state) and `dims` (model dimensions). fp32, 11.3 GB. The same file that was previously shared on OneDrive. |
+
+## Model
+
+- Initialized from `whisper-large-v2`. The audio encoder is frozen.
+- Reference encoder: token embedding initialized from the Whisper decoder, a learnable positional encoding with a 256-token context, and two residual blocks of 1D convolutions (kernel size 3, 1280 channels).
+- Decoder: the last 16 of the 32 blocks get a reference cross-attention layer and are fine-tuned. The earlier blocks and the token embedding are frozen.
+- Training data: synthesized speech (Google Speech API) of two consecutive lines from the lyrics of the *Anthology of Korean Traditional Folksongs*. The reference input is the annotated lyric of the song, with its line order shuffled.
+- AdamW, learning rate 1e-4 for new parameters and 3e-6 for pre-trained ones, batch size 8. This checkpoint was saved at the end of epoch 12 (70,776 steps).
+
+## Usage
+
+Install the code from [GitHub](https://github.com/MALerLab/RefWhisper) and run it from the repository root.
+
+```
+git clone https://github.com/MALerLab/RefWhisper.git && cd RefWhisper
+pip install pipenv && pipenv install && pipenv shell
+wget https://huggingface.co/MALer-Lab/refwhisper/resolve/main/ref_whisper_weight.pt
+```
+
+The reference lyric is a plain text file with one lyric line per line.
+
+```
+python -m ref_whisper.inference_timestamped song.wav song.txt --model ref_whisper_weight.pt --language ko --output_dir results
+```
+
+```python
+from ref_whisper.inference_timestamped import load_model, transcribe_timestamped
+
+model = load_model("ref_whisper_weight.pt", device="cuda")
+
+# transcription guided by the reference lyric
+result = model.transcribe(audio="song.wav", reference_text="song.txt", temperature=0, language="ko")
+
+# transcription with word timestamps and line-level alignment to the reference lyric
+result = transcribe_timestamped(model, "song.wav", "song.txt", language="ko")
+print(result["ref_align_segments"])
+```
+
+## Results
+
+Transcription accuracy on 201 manually transcribed segments (under 30 s) from folk songs that were not used for fine-tuning (Table 2 of the paper):
+
+| Model | WER | CER |
+|---|---|---|
+| Whisper large-v2 | 0.722 | 0.384 |
+| Whisper fine-tuned on the synthesized data | 0.804 | 0.478 |
+| RefWhisper | 0.319 | 0.169 |
+
+A later re-evaluation of this checkpoint on 352 surviving test segments gave WER 0.331 / CER 0.162 (Whisper large-v2: 0.744 / 0.380).
+
+The transcriptions and alignments for 13,801 songs of the anthology, produced with this model, are in
+[`transcription_result_csvs.tar.gz`](https://github.com/MALerLab/RefWhisper/blob/main/transcription_result_csvs.tar.gz).
+
+## License
+
+MIT, the same as the code. The audio and lyrics of the *Anthology of Korean Traditional Folksongs* are not included; see the [official website](http://urisori.co.kr/urisori-en/doku.php?id=start).
+
+## Citation
+
+```bibtex
+@inproceedings{han2023aligning,
+  title     = {Aligning Incomplete Lyrics of Korean Folk Song Dataset using Whisper},
+  author    = {Han, Danbinaerin and Kim, Daewoong and Jeong, Dasaem},
+  booktitle = {Proceedings of the 10th International Conference on Digital Libraries for Musicology (DLfM)},
+  pages     = {7--11},
+  year      = {2023},
+  address   = {Milan, Italy},
+  publisher = {ACM},
+  doi       = {10.1145/3625135.3625154}
+}
+```