| --- |
| license: apache-2.0 |
| --- |
| |
| A [SoundStream](https://arxiv.org/abs/2107.03312) decoder to reconstruct audio from a mel-spectrogram. |
|
|
| ## Overview |
|
|
| This model is a SoundStream decoder which inverts mel-spectrograms computed with the specific hyperparameters defined in the example below. This model was trained on music data and used in [Multi-instrument Music Synthesis with Spectrogram Diffusion](https://arxiv.org/abs/2206.05408) (ISMIR 2022). |
|
|
| A typical use-case is to simplify music generation by predicting mel-spectrograms (instead of a raw waveform), and then use this model to reconstruct audio. |
|
|
| If you use it, please consider citing: |
|
|
| ```bibtex |
| @article{zeghidour2021soundstream, |
| title={Soundstream: An end-to-end neural audio codec}, |
| author={Zeghidour, Neil and Luebs, Alejandro and Omran, Ahmed and Skoglund, Jan and Tagliasacchi, Marco}, |
| journal={IEEE/ACM Transactions on Audio, Speech, and Language Processing}, |
| volume={30}, |
| pages={495--507}, |
| year={2021}, |
| publisher={IEEE} |
| } |
| ``` |
|
|
| ## Example Use |
|
|
| ```python |
| from diffusers import OnnxRuntimeModel |
| |
| |
| SAMPLE_RATE = 16000 |
| N_FFT = 1024 |
| HOP_LENGTH = 320 |
| WIN_LENGTH = 640 |
| N_MEL_CHANNELS = 128 |
| MEL_FMIN = 0.0 |
| MEL_FMAX = int(SAMPLE_RATE // 2) |
| CLIP_VALUE_MIN = 1e-5 |
| CLIP_VALUE_MAX = 1e8 |
| |
| mel = ... |
| |
| melgan = OnnxRuntimeModel.from_pretrained("kashif/soundstream_mel_decoder") |
| |
| audio = melgan(input_features=mel.astype(np.float32)) |
| ``` |
