Upload folder using huggingface_hub
Browse files- .gitignore +37 -0
- README.md +48 -0
- bigvgan_v2_48khz_25hz_base_vae64.json +15 -0
- config.yml +16 -0
- model.pt +3 -0
.gitignore
ADDED
|
@@ -0,0 +1,37 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
.DS_Store
|
| 2 |
+
*.7z filter=lfs diff=lfs merge=lfs -text
|
| 3 |
+
*.arrow filter=lfs diff=lfs merge=lfs -text
|
| 4 |
+
*.bin filter=lfs diff=lfs merge=lfs -text
|
| 5 |
+
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
| 6 |
+
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
| 7 |
+
*.ftz filter=lfs diff=lfs merge=lfs -text
|
| 8 |
+
*.gz filter=lfs diff=lfs merge=lfs -text
|
| 9 |
+
*.h5 filter=lfs diff=lfs merge=lfs -text
|
| 10 |
+
*.joblib filter=lfs diff=lfs merge=lfs -text
|
| 11 |
+
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
| 12 |
+
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
| 13 |
+
*.model filter=lfs diff=lfs merge=lfs -text
|
| 14 |
+
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
| 15 |
+
*.npy filter=lfs diff=lfs merge=lfs -text
|
| 16 |
+
*.npz filter=lfs diff=lfs merge=lfs -text
|
| 17 |
+
*.onnx filter=lfs diff=lfs merge=lfs -text
|
| 18 |
+
*.ot filter=lfs diff=lfs merge=lfs -text
|
| 19 |
+
*.parquet filter=lfs diff=lfs merge=lfs -text
|
| 20 |
+
*.pb filter=lfs diff=lfs merge=lfs -text
|
| 21 |
+
*.pickle filter=lfs diff=lfs merge=lfs -text
|
| 22 |
+
*.pkl filter=lfs diff=lfs merge=lfs -text
|
| 23 |
+
*.pt filter=lfs diff=lfs merge=lfs -text
|
| 24 |
+
*.pth filter=lfs diff=lfs merge=lfs -text
|
| 25 |
+
*.rar filter=lfs diff=lfs merge=lfs -text
|
| 26 |
+
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
| 27 |
+
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
| 28 |
+
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
| 29 |
+
*.tar filter=lfs diff=lfs merge=lfs -text
|
| 30 |
+
*.tflite filter=lfs diff=lfs merge=lfs -text
|
| 31 |
+
*.tgz filter=lfs diff=lfs merge=lfs -text
|
| 32 |
+
*.wasm filter=lfs diff=lfs merge=lfs -text
|
| 33 |
+
*.xz filter=lfs diff=lfs merge=lfs -text
|
| 34 |
+
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 35 |
+
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 36 |
+
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
| 37 |
+
|
README.md
CHANGED
|
@@ -1,3 +1,51 @@
|
|
| 1 |
---
|
| 2 |
license: apache-2.0
|
| 3 |
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
license: apache-2.0
|
| 3 |
---
|
| 4 |
+
|
| 5 |
+
# MSVAE: Rethinking Musical Semantics in Continuous Latent Spaces
|
| 6 |
+
|
| 7 |
+
MSVAE is a music variational autoencoder with a structured, continuous latent space. This repository contains the **Stage II reconstruction checkpoint**: it accepts stereo audio at 24 kHz and produces stereo audio at 48 kHz. The encoder runs at 24 kHz, while the decoder generates the 48 kHz waveform. This is an audio reconstruction model, not a standalone text-to-music generator.
|
| 8 |
+
|
| 9 |
+
The model combines a frozen music self-supervised encoder with an acoustic encoder through a gated residual posterior. During training, separate parts of the 64-dimensional latent are aligned with compositional and linguistic-vocal references. The released checkpoint is for inference; training-time alignment features are not required to reconstruct audio.
|
| 10 |
+
|
| 11 |
+
## Files
|
| 12 |
+
|
| 13 |
+
| File | Description |
|
| 14 |
+
| --- | --- |
|
| 15 |
+
| `model.pt` | Stage II MSVAE reconstruction weights. The checkpoint contains a `model_state_dict`; it does not bundle the pretrained MuQ encoder. |
|
| 16 |
+
| `config.yml` | Inference settings: 24 kHz encoder input, 48 kHz stereo output, and model dimensions. |
|
| 17 |
+
| `bigvgan_v2_48khz_25hz_base_vae64.json` | Decoder configuration, also included in the companion code repository. |
|
| 18 |
+
|
| 19 |
+
The Stage I 24 kHz-to-24 kHz checkpoint is **not** included here. The `model.pt` file is approximately 721 MiB.
|
| 20 |
+
|
| 21 |
+
## Requirements
|
| 22 |
+
|
| 23 |
+
Inference requires the accompanying MSVAE code and its Python environment. The checkpoint is not directly loadable through `transformers.AutoModel`. The model also uses [OpenMuQ/MuQ-large-msd-iter](https://huggingface.co/OpenMuQ/MuQ-large-msd-iter), which the inference code loads separately with `MuQ.from_pretrained(...)`; MuQ weights are not redistributed in this repository. Please follow the [MuQ model license](https://huggingface.co/OpenMuQ/MuQ-large-msd-iter) for that dependency.
|
| 24 |
+
|
| 25 |
+
The code repository, installation instructions, paper, and audio demo links will be added when their public releases are available.
|
| 26 |
+
|
| 27 |
+
## Reconstruction
|
| 28 |
+
|
| 29 |
+
From the root of the accompanying MSVAE code repository, after installing its runtime dependencies:
|
| 30 |
+
|
| 31 |
+
```bash
|
| 32 |
+
CKPT_PATH=/absolute/path/to/model.pt \
|
| 33 |
+
bash scripts/infer.sh /absolute/path/to/input.wav ./reconstructions 0
|
| 34 |
+
```
|
| 35 |
+
|
| 36 |
+
The third positional argument selects GPU `0`. The script resamples input audio to 24 kHz before encoding and writes a 48 kHz stereo WAV file under `./reconstructions`. Mono input is duplicated to stereo. By default, it samples the VAE latent with seed 42. For deterministic decoding of the posterior mean, append `--latent-mode mean`:
|
| 37 |
+
|
| 38 |
+
```bash
|
| 39 |
+
CKPT_PATH=/absolute/path/to/model.pt \
|
| 40 |
+
bash scripts/infer.sh /absolute/path/to/input.wav ./reconstructions 0 --latent-mode mean
|
| 41 |
+
```
|
| 42 |
+
|
| 43 |
+
The default inference configuration in the code repository, `src/config/svae/example_new.yml`, has the same model settings as `config.yml` here. Its `DAC.bigvgan_conf` path is resolved relative to the code repository root, where the same decoder JSON is provided.
|
| 44 |
+
|
| 45 |
+
## Intended use and limitations
|
| 46 |
+
|
| 47 |
+
This checkpoint is intended for music reconstruction research and listening comparisons. The 48 kHz output is generated from a 24 kHz input; high-frequency detail is inferred by the decoder rather than recovered from the original waveform. Performance on other audio domains has not been established here. Use only input audio you are entitled to process and publish.
|
| 48 |
+
|
| 49 |
+
## License and citation
|
| 50 |
+
|
| 51 |
+
The MSVAE checkpoint license and paper citation have not yet been specified. Add the chosen license and public paper reference before publishing this model repository. The external MuQ weights have their own [CC BY-NC 4.0 license](https://huggingface.co/OpenMuQ/MuQ-large-msd-iter).
|
bigvgan_v2_48khz_25hz_base_vae64.json
ADDED
|
@@ -0,0 +1,15 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"resblock": "1",
|
| 3 |
+
"num_mels": 64,
|
| 4 |
+
"out_channels": 2,
|
| 5 |
+
"upsample_rates": [5,4,4,3,2,2,2],
|
| 6 |
+
"upsample_kernel_sizes": [9,8,8,5,4,4,4],
|
| 7 |
+
"upsample_initial_channel": 1536,
|
| 8 |
+
"resblock_kernel_sizes": [3,7,11],
|
| 9 |
+
"resblock_dilation_sizes": [[1,3,5], [1,3,5], [1,3,5]],
|
| 10 |
+
"use_tanh_at_final": false,
|
| 11 |
+
"use_bias_at_final": false,
|
| 12 |
+
"activation": "snakebeta",
|
| 13 |
+
"snake_logscale": true
|
| 14 |
+
|
| 15 |
+
}
|
config.yml
ADDED
|
@@ -0,0 +1,16 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Inference config for msvae_24kto48k/model.pt.
|
| 2 |
+
model_type: DAC
|
| 3 |
+
DAC.input_sample_rate: 24000
|
| 4 |
+
DAC.sample_rate: 48000
|
| 5 |
+
DAC.encoder_dim: 128
|
| 6 |
+
DAC.encoder_rates: [4, 5, 6, 8]
|
| 7 |
+
DAC.decoder_dim: 1536
|
| 8 |
+
DAC.decoder_rates: [5, 4, 4, 3, 2, 2, 2]
|
| 9 |
+
DAC.vae_dim: 64
|
| 10 |
+
DAC.in_channels: 2
|
| 11 |
+
DAC.out_channels: 2
|
| 12 |
+
DAC.bigvgan_conf: src/config/bigvgan_conf/bigvgan_v2_48khz_25hz_base_vae64.json
|
| 13 |
+
DAC.distill: false
|
| 14 |
+
DAC.muq_model_name: OpenMuQ/MuQ-large-msd-iter
|
| 15 |
+
DAC.muq_hidden_state_layer: 6
|
| 16 |
+
DAC.muq_channel_fusion: mean
|
model.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:61f0c50616b9d2d2df65da97d943b3b1dad5d45cd18d6613ff4f1bb4f060f376
|
| 3 |
+
size 756399522
|