ASLP-lab commited on
Commit
cc32c9f
·
verified ·
1 Parent(s): 2f67e62

Upload folder using huggingface_hub

Browse files
Files changed (5) hide show
  1. .gitignore +37 -0
  2. README.md +48 -0
  3. bigvgan_v2_48khz_25hz_base_vae64.json +15 -0
  4. config.yml +16 -0
  5. model.pt +3 -0
.gitignore ADDED
@@ -0,0 +1,37 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ .DS_Store
2
+ *.7z filter=lfs diff=lfs merge=lfs -text
3
+ *.arrow filter=lfs diff=lfs merge=lfs -text
4
+ *.bin filter=lfs diff=lfs merge=lfs -text
5
+ *.bz2 filter=lfs diff=lfs merge=lfs -text
6
+ *.ckpt filter=lfs diff=lfs merge=lfs -text
7
+ *.ftz filter=lfs diff=lfs merge=lfs -text
8
+ *.gz filter=lfs diff=lfs merge=lfs -text
9
+ *.h5 filter=lfs diff=lfs merge=lfs -text
10
+ *.joblib filter=lfs diff=lfs merge=lfs -text
11
+ *.lfs.* filter=lfs diff=lfs merge=lfs -text
12
+ *.mlmodel filter=lfs diff=lfs merge=lfs -text
13
+ *.model filter=lfs diff=lfs merge=lfs -text
14
+ *.msgpack filter=lfs diff=lfs merge=lfs -text
15
+ *.npy filter=lfs diff=lfs merge=lfs -text
16
+ *.npz filter=lfs diff=lfs merge=lfs -text
17
+ *.onnx filter=lfs diff=lfs merge=lfs -text
18
+ *.ot filter=lfs diff=lfs merge=lfs -text
19
+ *.parquet filter=lfs diff=lfs merge=lfs -text
20
+ *.pb filter=lfs diff=lfs merge=lfs -text
21
+ *.pickle filter=lfs diff=lfs merge=lfs -text
22
+ *.pkl filter=lfs diff=lfs merge=lfs -text
23
+ *.pt filter=lfs diff=lfs merge=lfs -text
24
+ *.pth filter=lfs diff=lfs merge=lfs -text
25
+ *.rar filter=lfs diff=lfs merge=lfs -text
26
+ *.safetensors filter=lfs diff=lfs merge=lfs -text
27
+ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
28
+ *.tar.* filter=lfs diff=lfs merge=lfs -text
29
+ *.tar filter=lfs diff=lfs merge=lfs -text
30
+ *.tflite filter=lfs diff=lfs merge=lfs -text
31
+ *.tgz filter=lfs diff=lfs merge=lfs -text
32
+ *.wasm filter=lfs diff=lfs merge=lfs -text
33
+ *.xz filter=lfs diff=lfs merge=lfs -text
34
+ *.zip filter=lfs diff=lfs merge=lfs -text
35
+ *.zst filter=lfs diff=lfs merge=lfs -text
36
+ *tfevents* filter=lfs diff=lfs merge=lfs -text
37
+
README.md CHANGED
@@ -1,3 +1,51 @@
1
  ---
2
  license: apache-2.0
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  license: apache-2.0
3
  ---
4
+
5
+ # MSVAE: Rethinking Musical Semantics in Continuous Latent Spaces
6
+
7
+ MSVAE is a music variational autoencoder with a structured, continuous latent space. This repository contains the **Stage II reconstruction checkpoint**: it accepts stereo audio at 24 kHz and produces stereo audio at 48 kHz. The encoder runs at 24 kHz, while the decoder generates the 48 kHz waveform. This is an audio reconstruction model, not a standalone text-to-music generator.
8
+
9
+ The model combines a frozen music self-supervised encoder with an acoustic encoder through a gated residual posterior. During training, separate parts of the 64-dimensional latent are aligned with compositional and linguistic-vocal references. The released checkpoint is for inference; training-time alignment features are not required to reconstruct audio.
10
+
11
+ ## Files
12
+
13
+ | File | Description |
14
+ | --- | --- |
15
+ | `model.pt` | Stage II MSVAE reconstruction weights. The checkpoint contains a `model_state_dict`; it does not bundle the pretrained MuQ encoder. |
16
+ | `config.yml` | Inference settings: 24 kHz encoder input, 48 kHz stereo output, and model dimensions. |
17
+ | `bigvgan_v2_48khz_25hz_base_vae64.json` | Decoder configuration, also included in the companion code repository. |
18
+
19
+ The Stage I 24 kHz-to-24 kHz checkpoint is **not** included here. The `model.pt` file is approximately 721 MiB.
20
+
21
+ ## Requirements
22
+
23
+ Inference requires the accompanying MSVAE code and its Python environment. The checkpoint is not directly loadable through `transformers.AutoModel`. The model also uses [OpenMuQ/MuQ-large-msd-iter](https://huggingface.co/OpenMuQ/MuQ-large-msd-iter), which the inference code loads separately with `MuQ.from_pretrained(...)`; MuQ weights are not redistributed in this repository. Please follow the [MuQ model license](https://huggingface.co/OpenMuQ/MuQ-large-msd-iter) for that dependency.
24
+
25
+ The code repository, installation instructions, paper, and audio demo links will be added when their public releases are available.
26
+
27
+ ## Reconstruction
28
+
29
+ From the root of the accompanying MSVAE code repository, after installing its runtime dependencies:
30
+
31
+ ```bash
32
+ CKPT_PATH=/absolute/path/to/model.pt \
33
+ bash scripts/infer.sh /absolute/path/to/input.wav ./reconstructions 0
34
+ ```
35
+
36
+ The third positional argument selects GPU `0`. The script resamples input audio to 24 kHz before encoding and writes a 48 kHz stereo WAV file under `./reconstructions`. Mono input is duplicated to stereo. By default, it samples the VAE latent with seed 42. For deterministic decoding of the posterior mean, append `--latent-mode mean`:
37
+
38
+ ```bash
39
+ CKPT_PATH=/absolute/path/to/model.pt \
40
+ bash scripts/infer.sh /absolute/path/to/input.wav ./reconstructions 0 --latent-mode mean
41
+ ```
42
+
43
+ The default inference configuration in the code repository, `src/config/svae/example_new.yml`, has the same model settings as `config.yml` here. Its `DAC.bigvgan_conf` path is resolved relative to the code repository root, where the same decoder JSON is provided.
44
+
45
+ ## Intended use and limitations
46
+
47
+ This checkpoint is intended for music reconstruction research and listening comparisons. The 48 kHz output is generated from a 24 kHz input; high-frequency detail is inferred by the decoder rather than recovered from the original waveform. Performance on other audio domains has not been established here. Use only input audio you are entitled to process and publish.
48
+
49
+ ## License and citation
50
+
51
+ The MSVAE checkpoint license and paper citation have not yet been specified. Add the chosen license and public paper reference before publishing this model repository. The external MuQ weights have their own [CC BY-NC 4.0 license](https://huggingface.co/OpenMuQ/MuQ-large-msd-iter).
bigvgan_v2_48khz_25hz_base_vae64.json ADDED
@@ -0,0 +1,15 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "resblock": "1",
3
+ "num_mels": 64,
4
+ "out_channels": 2,
5
+ "upsample_rates": [5,4,4,3,2,2,2],
6
+ "upsample_kernel_sizes": [9,8,8,5,4,4,4],
7
+ "upsample_initial_channel": 1536,
8
+ "resblock_kernel_sizes": [3,7,11],
9
+ "resblock_dilation_sizes": [[1,3,5], [1,3,5], [1,3,5]],
10
+ "use_tanh_at_final": false,
11
+ "use_bias_at_final": false,
12
+ "activation": "snakebeta",
13
+ "snake_logscale": true
14
+
15
+ }
config.yml ADDED
@@ -0,0 +1,16 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Inference config for msvae_24kto48k/model.pt.
2
+ model_type: DAC
3
+ DAC.input_sample_rate: 24000
4
+ DAC.sample_rate: 48000
5
+ DAC.encoder_dim: 128
6
+ DAC.encoder_rates: [4, 5, 6, 8]
7
+ DAC.decoder_dim: 1536
8
+ DAC.decoder_rates: [5, 4, 4, 3, 2, 2, 2]
9
+ DAC.vae_dim: 64
10
+ DAC.in_channels: 2
11
+ DAC.out_channels: 2
12
+ DAC.bigvgan_conf: src/config/bigvgan_conf/bigvgan_v2_48khz_25hz_base_vae64.json
13
+ DAC.distill: false
14
+ DAC.muq_model_name: OpenMuQ/MuQ-large-msd-iter
15
+ DAC.muq_hidden_state_layer: 6
16
+ DAC.muq_channel_fusion: mean
model.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:61f0c50616b9d2d2df65da97d943b3b1dad5d45cd18d6613ff4f1bb4f060f376
3
+ size 756399522