| --- |
| license: cc-by-nc-4.0 |
| pipeline_tag: audio-classification |
| base_model: |
| - m-a-p/MERT-v1-95M |
| - m-a-p/MERT-v1-330M |
| base_model_relation: finetune |
| tags: |
| - pytorch |
| - Transformers |
| - music |
| - streaming |
| - music-information-retrieval |
| - feature-extraction |
| - onnx |
| --- |
| |
| # MuSTream: A Streaming Music Representation for Online Music Understanding |
|
|
| MuSTream is a streaming music representation for online music understanding. A |
| two-stage distillation (capacity, then causality) turns the non-causal |
| [MERT](https://huggingface.co/m-a-p/MERT-v1-95M) into causal streaming models of 26M, |
| 95M and 330M parameters. They process audio as a stream, give one prediction every |
| 0.2 s with a block-wise KV cache, and reach near-offline performance on chord |
| recognition, beat and downbeat tracking and key detection, in real time on a CPU. |
|
|
| - **Code:** <https://github.com/yamaha-research/MuSTream> |
| - **Real-time demo:** <https://yamaha-research-mustream.static.hf.space> |
|
|
| ## What is included |
|
|
| Six streaming backbones and, for each, three task probes, in PyTorch and ONNX: |
|
|
| | Model | Parameters | Teacher | |
| |---|---|---| |
| | MuSTream-26M | 26M | 26M bidirectional student distilled from MERT-v1-95M | |
| | MuSTream-95M | 95M | MERT-v1-95M | |
| | MuSTream-330M | 330M | MERT-v1-330M | |
|
|
| On an Apple M1 CPU, MuSTream-26M runs at a real-time factor of 0.09 with 0.22 s |
| latency, and MuSTream-330M at 0.35 with 0.29 s. |
|
|
| - **Variants `r0` / `r15`.** The local-interval teacher used in distillation sees |
| $p = 0$ or $p = 15$ frames past each supervised block. |
| - **Streaming setup.** 24 kHz mono input, 75 fps features. Each frame attends to the |
| past plus the rest of its 15-frame (0.2 s) block, over a 750-frame (10 s) KV cache. |
| - **Probes.** LSTM probes for chord recognition, beat / downbeat tracking and key |
| detection on top of each backbone. |
| - **Training data.** The released backbones are distilled on the public |
| [MTG-Jamendo](https://github.com/MTG/mtg-jamendo-dataset) audio (no labels). |
| The paper's results use backbones distilled on internal data; see |
| [`docs/understanding_performance.md`](https://github.com/yamaha-research/MuSTream/blob/main/docs/understanding_performance.md) |
| for a comparison of the two. |
|
|
| ``` |
| pytorch/ |
| ├── backbone/ # MuSTream backbones |
| │ └── streaming_{26M,95M,330M}_{r0,r15}.ckpt |
| └── {chord,key,beat}/ # LSTM probes, one folder per backbone |
| └── streaming_{26M,95M,330M}_{r0,r15}/{best.pt, config.yaml} |
| onnx/ # the same models in ONNX |
| ├── backbone/ |
| │ └── streaming_{26M,95M,330M}_{r0,r15}.streaming.{onnx,json} |
| └── {chord,key,beat}/ |
| └── streaming_{26M,95M,330M}_{r0,r15}.streaming.{onnx,json} |
| ``` |
|
|
| ## How to use |
|
|
| ### Real-time demo in the browser |
|
|
| Open **<https://yamaha-research-mustream.static.hf.space>**, choose a task (chord, |
| beat, key or all), upload a song and press *Start streaming*, or press *Try a sample*. |
| The ONNX models run on your own device with ONNX Runtime Web: nothing to install, and |
| your audio is never uploaded. It uses MuSTream-26M by default. To run larger models or |
| the demo locally, see |
| [`streaming-demo/README.md`](https://github.com/yamaha-research/MuSTream/blob/main/streaming-demo/README.md). |
|
|
| ### Streaming feature extraction |
|
|
| Set up the code with [uv](https://docs.astral.sh/uv/) and download this repository |
| into `mustream/ckpt/`: |
|
|
| ```bash |
| git clone https://github.com/yamaha-research/MuSTream.git && cd MuSTream |
| uv sync && source .venv/bin/activate |
| hf download yamaha-research/MuSTream --local-dir mustream/ckpt |
| ``` |
|
|
| Extract streaming features from your own audio (`--model` 26M, 95M or 330M; |
| `--variant` `r0` or `r15`): |
|
|
| ```bash |
| python mustream/extract_features.py song.wav --model 95M --variant r15 |
| # -> features/song.npz: `features` [L, T, D] at 75 fps (L = 13, D = 768 for 95M) |
| ``` |
|
|
| From Python: |
|
|
| ```python |
| import sys; sys.path.insert(0, "mustream") |
| from extract_features import load_backbone, load_audio, extract |
| |
| model = load_backbone("mustream/ckpt/pytorch/backbone/streaming_95M_r15.ckpt", device="cuda") # or "cpu" |
| features = extract(model, load_audio("song.wav")) # torch.Tensor [L, T, D], 75 fps |
| ``` |
|
|
| ### Chord, beat and key predictions |
|
|
| Each probe's `config.yaml` already points at its backbone, so the probes run as |
| downloaded. For example, streaming chord recognition: |
|
|
| ```bash |
| cd mustream/chord_recognition |
| python infer_streaming.py ../ckpt/pytorch/chord/streaming_95M_r15/config.yaml \ |
| ../ckpt/pytorch/chord/streaming_95M_r15/best.pt --audio song.wav |
| ``` |
|
|
| See the [GitHub repository](https://github.com/yamaha-research/MuSTream) for beat and |
| key detection, ONNX inference without PyTorch, training and evaluation. |
|
|
| ## License |
|
|
| Copyright 2026 Yamaha Corporation and the MuSTream authors. |
|
|
| The model weights in this repository (backbones and probes, in PyTorch and ONNX) are |
| licensed under the |
| [Creative Commons Attribution-NonCommercial 4.0 International License](https://creativecommons.org/licenses/by-nc/4.0/) |
| (CC BY-NC 4.0). They are granted for non-commercial research purposes only. |
|
|
| The MuSTream backbones are distilled from [MERT](https://huggingface.co/m-a-p/MERT-v1-95M), |
| whose weights are also released under CC BY-NC 4.0, using audio from |
| [MTG-Jamendo](https://github.com/MTG/mtg-jamendo-dataset). |
|
|
| The code in the [GitHub repository](https://github.com/yamaha-research/MuSTream) is |
| licensed separately under the GNU Affero General Public License v3.0 (AGPL-3.0). |
|
|
