---
license: cc-by-nc-4.0
pipeline_tag: audio-classification
base_model:
  - m-a-p/MERT-v1-95M
  - m-a-p/MERT-v1-330M
base_model_relation: finetune
tags:
  - pytorch
  - Transformers
  - music
  - streaming
  - music-information-retrieval
  - feature-extraction
  - onnx
---

# MuSTream: A Streaming Music Representation for Online Music Understanding

MuSTream is a streaming music representation for online music understanding. A
two-stage distillation (capacity, then causality) turns the non-causal
[MERT](https://huggingface.co/m-a-p/MERT-v1-95M) into causal streaming models of 26M,
95M and 330M parameters. They process audio as a stream, give one prediction every
0.2 s with a block-wise KV cache, and reach near-offline performance on chord
recognition, beat and downbeat tracking and key detection, in real time on a CPU.

- **Code:** <https://github.com/yamaha-research/MuSTream>
- **Real-time demo:** <https://yamaha-research-mustream.static.hf.space>

## What is included

Six streaming backbones and, for each, three task probes, in PyTorch and ONNX:

| Model | Parameters | Teacher |
|---|---|---|
| MuSTream-26M | 26M | 26M bidirectional student distilled from MERT-v1-95M |
| MuSTream-95M | 95M | MERT-v1-95M |
| MuSTream-330M | 330M | MERT-v1-330M |

On an Apple M1 CPU, MuSTream-26M runs at a real-time factor of 0.09 with 0.22 s
latency, and MuSTream-330M at 0.35 with 0.29 s.

- **Variants `r0` / `r15`.** The local-interval teacher used in distillation sees
  $p = 0$ or $p = 15$ frames past each supervised block.
- **Streaming setup.** 24 kHz mono input, 75 fps features. Each frame attends to the
  past plus the rest of its 15-frame (0.2 s) block, over a 750-frame (10 s) KV cache.
- **Probes.** LSTM probes for chord recognition, beat / downbeat tracking and key
  detection on top of each backbone.
- **Training data.** The released backbones are distilled on the public
  [MTG-Jamendo](https://github.com/MTG/mtg-jamendo-dataset) audio (no labels).
  The paper's results use backbones distilled on internal data; see
  [`docs/understanding_performance.md`](https://github.com/yamaha-research/MuSTream/blob/main/docs/understanding_performance.md)
  for a comparison of the two.

```
pytorch/
├── backbone/                   # MuSTream backbones
│   └── streaming_{26M,95M,330M}_{r0,r15}.ckpt
└── {chord,key,beat}/           # LSTM probes, one folder per backbone
    └── streaming_{26M,95M,330M}_{r0,r15}/{best.pt, config.yaml}
onnx/                           # the same models in ONNX
├── backbone/
│   └── streaming_{26M,95M,330M}_{r0,r15}.streaming.{onnx,json}
└── {chord,key,beat}/
    └── streaming_{26M,95M,330M}_{r0,r15}.streaming.{onnx,json}
```

## How to use

### Real-time demo in the browser

Open **<https://yamaha-research-mustream.static.hf.space>**, choose a task (chord,
beat, key or all), upload a song and press *Start streaming*, or press *Try a sample*.
The ONNX models run on your own device with ONNX Runtime Web: nothing to install, and
your audio is never uploaded. It uses MuSTream-26M by default. To run larger models or
the demo locally, see
[`streaming-demo/README.md`](https://github.com/yamaha-research/MuSTream/blob/main/streaming-demo/README.md).

### Streaming feature extraction

Set up the code with [uv](https://docs.astral.sh/uv/) and download this repository
into `mustream/ckpt/`:

```bash
git clone https://github.com/yamaha-research/MuSTream.git && cd MuSTream
uv sync && source .venv/bin/activate
hf download yamaha-research/MuSTream --local-dir mustream/ckpt
```

Extract streaming features from your own audio (`--model` 26M, 95M or 330M;
`--variant` `r0` or `r15`):

```bash
python mustream/extract_features.py song.wav --model 95M --variant r15
# -> features/song.npz: `features` [L, T, D] at 75 fps (L = 13, D = 768 for 95M)
```

From Python:

```python
import sys; sys.path.insert(0, "mustream")
from extract_features import load_backbone, load_audio, extract

model = load_backbone("mustream/ckpt/pytorch/backbone/streaming_95M_r15.ckpt", device="cuda")  # or "cpu"
features = extract(model, load_audio("song.wav"))     # torch.Tensor [L, T, D], 75 fps
```

### Chord, beat and key predictions

Each probe's `config.yaml` already points at its backbone, so the probes run as
downloaded. For example, streaming chord recognition:

```bash
cd mustream/chord_recognition
python infer_streaming.py ../ckpt/pytorch/chord/streaming_95M_r15/config.yaml \
                          ../ckpt/pytorch/chord/streaming_95M_r15/best.pt --audio song.wav
```

See the [GitHub repository](https://github.com/yamaha-research/MuSTream) for beat and
key detection, ONNX inference without PyTorch, training and evaluation.

## License

Copyright 2026 Yamaha Corporation and the MuSTream authors.

The model weights in this repository (backbones and probes, in PyTorch and ONNX) are
licensed under the
[Creative Commons Attribution-NonCommercial 4.0 International License](https://creativecommons.org/licenses/by-nc/4.0/)
(CC BY-NC 4.0). They are granted for non-commercial research purposes only.

The MuSTream backbones are distilled from [MERT](https://huggingface.co/m-a-p/MERT-v1-95M),
whose weights are also released under CC BY-NC 4.0, using audio from
[MTG-Jamendo](https://github.com/MTG/mtg-jamendo-dataset).

The code in the [GitHub repository](https://github.com/yamaha-research/MuSTream) is
licensed separately under the GNU Affero General Public License v3.0 (AGPL-3.0).