README.md

metadata

license: cc-by-nc-4.0
pipeline_tag: audio-classification
base_model:
  - m-a-p/MERT-v1-95M
  - m-a-p/MERT-v1-330M
base_model_relation: finetune
tags:
  - pytorch
  - Transformers
  - music
  - streaming
  - music-information-retrieval
  - feature-extraction
  - onnx

MuSTream: A Streaming Music Representation for Online Music Understanding

MuSTream is a streaming music representation for online music understanding. A two-stage distillation (capacity, then causality) turns the non-causal MERT into causal streaming models of 26M, 95M and 330M parameters. They process audio as a stream, give one prediction every 0.2 s with a block-wise KV cache, and reach near-offline performance on chord recognition, beat and downbeat tracking and key detection, in real time on a CPU.

• Code: https://github.com/yamaha-research/MuSTream
• Real-time demo: https://yamaha-research-mustream.static.hf.space

What is included

Six streaming backbones and, for each, three task probes, in PyTorch and ONNX:

Model	Parameters	Teacher
MuSTream-26M	26M	26M bidirectional student distilled from MERT-v1-95M
MuSTream-95M	95M	MERT-v1-95M
MuSTream-330M	330M	MERT-v1-330M

On an Apple M1 CPU, MuSTream-26M runs at a real-time factor of 0.09 with 0.22 s latency, and MuSTream-330M at 0.35 with 0.29 s.

• Variants r0 / r15. The local-interval teacher used in distillation sees $p = 0$ or $p = 15$ frames past each supervised block.
• Streaming setup. 24 kHz mono input, 75 fps features. Each frame attends to the past plus the rest of its 15-frame (0.2 s) block, over a 750-frame (10 s) KV cache.
• Probes. LSTM probes for chord recognition, beat / downbeat tracking and key detection on top of each backbone.
• Training data. The released backbones are distilled on the public MTG-Jamendo audio (no labels). The paper's results use backbones distilled on internal data; see docs/understanding_performance.md for a comparison of the two.

pytorch/
├── backbone/                   # MuSTream backbones
│   └── streaming_{26M,95M,330M}_{r0,r15}.ckpt
└── {chord,key,beat}/           # LSTM probes, one folder per backbone
    └── streaming_{26M,95M,330M}_{r0,r15}/{best.pt, config.yaml}
onnx/                           # the same models in ONNX
├── backbone/
│   └── streaming_{26M,95M,330M}_{r0,r15}.streaming.{onnx,json}
└── {chord,key,beat}/
    └── streaming_{26M,95M,330M}_{r0,r15}.streaming.{onnx,json}

How to use

Real-time demo in the browser

Open https://yamaha-research-mustream.static.hf.space, choose a task (chord, beat, key or all), upload a song and press Start streaming, or press Try a sample. The ONNX models run on your own device with ONNX Runtime Web: nothing to install, and your audio is never uploaded. It uses MuSTream-26M by default. To run larger models or the demo locally, see streaming-demo/README.md.

Streaming feature extraction

Set up the code with uv and download this repository into mustream/ckpt/:

git clone https://github.com/yamaha-research/MuSTream.git && cd MuSTream
uv sync && source .venv/bin/activate
hf download yamaha-research/MuSTream --local-dir mustream/ckpt

Extract streaming features from your own audio (--model 26M, 95M or 330M; --variant r0 or r15):

python mustream/extract_features.py song.wav --model 95M --variant r15
# -> features/song.npz: `features` [L, T, D] at 75 fps (L = 13, D = 768 for 95M)

From Python:

import sys; sys.path.insert(0, "mustream")
from extract_features import load_backbone, load_audio, extract

model = load_backbone("mustream/ckpt/pytorch/backbone/streaming_95M_r15.ckpt", device="cuda")  # or "cpu"
features = extract(model, load_audio("song.wav"))     # torch.Tensor [L, T, D], 75 fps

Chord, beat and key predictions

Each probe's config.yaml already points at its backbone, so the probes run as downloaded. For example, streaming chord recognition:

cd mustream/chord_recognition
python infer_streaming.py ../ckpt/pytorch/chord/streaming_95M_r15/config.yaml \
                          ../ckpt/pytorch/chord/streaming_95M_r15/best.pt --audio song.wav

See the GitHub repository for beat and key detection, ONNX inference without PyTorch, training and evaluation.

License

Copyright 2026 Yamaha Corporation and the MuSTream authors.

The model weights in this repository (backbones and probes, in PyTorch and ONNX) are licensed under the Creative Commons Attribution-NonCommercial 4.0 International License (CC BY-NC 4.0). They are granted for non-commercial research purposes only.

The MuSTream backbones are distilled from MERT, whose weights are also released under CC BY-NC 4.0, using audio from MTG-Jamendo.

The code in the GitHub repository is licensed separately under the GNU Affero General Public License v3.0 (AGPL-3.0).