| --- |
| license: cc-by-4.0 |
| library_name: amtflow |
| tags: |
| - audio |
| - music |
| - instrument-recognition |
| - music-transcription |
| datasets: |
| - openmic-2018 |
| --- |
| |
| # AMTFlow |
|
|
| Model assets for [AMTFlow](https://github.com/vaclisinc/AMTFlow), a modular |
| instrument-conditioned automatic music transcription pipeline (instrument tagging → |
| instrument-family routing → MuScriptor). AMTFlow downloads these files on first use; |
| you do not need to fetch them by hand. |
|
|
| ## `heads/openmic/` — instrument-tagging heads |
|
|
| One classification head per frozen audio encoder, trained on |
| [OpenMIC-2018](https://zenodo.org/records/1432913) (20 instrument classes). AMTFlow's |
| `probe` tagger runs the encoder, applies the head to one layer's time-averaged |
| features, and marks an instrument family present when any of its classes reaches that |
| class's threshold. Several taggers, including audio-language models, can be combined |
| by union or majority vote. |
|
|
| | head | encoder | layer | OpenMIC test mAP | macro-F1 | |
| |---|---|---:|---:|---:| |
| | `openmic/af3-whisper-mlp` | [nvidia/audio-flamingo-3-hf](https://huggingface.co/nvidia/audio-flamingo-3-hf) audio tower | 31 | 0.861 ± 0.001 | 0.793 | |
| | `openmic/afnext-whisper-mlp` | [nvidia/audio-flamingo-next-hf](https://huggingface.co/nvidia/audio-flamingo-next-hf) audio tower | 29 | 0.860 ± 0.003 | 0.788 | |
| | `openmic/matpac-music-mlp` | [auriankelen/matpac_music](https://huggingface.co/auriankelen/matpac_music) | 10 | 0.860 ± 0.002 | 0.792 | |
| | `openmic/matpac-general-mlp` | [auriankelen/matpac_general_audio](https://huggingface.co/auriankelen/matpac_general_audio) | 10 | 0.858 ± 0.002 | 0.787 | |
| | `openmic/pupum2d-mlp` | [spellbrush/PupuM2D](https://huggingface.co/spellbrush/PupuM2D) (large) | 19 | 0.858 ± 0.001 | 0.790 | |
| | `openmic/muq-mlp` | [OpenMuQ/MuQ-large-msd-iter](https://huggingface.co/OpenMuQ/MuQ-large-msd-iter) | 2 | 0.832 ± 0.001 | 0.764 | |
| | `openmic/qwen3-omni-aut-mlp` | [Qwen/Qwen3-Omni-30B-A3B-Instruct](https://huggingface.co/Qwen/Qwen3-Omni-30B-A3B-Instruct) AuT encoder | 23 | 0.821 ± 0.001 | 0.753 | |
| | `openmic/mert-mlp` | [m-a-p/MERT-v1-330M](https://huggingface.co/m-a-p/MERT-v1-330M) | 9 | 0.818 ± 0.002 | 0.740 | |
|
|
| Test mAP is the mean over five probe seeds with a 95% t-interval over seeds (the file |
| is the best-validation seed). For reference, the MATPAC++ authors' released OpenMIC |
| probe scores 0.861 on the same test clips. |
|
|
| **Protocol.** Official `split01` test partition (5,085 clips); the official train |
| partition split 80/20 by track (seed 0) into train/validation. Labels: aggregated |
| relevance ≥ 0.5 is positive; unobserved (clip, class) labels are masked out. Every |
| encoder layer is averaged over the 10 s clip and z-scored on train; the head is an MLP |
| (512 hidden, ReLU, dropout 0.2) trained with masked BCE; layer and learning rate are |
| selected on validation mAP. Per-class thresholds are calibrated on validation by |
| F-beta (β = 2, precision ≥ 0.25), i.e. recall-oriented, because the transcriber |
| cannot produce an instrument outside its conditioning set. |
|
|
| **Files.** `<encoder>-mlp.safetensors` holds the head weights and the train-set |
| feature mean/std; its metadata (`amtflow` key, JSON) holds the encoder, layer, class |
| names, thresholds and calibration details. `index.json` lists every head with its |
| test scores and SHA-256. |
|
|
| ## License |
|
|
| The heads are released under CC BY 4.0 (OpenMIC-2018 is CC BY 4.0). A head is only |
| usable together with its encoder, and the encoder's own license applies: MATPAC++ |
| (Apache-2.0), PupuM2D (MIT), Qwen3-Omni (Apache-2.0), MERT and MuQ (CC BY-NC 4.0), |
| Audio Flamingo 3 / Next (NVIDIA non-commercial). MuScriptor, the transcriber AMTFlow |
| uses, is CC BY-NC 4.0. |
|
|
| ## Citation |
|
|
| AMTFlow is the transcription pipeline of the |
| [MIDI-REPA paper](https://arxiv.org/abs/2610.20967). If you use it, please cite: |
|
|
| ```bibtex |
| @misc{yu2026midirepa, |
| title={MIDI-REPA: Improving Text-to-Music Generation with Symbolic Representation Alignment}, |
| author={Song-Ze Yu and Fang-Duo Tsai and Yi-Hsuan Yang}, |
| year={2026}, |
| eprint={2610.20967}, |
| archivePrefix={arXiv}, |
| primaryClass={cs.SD}, |
| url={https://arxiv.org/abs/2610.20967}, |
| } |
| ``` |
|
|
