| --- |
| license: mit |
| tags: |
| - music |
| - music-evaluation |
| - musicgen |
| - audio |
| library_name: pytorch |
| --- |
| |
| # MEva evaluator checkpoints |
|
|
| Trained CNN evaluators for the ISMIR 2026 paper |
| **"Do Music Generative Models Understand Musical Qualities? Automatic Music Evaluation with Model-Intrinsic Signals"** |
| (Xiaosha Li, Chun Liu, Ziyu Wang). |
|
|
| - Code: https://github.com/YoEv/MEva |
| - Demo: https://yoev.github.io/MEva |
|
|
| Each checkpoint maps intrinsic signals of a frozen MusicGen (per-token loss, |
| predictive entropy, and SAE latents under teacher forcing, delay pattern) to a |
| human quality rating. 14 variants: 7 signal combinations × MusicGen-{small,large}. |
|
|
| ## Layout |
|
|
| ``` |
| per_benchmark/{small,large}/{musiceval,songeval,aime,musicpref,music_arena}/ |
| f01_loss_only_cnn_clean_best.pth |
| f02_entropy_only_cnn_clean_best.pth |
| f03_sae_only_cnn_clean_best.pth |
| f04_loss_entropy_cnn_clean_best.pth |
| f05_entropy_sae_cnn_clean_best.pth |
| f06_loss_sae_cnn_clean_best.pth |
| f07_loss_entropy_sae_cnn_clean_best.pth |
| pooled/{small,large}/ # trained on the union of the five benchmarks |
| (same seven files) |
| ``` |
|
|
| `f01`–`f07` follow the paper's experiment registry (single-signal, pairwise, |
| and full hybrids). Per-benchmark models produce Table 1's per-benchmark |
| columns; `pooled/` models produce the *All benchmarks* column. |
|
|
| ## Usage |
|
|
| See the [training/eval scripts](https://github.com/YoEv/MEva) — |
| `training/hybrid/train_cnn.py` (architecture) and |
| `scripts/eval/eval_14_experiments.py` (evaluation protocol). |
|
|
| ## Citation |
|
|
| ```bibtex |
| @inproceedings{li2026meva, |
| title = {Do Music Generative Models Understand Musical Qualities? Automatic Music Evaluation with Model-Intrinsic Signals}, |
| author = {Li, Xiaosha and Liu, Chun and Wang, Ziyu}, |
| booktitle = {Proc. of the 27th Int. Society for Music Information Retrieval Conf. (ISMIR)}, |
| year = {2026} |
| } |
| ``` |
|
|
