| --- |
| license: mit |
| tags: |
| - music |
| - music-generation |
| - symbolic-music |
| - midi |
| - orchestral-music |
| - transformer |
| - grpo |
| - reinforcement-learning |
| library_name: pytorch |
| --- |
| |
| # SymphonyGen: 3D Hierarchical Orchestral Generation with Controllable Harmony Skeleton |
|
|
| [](https://arxiv.org/abs/2604.25498) |
| [](https://github.com/symphonygen/symphonygen) |
| [](https://symphonygen.github.io) |
|
|
| SymphonyGen is a 3D hierarchical framework for contemporary cinematic orchestration, accepted at **ISMIR 2026**. It decomposes symphonic scores along the Bar, Track, and Event axes with a cascading decoder architecture, conditions generation on a beat-quantized multi-voice **harmony skeleton** ("short-score" conditioning), refines the model with **GRPO** using a cross-modal audio-perceptual reward (CLaMP 3 audio embeddings), and suppresses tonal clashes at inference time with **dissonance-averse sampling**. |
|
|
| - 📄 **Paper:** [SymphonyGen: 3D Hierarchical Orchestral Generation with Controllable Harmony Skeleton](https://arxiv.org/abs/2604.25498) (ISMIR 2026) |
| - 💻 **Code:** [github.com/symphonygen/symphonygen](https://github.com/symphonygen/symphonygen) |
| - 🌐 **Audio demo page:** [symphonygen.github.io](https://symphonygen.github.io) (listening-test excerpts for composition and re-orchestration, plus baselines) |
|
|
| ## Released checkpoints |
|
|
| All checkpoints are **packed**: each `.pt` bundles the model config together with the weights, and is loaded by `ModelBase.from_pretrained` in the codebase. |
|
|
| | File | Model | Role | |
| | --- | --- | --- | |
| | `stage_one_pretrained.pt` | 1D harmony skeleton model (12 layers, hidden 768, 87M) | Samples harmony skeletons | |
| | `stage_two_pretrained.pt` | 3D symphony model (33 layers, hidden 512, 124M, 2-stream cross-attention) | Pretrained on the SymphonyNet Dataset; starting point for GRPO / finetuning | |
| | `grpo_clamp_epoch_10.pt` | 3D symphony model ("reinforced") | GRPO with the pure CLaMP 3 audio reward (paper round-1 subjective test) | |
| | `grpo_clamp+track_epoch_6.pt` | 3D symphony model ("reinforced+track") | GRPO with the composite CLaMP 3 + track density reward (paper round-2 subjective test) | |
|
|
| Recommended dissonance-averse sampling settings: `(λ_hn, λ_nn) = (1, 10)`; use register decay (`--register_decay 1`) with `reinforced+track` and `--register_decay 0` with `reinforced`. |
|
|
| ## Usage |
|
|
| Generation is run through the [codebase](https://github.com/symphonygen/symphonygen) (see its README for full setup): |
|
|
| ```bash |
| git clone https://github.com/symphonygen/symphonygen |
| cd symphonygen && pip install -r requirements.txt |
| export PYTHONPATH=$(pwd) |
| |
| # Download the checkpoints into $ASSET_DIR (default: asset/) |
| hf download SymphonyGen/SymphonyGen --local-dir asset/ |
| |
| # Generate a harmony skeleton, then orchestrate it |
| python arch/harmo/generator.py asset/stage_one_pretrained.pt --batch_size 4 --save_dir harmony_out |
| python arch/symph/generator.py asset/grpo_clamp+track_epoch_6.pt harmony_out \ |
| --group_size 1 --save_dir songs --forbid_piano |
| |
| # Or re-orchestrate any MIDI by analyzing its harmony skeleton |
| python arch/symph/generator.py asset/grpo_clamp+track_epoch_6.pt your_song.mid \ |
| --analyze_harmo --save_dir songs |
| ``` |
|
|
| The harmony skeleton may also be **user-written** (e.g. a few chords played on a MIDI keyboard, exported as MIDI) — pass it as the conditioning MIDI without `--analyze_harmo`. |
|
|
| ## Training |
|
|
| - **Data:** the [SymphonyNet Dataset](https://symphonynet.github.io/) (728 classical + 45,632 contemporary MIDI files), 90/10 train/validation split. |
| - **Pretraining:** four NVIDIA H800 GPUs for one day, AdamW, lr 1e-4 with cosine annealing. |
| - **GRPO:** single GPU, lr 4e-5, group size 32 over 16 skeletons per epoch; reward = CLaMP 3 audio-embedding cosine similarity against the centroid of a small cinematic reference set (rendered with MuseScore 3), optionally + 0.2 × track-density term. |
|
|
| See the paper and the repository's `DIFF.md` for implementation details that go beyond the paper's description. |
|
|
| ## Evaluation |
|
|
| Objective metrics (CLaMP similarity, dissonance, track density, harmony precision/recall, melodic movement/ornament) and two rounds of subjective listening tests are reported in the paper; reproduction scripts live under `Results/` in the codebase. |
|
|
| > **Reproducibility disclaimer:** the objective metrics depend on the exact implementation of the harmony analysis and of the harmony-skeleton filters (see `DIFF.md` in the code repository). As a result, you might see reproduced objective evaluation numbers that differ from the paper's tables. The trends, however, should be the same. |
|
|
| ## Intended use and limitations |
|
|
| SymphonyGen is intended as a collaborative aid for composers, not as an autonomous replacement. It is oriented toward contemporary cinematic scoring through its data, reward, and hyperparameters. Generative outputs may inadvertently resemble existing works; users should not assume all outputs are fully original without verification. |
|
|
| ## Citation |
|
|
| ```bibtex |
| @inproceedings{symphonygen2026, |
| title = {SymphonyGen: 3D Hierarchical Orchestral Generation with Controllable Harmony Skeleton}, |
| author = {He, Xuzheng and Nan, Nan and Wang, Zhilin and Kang, Ziyue and Mo, Zhuoru and Li, Ao and Pan, Yu and Li, Xiaobing and Yu, Feng and Guan, Xiaohong}, |
| booktitle = {Proceedings of the 27th International Society for Music Information Retrieval Conference (ISMIR)}, |
| year = {2026} |
| } |
| ``` |
|
|
