| --- |
| license: mit |
| --- |
| # Audio DiT |
|
|
| Tag-conditioned music generation focused on instrumental game and anime songs. Choose 1-8 tags (recommend to start with 1). Generates 95s songs. |
| Set a seed to reproduce a generation, or leave it at `-1` for a random seed. |
|
|
| ## Run it |
|
|
| Needs [uv](https://docs.astral.sh/uv/). CUDA, MPS or CPU. |
|
|
| ```bash |
| hf download Localsong/Localsong --local-dir Localsong |
| uv run webui.py # http://127.0.0.1:7860 |
| ``` |
|
|
| ## Architecture |
|
|
| ``` |
| tags ──► AudioDiT ──► 256×1024 latents ──► SAME-L decoder ──► 95 s of stereo audio |
| 0.84B params |
| ``` |
|
|
| **Latents.** [SAME-L](https://huggingface.co/stabilityai/SAME-L) autoencoder by Stability AI. |
|
|
| **Denoiser.** `audio_dit.py` is a 1-D latent transformer DiT. |
| RMSNorm and QK-RMSNorm, RoPE, SwiGLU |
| Timestep conditioning is adaLN-single (PixArt) |
| offset. Tags condition via cross-attention. |
|
|
| **Sparse-dense fusion** ([SPRINT](https://arxiv.org/abs/2510.21986)). The blocks |
| are split into a dense 2-block encoder, a 20-block middle stack, and a dense |
| 2-block decoder. The encoder output is concatenated with the middle output and |
| projected back down before the decoder: |
|
|
| ``` |
| frames ──► encoder ──┬─────────────────────────► f ──┐ |
| └─► middle (20 blocks) ──► g ──► linear([f;g]) ──► decoder ──► velocity |
| ``` |
|
|
| Training ran the middle stack on a random 25 % of frames and sometimes dropped |
| it entirely, which makes the shallow encoder→fusion→decoder route a usable weak |
| model on its own. Inference runs the middle stack on every frame |
| (`AudioDiT.forward`) and keeps the shallow route as the guidance branch |
| (`AudioDiT.shallow`). |
|
|
| **Sampling.** Rectified flow: v-prediction, 50 steps Euler shift=2 |
| Guidance alternates between normal CFG and the shallow-path PDG. |
|
|
| ## License |
|
|
| MIT, see [LICENSE](LICENSE). |
|
|
| `same_l_decoder.py` is adapted from Stability AI's |
| [stable-audio-3](https://github.com/Stability-AI/stable-audio-3) (MIT). The |
| SAME-L weights it loads are released by Stability AI under the Stable Audio |
| Community License; commercial use of them is subject to |
| <https://stability.ai/license>. |
|
|
| The model also builds on SPRINT ([2510.21986](https://arxiv.org/abs/2510.21986)), |
| PixArt-α ([2310.00426](https://arxiv.org/abs/2310.00426)), |
| DiT ([2212.09748](https://arxiv.org/abs/2212.09748)) and |
| rectified flow ([2209.03003](https://arxiv.org/abs/2209.03003)). |
