A Tutorial on Generative Models for Music Audio

𝗡𝗲𝘄: 𝗮 𝟭𝟯𝟮-𝗽𝗮𝗴𝗲 𝘁𝘂𝘁𝗼𝗿𝗶𝗮𝗹 𝗼𝗻 𝗴𝗲𝗻𝗲𝗿𝗮𝘁𝗶𝘃𝗲 𝗺𝗼𝗱𝗲𝗹𝘀 𝗳𝗼𝗿 𝗺𝘂𝘀𝗶𝗰 𝗮𝘂𝗱𝗶𝗼 🎶
A few years ago, generating a full song with vocals from a text prompt was unthinkable. In our review, we counted 25 song generation systems released in 2025 and 2026 alone. Prof. Mateusz Modrzejewski, Joanna frota Kurkowska and I wrote this tutorial to explain how they work.

It explains codecs and autoencoders, token models, diffusion and flow matching, faster sampling and preference optimization in one notation, with worked examples from published systems. It then looks at what instrumental generation, full songs with vocals, editing and live performance each need from a model, and how to evaluate and release these models responsibly.

It's written for students, researchers and engineers getting into the field, and for musicians who want to know what's behind the tools they're starting to use.

Fast progress doesn't mean the field is done. Going through all these systems, we kept running into the same unsolved problems. 𝗪𝗵𝗮𝘁'𝘀 𝘀𝘁𝗶𝗹𝗹 𝗼𝗽𝗲𝗻:

• Long-range structure. Models can now produce full-length songs, but no widely adopted protocol evaluates form, motif development and tension over several minutes.
• Evaluation. Some systems use the same aesthetics predictor as a training reward and as the evaluation metric. A higher score can then just mean the model learned to please that judge.
• Data and memorization. Most training data is undisclosed. Lyric filters miss phonetic memorization, and detectors for generated audio get worse after mixing or broadcast.
• Coverage. Song models focus mostly on English and Mandarin pop. Most of the world's musical traditions are barely in the training data.
• Interpretability. We know very little about how these models represent music internally, so control and safety interventions are still found by trial and error.

Systems also differ so much in data, scale and evaluation that comparing them side by side says little about where the reported gains come from. Part of the tutorial is about learning to interpret those claims.

Whether you're new to music generation or you want to fill in the gaps in what you already know, the full tutorial goes much deeper than this post can.