Inside the mix
Explore how MVSep Mega 53 separates music into vocals, instruments and other musical parts. Compare the original with the sum of all 53 model outputs, then open any stem to listen.
Model release & 53 labels · Training and inference code · MSST framework paper · MVSep model description · Model license discussion
What is this model?
A stem is one part of a recording, such as a voice, drums or a piano. Mega 53 is a BS-RoFormer model: it looks at a time–frequency representation of the stereo recording and estimates a separate audio signal for each of 53 labels. Its shared network feeds 53 output heads. The model estimates these parts; it does not have access to the original studio tracks.
The code and model weights are reported as MIT licensed by their author. The MSST paper describes the broader training framework; it is not a detailed training report for this particular checkpoint. The public checkpoint includes its architecture and settings, but we could not verify its complete training dataset, training recipe, or whether a sum-to-mixture loss was used. A typical supervised training example gives the model a mixture and known target stems; it learns by reducing differences between each predicted stem and its target. The released architecture supports a waveform loss and a multi-resolution frequency loss for this kind of comparison. This does not establish which losses were used in the final training run, or that all 53 outputs were forced to add up to the original.
Why the reconstruction may sound wrong: the outputs overlap. For example, “vocal” can contain the same sound as “lead-vocal”; “drums” can overlap with “kick” and “snare.” Adding all 53 together can count sounds several times, and the model is not guaranteed to put every sound in exactly one output. The reconstruction below is the literal, unweighted sum of all 53 estimates, with a single volume reduction for playback if needed to prevent clipping. Individual stem previews may also be reduced in volume if their peaks exceed the playable range; the sum is calculated before these reductions.
| What you hear | What it means |
|---|---|
| Original | The same 20-second stereo excerpt sent to the model. |
| All 53 added together | A deliberately literal sum, not a guaranteed clean remix. The displayed playback gain prevents the MP3 from clipping; it does not change the measurements. |
| One stem | The model’s estimate for a label. A quiet output does not prove the instrument is absent; unwanted sounds can leak into any stem. |
Measurements: SNR compares the unadjusted sum to the original, including volume errors; higher means a closer waveform. SI-SDR also permits one overall volume adjustment, so it is less sensitive to loudness; higher is better. For both scores, −2 dB is better than −10 dB; negative values do not mean silence. Both are sensitive to timing and phase, and neither tells us whether an individual stem is clean or sounds good. These are not separation scores: we do not have the isolated original stems. Values use the 44.1 kHz signals before MP3 encoding. Model compute times are per 20-second excerpt on one NVIDIA GH200 with float16 autocasting; they exclude loading, resampling and file processing.
Loading measured summary…
Listen to the excerpts
Open a song’s stem list, then press Load & play beside a sound. The audio is fetched only when requested. Stems are ordered by their measured energy; energy is not an instrument detector.
Loading songs…
Beyond music: separating everyday sounds
A similar idea can isolate laughter, footsteps, birds, rain or other sounds from a scene. These systems differ in how you ask for a target and what kind of training data they need. None of the models below was tested in this music demo.
| Model / approach | How to ask | Why it is interesting |
|---|---|---|
| Universal Source Separation (USS) · paper | Choose a sound class or use a sound detector | Research system trained with weak AudioSet labels; supports a broad hierarchy of up to 527 sound classes. It is a useful starting point for class-based experiments. |
| AudioSep · paper | Describe the target in text, e.g. “people laughing” | One model can be asked for different sound events, instruments and speech. The code is MIT; check the exact model checkpoint and its dependencies for their own terms. |
| SAM Audio · paper | Text, a time span, or a visual cue | Designed for general audio, speech and music. Model access is gated and the SAM license has its own terms; it is not simply an MIT checkpoint. |
| Google sound-separation / MixIT · paper | Separate a scene into sources without naming them in advance | MixIT can learn from mixtures when isolated ground-truth sources are scarce. The resulting outputs are not automatically named sound classes. |
Could we train a model for sound-effect classes?
Yes. One practical design is a shared audio network plus either a head per class, or a single separator conditioned on a requested class or text. The first can produce many outputs at once; the second handles a flexible list of targets but may require another pass for each request. A sound-event detector can suggest which classes to try, but its mistakes can hide sounds.
- Build examples with known answers. Start with isolated, appropriately licensed sound clips, such as laughter, footsteps and rain. Mix them at varied volumes, positions and room acoustics. Keep each original clip as the target, and include examples where the requested sound is absent.
- Train and check the outputs. Compare each predicted sound with its known target using waveform and frequency losses. For mutually exclusive, exhaustive targets, add a loss comparing their sum with the input, or include an “other” output. Do not force overlapping labels such as “human sound” and “laughter” to sum independently: that would count the same audio twice.
- Handle real scenes. AudioSet-style labels often say what is present in a clip but do not provide isolated sources. Weak-label methods, teacher models and MixIT can help use these recordings, but their targets may be noisy or ambiguously grouped. Synthetic mixtures and real scenes have different acoustics, so test both.
- Evaluate beyond a single average. Keep recording sources separate between train and test. Test missing targets, overlapping sounds of the same class, quiet backgrounds, stereo placement and real-world scenes. Use reference-based scores when isolated ground truth exists, and listening tests plus human annotations when it does not.
A reconstruction loss is useful but not sufficient: a model could put the whole input in one output and silence in the others while still adding back perfectly. It also needs target, semantic or other supervision to learn which sound goes where. Class labels can overlap, rare classes need enough examples, and licenses must be checked separately for the audio, model code and weights.
Research references: USS, AudioSep, SAM Audio, MixIT, and AudioSet ontology. Source excerpts in this private Space come from a collection whose redistribution license is not established.