Is the model trainable?
Lora, Lokrs, etc
Important, important thing for making the model relevant on actual use.
Thankss
Until they release the Encoder, it's not.
yes: https://docs.simpletuner.io/quickstart/MINIMAX_MUSIC/
LyCORIS and PEFT LoRA trainable via SimpleTuner, with lyrics + caption validation libraries too. though that's not easily configured via the webui just yet (the validation library), the example prompts and captions from their website are included as a built in validation prompt library if you're doing eg. a broad finetune and want to validate a number of separate concepts.
it works with distillation too, so, the next project i'm on with MM Music is to throw AnyFlow at it and bring it down to a decent 4-step generation pipeline in the flow transformer.
what we're limited by is the lack of RVQ encoder's semantic audio tokens from raw audio. i think we could reverse engineer one though if so inclined, i'm not sure whether i'd be going against the license to do so.
Nice.. Its something.
Would it make any sense to opensource a model to then not share a fundamental piece? A model that is not trainable/conditionable is pretty much a toy.
The RVQ encoder is required for properly fine-tuning the model. Without it, you can maybe adjust some tone and pitches, though you would be doing so 100% without proper conditioning. It is like training an image model but instead of captions, you are just typing pure nonsense random text in the encoder for the conditioning. I mean sure, you CAN train it that way. But you will not get good results. The RVQ encoder is absolutely needed.
that's not true, look at how the pipeline actually works. the depth decoder gives the model its tokens from text. the encoder just does it from raw audio.
the inference path mirrors training. you're not required to supply RVQ encoder output (audio references) at inference time. why would you think it'd be mandatory at training?
our training target for the flow matching objective is the VAE audio latents, not the RVQ encoder output.
we wouldn't even be able to run the model for inference if the encoder were required for training.
@bghira You are attempting to train on non aligned tokens. During inference, they are aligned because they were trained to be that way, with the RVQ tokenizer, something you will be breaking with your "training" method.. You are just swapping out a non aligned target and breaking the alignment and saying you trained it. It is like training an LLM without a tokenizer. You have your output tokens, but you are feeding complete garbage on the input.
Think of it this way. RVQ hidden state token are like a MIDI track. A super high detail, frame by frame, MIDI track that contains lyrics for each time step, drums, the beat, when the guitar strums are, to the millisecond, when the chorus starts, everything. It is play by play, perfectly time aligned sheet music and detailed composition for each frame of the song. Without the RVQ tokenizer, you are generating this precise midi track, and throwing a completely different song at it where none of the timing aligns , the lyrics are completely different, this one does not even have drums, the "E note piano key" is now a "F note flute". It is in a different key, pitch, everything is 100% misaligned. And you are teaching the model,"make this sound like that" You are just completely breaking down the model doing this. 100%. And maybe, just maybe, before the model is completely broken, it may sound a little bit like the 100% misaligned song you fed into it. But that does not mean you trained it, you just broke it pretty.
i don't get where you're getting any of that from. sorry, it's just not the case.
- RVQ encoder gives target-audio-derived discrete tokens.
- RVQ depth decoder gives text-derived residual-code hidden states.
- the transformer consumes condition-encoder outputs, not raw RVQ encoder states.
- the model supports text-to-audio inference, which is conditioning generated from text via the feedback loop with the LM and depth decoder, not from the RVQ encoder
- text-to-audio LoRAs train against the same conditioning used at inference
- RVQ encoder is ONLY needed for target-token supervision, reference-audio conditioning
we're not training on "garbage text", but sure, if you have really bad captions and lyrics inputs that don't align at all with the latents the audio latent encoder creates, you'd get bad results.
have you actually tried training it, or just repeating yourself hoping that people believe it?
No point arguing with arrogance.
agreed
I don't understand enough of the tech details to get those intricacies but Ostris and Bghira arguing on my question thread is an honor I guess.. 😁
agreed
@bghira you've implemented training modules for it on SimpleTuner, do you have any examples we can test? Have you produced any LoRA/LyCORIS/finetune with your method that we can try? I can provide a small dataset with some relatively obscure genres/styles, and evaluate the results if you want.
That's the easiest way to find out whether it works or not without the RVQ encoder.
yes, but it was using internal datasets and i'm contractually barred from sharing those results. i've put together a public dataset after identifying that Jamaican patois / reggaeton is really badly generated by the model (its' actually kind of funny how bad it is), you can check the data i'm using under RareConcepts/suno-reggae-test-dataset
Isn't their whole documentation about the LMs (both Global and Local) giving structure that the DiT then turns into decodable latents?
Think ACE-Step 1.5, i actually wrote Side-Step (which to be fair, compared to SimpleTuner and ai-toolkit, not even God himself knows), they do use an LM as well for "planning" and pre-conditioning their generations. On ACE-Step 1.5 it is not mandatory to use, and text encoding is provided by another Qwen model separate from the causal LM providing their so-called audio codes, but Minimax Music 3 apparently uses its hidden states which are already text-conditioned, which for training is a gut punch.
You said "the inference path mirrors training... we wouldn't even be able to run the model for inference if the encoder were required for training."
But at inference, condition and latent are aligned automatically, no? The DiT generates the latent from the condition, so the alignment is just given, there's no external target it has to align to.
At training there is an external target, and something has to supply that alignment between the condition and the target. MiniMax supplied it with their RVQ tokenizer/encoder, which is basically what we're asking for here.
I don't think @ostris or anyone else is saying that the RVQ output was the target during training. It's what makes the conditioning actually mean the target in the first place. /shrug
All that said, I'll gladly be educated by anyone on this thread. if misaligned rollout conditioning really is sufficient for adapter training and/or If the LMs are an optional path as it is on ACE-Step 1.5 during inference, it is a win for everyone training adapters, arguably even people just generating without them.
In any case @bghira i was personally also in the side-quest of reverse engineering the tokenizer, but for a solo dev this is high risk, no reward, not to mention im broke. If you guys ever decide to go that path, i will do everything on my power to help out, i figured what is the most time-consuming is just having a sheer volume of generations to save intermediary artifacts; that alone (given how slow a generation is to, well, generate) on consumer hardware at batch 1 already talked me out of doing it solo.
so does this work or do i have to keep banging my head with the RVQ encoder. i guess we need it for real reference, but training? and i agree, imma give it 9 hours to finish this run and then done if it doesn't work lol