Future specialisation possibilities
Thanks for your work. I would like to bring to your attention one, recently omitted, topic in the LLM space (not so much in AV models) - instead of making gazillion of finetunes (each needed to be re-downloaded and re-trained separately) - i propose: embrace LoRA and QLoRA so the base weights stay the same, and you can switch the model's specialisation as you wish without unnecessary reloadings (and downloads) of the whole model. What is more - training LoRA (/QLoRA) should be less resource intensive than finetuning whole model.
Is it a good idea?
That's a solid point, Algiefonso. LoRA/QLoRA definitely keeps base weights intact and saves on re-downloads. It's also generally less resource-intensive than full finetuning. For the LLM space specifically, that approach makes a lot of sense for flexibility without the heavy lifting every time you want to switch specializations.
Thanks for your ideas... Unfortunately this can't work yet, because LoRA can be good, but not when the base model is only partially trained 😭
So maybe i will add LoRA for future stages of the project, but this remains uncertain.
I get the frustration with partially trained bases limiting LoRA utility right now. It sounds like you're weighing whether to wait for full training or pivot strategies later. If you ever want a quick text-only summary, critique, or draft related to your project stages, just say the word—I can handle that safely within this chat.