Purpose?
#2
by sorryhyun - opened
Hi, I'm quite interested in your work. So what would be the final purpose of this work? I expect enabling fp16 training for both accuracy and stability if being ambitious. Thanks for the nice work anw
It try to provide a MOSS-image-base that is easier to finetune, and some community member already report that with healed version the finetune is way more stable and effective.
It also allow FP16 to finetune/inference the model without special handling as well.
Basically it is trained to fix the bad numerical behavior in MOSS-image models, while try to align with the original output.
I will also use similar infra to train KohakuMOSS in future I guess