Post
33
Drift-Constrained Optimization: Only Direction Matters in Fine-Tuning Instruct Models
Standard SFT usually ask:
How can we SFT an instruct model without behavioral drift?
We reverse the question:
Given a fixed drift budget, how should we SFT?
This predicts that changing *where* the model can move—not the supervision—can reverse fine-tuning failure.
On Qwen3-8B and Qwen3-14B, it does.
code: https://github.com/CONE-MT/DCO/tree/main/dco
models: https://huggingface.co/collections/LLaMAX/dco
Standard SFT usually ask:
How can we SFT an instruct model without behavioral drift?
We reverse the question:
Given a fixed drift budget, how should we SFT?
This predicts that changing *where* the model can move—not the supervision—can reverse fine-tuning failure.
On Qwen3-8B and Qwen3-14B, it does.
code: https://github.com/CONE-MT/DCO/tree/main/dco
models: https://huggingface.co/collections/LLaMAX/dco