escapist413/LLaCO-8B

LLaCO: Continuous Diffusion for Large Language Models with Competition-Aware Semantic Scheduling

Abstract

Continuous diffusion language models suffer from a fundamental geometric misalignment between continuous trajectories and discrete vocabulary identities. This compresses identity decay into a narrow terminal window, forcing standard diffusion to misallocate optimization budgets. To resolve this, we establish the Competition Threshold Theory to model directional token competition, and propose Semantic Retention Scheduling for logarithmic rank decay. Based on these principles, we propose LLaCO, a framework equipped with an embedding directional adapter, timestep-conditioned modulation, and mask-biased spherical noising. Extensive evaluations demonstrate that LLaCO-8B comprehensively outperforms prior continuous baselines and rivals discrete models.