| # CANDI: Hybrid Discrete-Continuous Diffusion Models |
|
|
| <div align="center"> |
| <strong><a href="https://patrickpynadath1.github.io/candi-lander/#">🌐 Project Page</a></strong> |
| </div> |
|
|
| ## Overview |
|
|
| CANDI (Continuous and Discrete Diffusion) is a novel hybrid diffusion model that combines continuous and discrete noise processes for high-quality text generation. This approach bridges the gap between continuous diffusion models used in computer vision and discrete token-based language modeling, offering the best of both worlds. |
|
|
|  |
|
|
| ## Method |
|
|
| CANDI uses a hybrid kernel to coordinate both discrete and continuous corruption explicitly throughout training: |
|
|
|  |
|
|
| This code base is built on the DUO codebase, which is available at this link: https://github.com/s-sahoo/duo?tab=readme-ov-file. |
|
|
|
|
| ## Installation |
|
|
| 1. Clone the repository: |
| ```bash |
| git clone https://github.com/patrickpynadath1/candi.git |
| cd candi |
| ``` |
|
|
| 2. Install dependencies: |
| ```bash |
| pip install -r requirements.txt |
| ``` |
|
|
| 3. (Optional) Install Flash Attention for faster training: |
| ```bash |
| pip install flash-attn --no-build-isolation |
| ``` |
|
|
| 4. Download the OWT data |
| ```bash |
| bash manual_download.sh |
| ``` |
|
|
| ## Experiments |
|
|
| This codebase includes the code for running experiments on Text8 and OWT. We will integrate the QM9 experiments later. In general, we re-use the same experimental methodology and codebase from https://github.com/kuleshov-group/discrete-diffusion-guidance. |
|
|
| ### Training |
|
|
| We include scripts for training models in scripts/slurm_scripts. |
| |
| |
| ### Frontier Analysis |
| |
| Run temperature sweeps for frontier analysis using the following scripts: |
| ```bash |
| # OpenWebText sweeps |
| bash scripts/gen_ppl_owt_candi_sweep.sh |
| |
| # Text8 sweeps |
| bash scripts/gen_text8_candi_sweep.sh |
| ``` |
| |
| ## Configuration |
| |
| The project uses Hydra for configuration management. Key configuration files: |
| |
| - **Algorithm configs**: `configs/algo/` - Different diffusion algorithms (CANDI, MDLM, SEDD, etc.) |
| - **Data configs**: `configs/data/` - Dataset configurations |
| - **Model configs**: `configs/model/` - Model architecture settings |
| - **Base config**: `configs/config.yaml` - Main configuration file |
| |
| |
| |
| ## Key Components |
| |
| - **[`algo.py`](algo.py)**: Core algorithm implementations (CANDI, MDLM, DUO, etc.) |
| - **[`main.py`](main.py)**: Main training and evaluation script |
| - **[`dataloader.py`](dataloader.py)**: Data loading and preprocessing utilities |
| - **[`models/`](models/)**: Model architectures (DiT) |
| - **[`metrics.py`](metrics.py)**: Evaluation metrics and utilities |
| - **[`trainer_base.py`](trainer_base.py)**: Base trainer class with common functionality |
| |
| ## Citation |
| |
| If you use this code in your research, please cite: |
| |
| ```bibtex |
| @article{pynadath2025candi, |
| title={CANDI: Hybrid Discrete-Continuous Diffusion Models}, |
| author={Patrick Pynadath, Jiaxin Shi, and Ruqi Zhang}, |
| journal={arXiv preprint}, |
| year={2025} |
| } |
| ``` |
| |
| |
| |
| For more details, visit our [project page](https://patrickpynadath1.github.io/candi-lander/#) or check out the [paper](#). |
| |
