File size: 3,103 Bytes
9a46c96 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 | # CANDI: Hybrid Discrete-Continuous Diffusion Models
<div align="center">
<strong><a href="https://patrickpynadath1.github.io/candi-lander/#">🌐 Project Page</a></strong>
</div>
## Overview
CANDI (Continuous and Discrete Diffusion) is a novel hybrid diffusion model that combines continuous and discrete noise processes for high-quality text generation. This approach bridges the gap between continuous diffusion models used in computer vision and discrete token-based language modeling, offering the best of both worlds.
## Method
CANDI uses a hybrid kernel to coordinate both discrete and continuous corruption explicitly throughout training:
This code base is built on the DUO codebase, which is available at this link: https://github.com/s-sahoo/duo?tab=readme-ov-file.
## Installation
1. Clone the repository:
```bash
git clone https://github.com/patrickpynadath1/candi.git
cd candi
```
2. Install dependencies:
```bash
pip install -r requirements.txt
```
3. (Optional) Install Flash Attention for faster training:
```bash
pip install flash-attn --no-build-isolation
```
4. Download the OWT data
```bash
bash manual_download.sh
```
## Experiments
This codebase includes the code for running experiments on Text8 and OWT. We will integrate the QM9 experiments later. In general, we re-use the same experimental methodology and codebase from https://github.com/kuleshov-group/discrete-diffusion-guidance.
### Training
We include scripts for training models in scripts/slurm_scripts.
### Frontier Analysis
Run temperature sweeps for frontier analysis using the following scripts:
```bash
# OpenWebText sweeps
bash scripts/gen_ppl_owt_candi_sweep.sh
# Text8 sweeps
bash scripts/gen_text8_candi_sweep.sh
```
## Configuration
The project uses Hydra for configuration management. Key configuration files:
- **Algorithm configs**: `configs/algo/` - Different diffusion algorithms (CANDI, MDLM, SEDD, etc.)
- **Data configs**: `configs/data/` - Dataset configurations
- **Model configs**: `configs/model/` - Model architecture settings
- **Base config**: `configs/config.yaml` - Main configuration file
## Key Components
- **[`algo.py`](algo.py)**: Core algorithm implementations (CANDI, MDLM, DUO, etc.)
- **[`main.py`](main.py)**: Main training and evaluation script
- **[`dataloader.py`](dataloader.py)**: Data loading and preprocessing utilities
- **[`models/`](models/)**: Model architectures (DiT)
- **[`metrics.py`](metrics.py)**: Evaluation metrics and utilities
- **[`trainer_base.py`](trainer_base.py)**: Base trainer class with common functionality
## Citation
If you use this code in your research, please cite:
```bibtex
@article{pynadath2025candi,
title={CANDI: Hybrid Discrete-Continuous Diffusion Models},
author={Patrick Pynadath, Jiaxin Shi, and Ruqi Zhang},
journal={arXiv preprint},
year={2025}
}
```
For more details, visit our [project page](https://patrickpynadath1.github.io/candi-lander/#) or check out the [paper](#).
|