| # PC-SHO-DLM |
|
|
| **Predictive-Coding Diffusion Language Models with Precision-Conditioned Second-Order Settling** |
|
|
| ## Overview |
|
|
| PC-SHO-DLM is a research architecture that unifies three individually mature research threads: |
|
|
| 1. **Masked discrete diffusion** for iterative text denoising |
| 2. **Predictive coding** for local error-driven inference and learning |
| 3. **Second-order dynamical systems** for accelerated latent settling |
|
|
| The core insight: diffusion denoising and predictive coding are both iterative residual-correction procedures, making diffusion a structurally natural host for predictive-coding-based language modeling. |
|
|
| ## Key Contributions |
|
|
| - **Bidirectional predictive transformer** with both bottom-up and top-down pathways |
| - **Precision-conditioned second-order settling** (damped inertial dynamics with learned uncertainty) |
| - **State continuation across diffusion steps** (warm-start from previous denoising step) |
| - **Tokenwise adaptive settling** (selective compute allocation to uncertain tokens) |
| - **Globally backpropagation-free training** via local prediction-error-driven updates |
|
|
| ## Theoretical Results |
|
|
| | Theorem | Result | |
| |---------|--------| |
| | **A: Reduced-objective exactness** | Local post-settling updates = exact gradients via envelope theorem | |
| | **B: Accelerated settling** | O(sqrt(kappa)) vs O(kappa) microsteps over first-order PC | |
| | **C: Warm-start tracking** | Settling cost depends on inter-step drift, not full re-solve | |
| | **D: Precision optimality** | GLS-optimal under heteroscedastic prediction errors | |
|
|
| ## Project Structure |
|
|
| ``` |
| pc-sho-dlm/ |
| ├── paper/ |
| │ ├── main.tex # Full LaTeX manuscript |
| │ └── references.bib # Bibliography |
| ├── src/ |
| │ ├── model.py # Core PC-SHO-DLM architecture |
| │ └── train.py # Training loop (local + backprop modes) |
| ├── experiments/ |
| │ ├── ablation_runner.py # Ablation study runner |
| │ └── settling_analysis.py # Settling dynamics analysis |
| └── README.md |
| ``` |
|
|
| ## Quick Start |
|
|
| ### Sanity Check |
| ```bash |
| cd src |
| python model.py |
| ``` |
|
|
| ### Train (Local PC mode) |
| ```bash |
| python src/train.py --mode local --device cpu --max_steps 5000 |
| ``` |
|
|
| ### Train (Backprop baseline) |
| ```bash |
| python src/train.py --mode backprop --device cpu --max_steps 5000 |
| ``` |
|
|
| ### Run Ablations |
| ```bash |
| python experiments/ablation_runner.py --device cpu |
| ``` |
|
|
| ### Analyze Settling Dynamics |
| ```bash |
| python experiments/settling_analysis.py --device cpu |
| ``` |
|
|
| ## Experimental Program |
|
|
| ### Stage 1: Proof of Mechanism (20M-100M params) |
| - Character/subword corpora (text8, enwik8, WikiText-103) |
| - Establish second-order PC is more compute-efficient than first-order PC |
|
|
| ### Stage 2: Medium Scale (100M-1B params) |
| - Open web corpus subset |
| - Compare against backprop-trained masked diffusion |
|
|
| ### Stage 3: Structured Generation |
| - Infilling, editing, code completion |
| - Test selective settling and state continuation |
|
|
| ## Falsification Criteria |
|
|
| This proposal should be judged a failure if: |
| 1. Second-order settling does not reduce required microsteps |
| 2. Local learning creates an irrecoverable quality gap vs backprop |
| 3. Precision heads fail to calibrate |
| 4. Dynamics frequently enter limit cycles |
| 5. State continuation does not materially reduce compute |
|
|
| ## Citation |
|
|
| ```bibtex |
| @article{pcshodlm2026, |
| title={PC-SHO-DLM: Predictive-Coding Diffusion Language Models |
| with Precision-Conditioned Second-Order Settling}, |
| year={2026} |
| } |
| ``` |
|
|
