Document paper arXiv:2604.26841 (abstract, authors, BibTeX)

README.md

@@ -5,18 +5,45 @@ tags:
   - diffusion-language-model
   - associative-memory
   - memorization
+  - discrete-diffusion
   - lm1b
+  - arxiv:2604.26841
 datasets:
   - lm1b
 ---
 
-# LLDMs — Associative Memory
+# Language Diffusion Models are Associative Memories Capable of Retrieving Unseen Data
 
-Training checkpoints for a study of **associative memory / memorization in large language diffusion
-models**. The release is a two-axis sweep: **model size** × **training-set fraction**.
+Model checkpoints for the paper:
 
-Every checkpoint is a discrete diffusion language model (DUO / masked-diffusion style, `ddit`
-backbone) trained on [LM1B](https://huggingface.co/datasets/lm1b) at sequence length 1024.
+> **Language Diffusion Models are Associative Memories Capable of Retrieving Unseen Data**
+> Bao Pham, Mohammed J. Zaki, Luca Ambrogioni, Dmitry Krotov, Matteo Negri
+> arXiv:2604.26841 · [paper](https://arxiv.org/abs/2604.26841)
+
+## Abstract
+
+When do language diffusion models memorize their training data, and how to quantitatively assess
+their true generative regime? We address these questions by showing that Uniform-based Discrete
+Diffusion Models (UDDMs) fundamentally behave as Associative Memories (AMs) with emergent creative
+capabilities. The core idea of an AM is to reliably recover stored data points as memories by
+establishing distinct basins of attraction around them. Historically, models like Hopfield networks
+use an explicit energy function to guarantee these stable attractors. We broaden this perspective by
+leveraging the observation that energy is not strictly necessary, as basins of attraction can also
+be formed via conditional likelihood maximization. By evaluating token recovery of training and test
+examples, we identify in UDDMs a sharp memorization-to-generalization transition governed by the
+size of the training dataset: as it increases, basins around training examples shrink and basins
+around unseen test examples expand, until both later converge to the same level. Crucially, we can
+detect this transition using only the conditional entropy of predicted token sequences: memorization
+is characterized by vanishing conditional entropy, while in the generalization regime the
+conditional entropy of most tokens remains finite. Thus, conditional entropy offers a practical
+probe for the memorization-to-generalization transition in deployed models.
+
+## What is released here
+
+The **training-set size is the axis the paper's transition is measured along**, so this repo is a
+full sweep over it: three model sizes × 54 nested subsets of
+[LM1B](https://huggingface.co/datasets/lm1b), from 0.01% of the corpus to 100%, every checkpoint
+trained for exactly 1,000,000 steps. 162 checkpoints, ~473 GB.
 
 ## Repository layout
 
@@ -129,4 +156,14 @@ python main.py model=medium data.subset=0.25
 
 ## Citation
 
-Please cite the accompanying paper if you use these checkpoints. _(Citation to be added.)_
+```bibtex
+@misc{pham2026languagediffusionmodelsassociative,
+      title={Language Diffusion Models are Associative Memories Capable of Retrieving Unseen Data},
+      author={Bao Pham and Mohammed J. Zaki and Luca Ambrogioni and Dmitry Krotov and Matteo Negri},
+      year={2026},
+      eprint={2604.26841},
+      archivePrefix={arXiv},
+      primaryClass={cs.LG},
+      url={https://arxiv.org/abs/2604.26841},
+}
+```