nadjushenka12/law

Loopholing All the Way: Language Modeling with pure Self-Conditioning

By Adrien Deschenaux*, Nadezhda Ilieva*, Justin Deschenaux*, Manuel Madeira, Dorina Thanou, Aurelien Lucchi, Antonio Orvieto, Caglar Gulcehre and Pascal Frossard.

EPFL · University of Basel · Microsoft AI · ELLIS Institute Tübingen, MPI for Intelligent Systems, Tübingen AI Center

*Equal contribution.

This repository contains the pretrained checkpoints for Loopholing All the Way: Language Modeling with pure Self-Conditioning (LAW). For the training/sampling code, and reproduction scripts, see the code repository: ilievanadezhda/law.

Checkpoints

LAW, LDDM, and the baselines we compare against (AR, MDLM, Duo, FLM, 𝕊-FLM), trained on OpenWebText (400k baseline steps, GPT-2 tokenizer) and TinyGSM (250k baseline steps, SmolLM-135M tokenizer). LAW and LDDM are trained with the same FLOPs as the baselines, so they train for fewer steps. Each folder has a README describing its runs and how the step count was matched. The TinyGSM baselines are in jdeschena/s-flm.

owt/law/<run>/<epoch>-<step>.ckpt
owt/lddm/<run>/<epoch>-<step>.ckpt
owt/baselines/{ar,duo,flm,mdlm,sfm-std,sfm-ngpt}-400000.ckpt

tinygsm/pretrain/law/{01_carry_x_loss,
                      02_temperature_x_mode_embedding,
                      03_pass_count_x_probability,
                      04_backprop_through_loop}/<run>/<epoch>-<step>.ckpt
tinygsm/pretrain/lddm/<run>/<epoch>-<step>.ckpt
tinygsm/finetune/duo_original/duo-250000.ckpt
tinygsm/finetune/law/<run>/<epoch>-<step>.ckpt

hf download nadjushenka12/law owt/baselines/duo-400000.ckpt --local-dir ./checkpoints

Each file is a full PyTorch Lightning checkpoint (weights, EMA weights, optimizer state and training config). Loading and sampling will be handled by the code repository.

Citation

@misc{deschenaux2026loopholingwaylanguagemodeling,
      title={Loopholing All the Way: Language Modeling with pure Self-Conditioning}, 
      author={Adrien Deschenaux and Nadezhda Ilieva and Justin Deschenaux and Manuel Madeira and Dorina Thanou and Aurélien Lucchi and Antonio Orvieto and Caglar Gulcehre and Pascal Frossard},
      year={2026},
      eprint={2610.30068},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2610.30068}, 
}