nthngdy/cplm-baselines-360M

Vanilla 360M baselines for Copy-Paste Language Models

360M-parameter vanilla language models used as baselines in the paper Copy-Paste Language Models (Nathan Godey & Yoav Artzi, Cornell University). [Paper: TODO arXiv link]

The Copy-Paste counterparts (CPLMs), trained with the exact same recipe, are available at nthngdy/cplm-360M.

Checkpoints

Each model is stored on its own branch. Three backbones are available:

Backbone	Description
full	Full attention in all 32 layers
hybrid	3 sliding-window (w=256) layers for every full-attention layer
sw	Sliding-window attention (w=256) in all layers

These are the final checkpoints (20B tokens, step 19,000, after the LR cooldown).

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "nthngdy/cplm-baselines-360M"
tok = AutoTokenizer.from_pretrained(repo, revision="full")
model = AutoModelForCausalLM.from_pretrained(repo, revision="full")

ids = tok("The capital of France is", return_tensors="pt").input_ids
print(tok.decode(model.generate(ids, max_new_tokens=32)[0]))

Training

• Architecture: SmolLM3 with 32 layers, hidden size 960, 15 query / 5 key-value heads, tied embeddings and a 49,152-token vocabulary.
• Data: 20B tokens of FineWeb-Edu, 2,048-token sequences.
• Recipe: AdamW (betas 0.9/0.95, weight decay 0.01), peak LR 3e-3, WSD schedule (500 warmup steps, 1,900 cooldown steps, 19,000 steps in total) and batch size 512.

Citation

@article{godey2026cplm,
  title   = {Copy-Paste Language Models},
  author  = {Godey, Nathan and Artzi, Yoav},
  journal = {TODO},
  year    = {2026}
}