nthngdy/cplm-360M

Copy-Paste Language Models — 360M

360M-parameter Copy-Paste Language Models (CPLMs) from the paper Copy-Paste Language Models (Nathan Godey & Yoav Artzi, Cornell University). [Paper: TODO arXiv link]

A CPLM adds a lightweight Copy-Paste (CP) head next to the standard LM head. The CP head attends over past positions and copies the token found there. It is triggered by a dummy <copy> token in the LM head and can fall back to the LM head through a learnable sink. The final next-token distribution is a mixture of the two heads, trained end-to-end with the standard next-token loss only.

Vanilla baselines trained with the exact same recipe are available at nthngdy/cplm-baselines-360M. Larger models: nthngdy/cplm-1.7B.

Checkpoints

Each model is stored on its own branch. Three backbones are available:

Backbone	Description
full	Full attention in all 32 layers
hybrid	3 sliding-window (w=256) layers for every full-attention layer
sw	Sliding-window attention (w=256) in all layers

These are the final checkpoints (20B tokens, step 19,000, after the LR cooldown).

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "nthngdy/cplm-360M"
tok = AutoTokenizer.from_pretrained(repo, revision="full")
model = AutoModelForCausalLM.from_pretrained(repo, revision="full", trust_remote_code=True)

ids = tok("The capital of France is", return_tensors="pt").input_ids
print(tok.decode(model.generate(ids, max_new_tokens=32)[0]))

model.config.inference_mode controls decoding: "natural" (default, mixture of both heads), "no_copy" (LM head only) or "force_copy" (CP head only).

Training

• Architecture: SmolLM3-style backbone with 32 layers, hidden size 960, 15 query / 5 key-value heads and tied embeddings, plus the CP head (+1.8M parameters). The vocabulary has 49,152 tokens plus <copy>.
• Data: 20B tokens of FineWeb-Edu, 2,048-token sequences.
• Recipe: AdamW (betas 0.9/0.95, weight decay 0.01), peak LR 3e-3, WSD schedule (500 warmup steps, 1,900 cooldown steps, 19,000 steps in total) and batch size 512.

Results

Final validation perplexity on held-out FineWeb-Edu:

Backbone	Vanilla	CPLM
Full	13.50	13.24
Hybrid	13.52	13.26
SW	13.91	13.41

Citation

@article{godey2026cplm,
  title   = {Copy-Paste Language Models},
  author  = {Godey, Nathan and Artzi, Yoav},
  journal = {TODO},
  year    = {2026}
}