nthngdy/cplm-baselines-1.7B

Vanilla 1.7B baselines for Copy-Paste Language Models

1.7B-parameter vanilla language models used as baselines in the paper Copy-Paste Language Models (Nathan Godey & Yoav Artzi, Cornell University). [Paper: TODO arXiv link]

The Copy-Paste counterparts (CPLMs), trained with the exact same recipe, are available at nthngdy/cplm-1.7B.

Checkpoints

Each model is stored on its own branch. Three backbones are available:

Backbone	Description
full	Full attention in all 24 layers
hybrid	3 sliding-window (w=256) layers for every full-attention layer
sw	Sliding-window attention (w=256) in all layers

The final checkpoints (200B tokens, step 95,000, after LR cooldown) are on branches {backbone} and {backbone}_200B.

For hybrid and sw, intermediate checkpoints taken every 5,000 steps (before LR cooldown) are available as {backbone}_{tokens}B: 11B, 21B, 32B, 42B, 53B, 63B, 74B, 84B, 95B, 105B, 116B, 126B, 137B, 147B, 158B, 168B, 179B and 189B. Token counts are rounded up (512 sequences × 4,096 tokens per step).

SFT checkpoints

Each final checkpoint was also fine-tuned on Tulu-3 (sequences of up to 16k tokens, LR 1e-4, 12,138 steps). These checkpoints are on branches {backbone}_sft and {backbone}_200B_sft, which point to the same commit. They include a chat template. Full and Hybrid SFT models were fine-tuned with YaRN (×4).

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "nthngdy/cplm-baselines-1.7B"
tok = AutoTokenizer.from_pretrained(repo, revision="full")
model = AutoModelForCausalLM.from_pretrained(repo, revision="full")

ids = tok("The capital of France is", return_tensors="pt").input_ids
print(tok.decode(model.generate(ids, max_new_tokens=32)[0]))

Training

• Architecture: SmolLM3 with 24 layers, hidden size 2,048, 32 heads, tied embeddings and a 49,152-token vocabulary.
• Data: 200B tokens of FineWeb-Edu, 4,096-token sequences.
• Recipe: SmolLM2 recipe with AdamW, peak LR 5e-4, WSD schedule (2,000 warmup steps, 4,750 cooldown steps) and batch size 512.

Citation

@article{godey2026cplm,
  title   = {Copy-Paste Language Models},
  author  = {Godey, Nathan and Artzi, Yoav},
  journal = {TODO},
  year    = {2026}
}