nthngdy/cplm-1.7B

Copy-Paste Language Models — 1.7B

1.7B-parameter Copy-Paste Language Models (CPLMs) from the paper Copy-Paste Language Models (Nathan Godey & Yoav Artzi, Cornell University).

A CPLM adds a lightweight Copy-Paste (CP) head next to the standard LM head. The CP head attends over past positions and copies the token found there. It is triggered by a dummy <copy> token in the LM head and can fall back to the LM head through a learnable sink. The final next-token distribution is a mixture of the two heads, trained end-to-end with the standard next-token loss only.

Vanilla baselines trained with the exact same recipe are available at nthngdy/cplm-baselines-1.7B.

Checkpoints

Each model is stored on its own branch. Three backbones are available:

Backbone	Description
full	Full attention in all 24 layers
hybrid	3 sliding-window (w=256) layers for every full-attention layer
sw	Sliding-window attention (w=256) in all layers

Branch	Tokens	Step
{backbone} / {backbone}_200B	200B	95,000 (final, after LR cooldown)
{backbone}_168B	168B	80,000
{backbone}_126B	126B	60,000
{backbone}_84B	84B	40,000
{backbone}_42B	42B	20,000

Intermediate checkpoints are taken before the LR cooldown. Token counts are rounded up (512 sequences × 4,096 tokens per step).

SFT checkpoints

Each final checkpoint was also fine-tuned on Tulu-3 (sequences of up to 16k tokens, LR 1e-4, 12,138 steps). These checkpoints are on branches {backbone}_sft and {backbone}_200B_sft, which point to the same commit. They include a chat template. Full and Hybrid SFT models were fine-tuned with YaRN (×4).

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "nthngdy/cplm-1.7B"
tok = AutoTokenizer.from_pretrained(repo, revision="full")
model = AutoModelForCausalLM.from_pretrained(repo, revision="full", trust_remote_code=True)

ids = tok("The capital of France is", return_tensors="pt").input_ids
print(tok.decode(model.generate(ids, max_new_tokens=32)[0]))

model.config.inference_mode controls decoding: "natural" (default, mixture of both heads), "no_copy" (LM head only) or "force_copy" (CP head only).

Training

• Architecture: SmolLM3-style backbone with 24 layers, hidden size 2,048, 32 heads and tied embeddings, plus the CP head. The vocabulary has 49,152 tokens plus <copy>.
• Data: 200B tokens of FineWeb-Edu, 4,096-token sequences.
• Recipe: SmolLM2 recipe with AdamW, peak LR 5e-4, WSD schedule (2,000 warmup steps, 4,750 cooldown steps) and batch size 512.

Results

Zero-shot accuracy (%) of the final checkpoints, averaged over HellaSwag, LAMBADA, PIQA, SciQ, ARC-e, ARC-c and WinoGrande:

Backbone	Vanilla	CPLM
Full	62.45	63.06
Hybrid	62.22	62.59
SW	61.63	62.48

See the paper for long-context, retrieval and SFT results.

Citation

@misc{godey2026copypastelanguagemodels,
      title={Copy-Paste Language Models}, 
      author={Nathan Godey and Yoav Artzi},
      year={2026},
      eprint={2610.19247},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2610.19247}, 
}