IanMagnusson/count-checkpoints-standard-transformer

Standard transformer checkpoints

The standard transformer baseline from The COUNT: Counting N-grams Unlocks Scaling on Host Memory and Training-Free Adaptation (paper, code): plain OLMo-core ladder models with no n-gram components, the reference that COUNT and the learned n-gram embeddings are compared against.

Checkpoints

Seven backbone sizes on the OLMo-core model ladder, each trained on Dolma 3 (allenai/dolma3_mix-6T-1025-7B) and saved at three training budgets: 10, 20 and 40 tokens per backbone parameter (D/N). Each size uses the same global batch as the COUNT and learned n-gram models of that size. Every checkpoint is a full OLMo-core distributed checkpoint with model weights, optimizer state and trainer state.

Size	D/N = 10	D/N = 20	D/N = 40	Global batch (tokens)
60M	step 2190	step 4381	step 8762	262,144
100M	step 3881	step 7762	step 15524	262,144
190M	step 3631	step 7261	step 14523	524,288
370M	step 4721	step 9442	step 18883	786,432
600M	step 5574	step 11148	step 22297	983,040
760M	step 5785	step 11570	step 23139	1,310,720
1B	step 8134	step 16268	step 32537	1,572,864

Paths follow the code's ladder layout, model-ladders/vanilla-<size>/<SIZE>/step<N>/, and each step directory has a release_args.json with its ladder settings. For the full configuration of each model, see the paper's appendix, the config.json and release_args.json in each step directory, and the ladder configuration in the code.

Usage

Evaluate a checkpoint with the COUNT code's ladder, with no method flags:

hf download IanMagnusson/count-checkpoints-standard-transformer --include "model-ladders/vanilla-190m/*" \
  --local-dir $OLMO_ROOT_DIR
torchrun --standalone --nproc-per-node=1 olmo-core/src/scripts/train/ladder/count_ladder.py \
  eval-checkpoint --name vanilla-190m --cluster local/h200 --size 190M --sequence-length 8192 \
  --chinchilla-multiple 2.0 --max-gpus 1 --minimum-world-size 1 --fixed-global-batch-size 524288 \
  --rank-mbz 8 --data-root /path/to/tokenized \
  --checkpoint-path $OLMO_ROOT_DIR/model-ladders/vanilla-190m/190M/step14523 \
  --eval-output-dir /path/to/eval-out --no-load-trainer-state

Evaluation loads model weights only. To resume training on your own copy of the tokenized data, pass --ignore-data-fingerprints: the checkpoints record a fingerprint of the data paths they were trained on, and without the flag the code refuses to resume from different paths.

Related

• COUNT checkpoints
• Learned n-gram embedding checkpoints
• N-gram tables

Citation

@article{magnusson2026count,
  title   = {The {COUNT}: Counting N-grams Unlocks Scaling on Host Memory and Training-Free Adaptation},
  author  = {Magnusson, Ian and Smith, Noah A. and Koh, Pang Wei},
  journal = {arXiv preprint arXiv:2610.20456},
  year    = {2026}
}