ncm-12gb: Next-Concept Model weights trained on a single 12 GB GPU

Gianluca Meola · v0.1, November 2026
Contact: gianluca.meola89@gmail.com

Weights from the study Training a Next-Concept Model on a single 12 GB GPU. Next-Concept Models (paper) generate text coarse to fine: a VQ-VAE tokenizer encodes a whole story into a 12-level hierarchy of discrete codes, and a generator predicts that hierarchy level by level. These are research checkpoints from short training budgets, not production models. Results, methods and limits are documented in the code repository.

Contents

The folders mirror runs/ in the code repository, so downloading this repository into the root of a clone of the code repository is enough to use them.

Folder What it is Parameters Gen. PPL
runs/mytok-12std-dati-rv_9830400_38400x256_s42/ Tokenizer: 12 levels, codebooks initialised from data, dead codes restarted during the first 1,200 steps, 38,400 steps × 256 stories, seed 42 36.4M 91.46 with a generator trained on 51,200 stories
runs/mytok-12std-dati-rv_614400_2400x256_s42/ Tokenizer: same recipe, 2,400 steps × 256 stories (needed by the generator below) 36.4M 108.17 (mean of 2 generator seeds, 51,200 stories)
runs/gen-mytok12std-dati-rv-2400x256_409600_3200x128_c50_lr1e-3_flex_s42/ Generator on the 2,400-step tokenizer above, 409,600 stories, constant corruption 0.5, FlexAttention, lr 1e-3, seed 42 88.6M 42.76
runs/gen-tokuff_409600_3200x128_c50_lr1e-3_flex_s42/ Generator on the authors' official tokenizer (not included here), same recipe 88.6M 26.04

Gen. PPL is generative perplexity under GPT-2 large on 1,280 unconditional samples (lower is better), measured with the authors' evaluation scripts. For reference, the authors' released TinyStories model scores 16.58 in the same measurement and real text 9.67 (as reported in the paper).

Each folder holds config.yaml (the exact training settings) and export/ with the weights and the evaluation summary. Tokenizer folders also contain export/config.yaml, the configuration a generator loads as base_config.

Requirements

  • The code repository, with the upstream NCM code at the pinned commit (see its third_party/README.md).
  • The authors' TinyStories files in artifacts/tinystories/ of the code repository (config.yaml, BPE vocabulary and merges; the official tokenizer checkpoint for the official-tokenizer generator), from hazyresearch/ncm-checkpoints. They are not redistributed here.

Loading

The .pt files contain only tensors and plain containers and load with torch.load(path, map_location="cpu", weights_only=True).

To evaluate a generator from the code repository root:

python -I evaluation/gen_ppl_run.py --run gen-mytok12std-dati-rv-2400x256_409600_3200x128_c50_lr1e-3_flex_s42

Training data

TinyStories (Eldan and Li, 2023), as pre-tokenised by the NCM authors (dataset hazyresearch/ncm-datasets, folder tinystories). See the TinyStories dataset card for its terms.

Limitations

  • Trained with budgets far below the paper's (409,600 stories for the generators against 102.4M reported in the paper).
  • TinyStories only: short, simple English children's stories. The models are not meant for any other use.
  • The 38,400-step tokenizer gives a Gen. PPL ratio of about 1.4× to the official tokenizer with a 51,200-story generator. The packaged generator instead uses the 2,400-step tokenizer and gives about 1.6× at 409,600 stories. These are different comparisons. No generator trained on the 38,400-step tokenizer is included; generators require their matching tokenizer.

License

MIT, like the code repository. The models depend on the NCM code by the paper's authors (MIT) and on TinyStories data.

Citation

Meola, G. (2026). Training a Next-Concept Model on a single 12 GB GPU (v0.1). Zenodo. https://doi.org/10.5281/zenodo.24408541

Code and all training records: https://github.com/janlu89/ncm-12gb

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Dataset used to train janlu89/ncm-12gb

Paper for janlu89/ncm-12gb