C-LPNCM-Tiny-124M

Calibrated Looped-Pointer Next-Concept Model (123.6M · TinyStories)
Pure-Transformer Hierarchical Concept Planning · Weight-Tied Equilibrium Refinement · Dual-Path Pointer Decoding

License: MIT   Model on Hugging Face   NCM Paper (arXiv:2610.14162)   C-LPNCM Collection   Parameters: 123.6M


entity2260/cl-pncm-tiny-124m is a compact 123.6M-parameter pure-Transformer Calibrated Looped-Pointer Next-Concept Model (C-LPNCM) trained on TinyStories (hazyresearch/ncm-datasets) for fast single-GPU experimentation (fits easily on a single Tesla T4 / Colab GPU).

Looking for the larger flagship checkpoint? See entity2260/cl-pncm-2.8b (2.89B physical / 3.60B effective parameters) in the C-LPNCM Collection.

Highlights

  • Zero Diffusion, 100% Pure Transformer — Shifts generation from flat token-by-token autoregression to discrete multiscale quantization (Cosine VQ + 3× FSQ) and parallel span decoding.
  • Leak-Free Dual-Path Surface Decoding — Path A decodes strictly from quantized representations $E_{\text{quant}} = U_0 + \sum_{\ell=2}^{4} \hat{z}^{(\ell)}$, while Path B queries an unquantized $(K_{\text{raw}}, V_{\text{raw}})$ skip bank strictly at prior tokens $j < t_{\text{start}}(m)$ (with <NULL_PTR>) to preserve proper nouns (e.g., "Tom") verbatim.
  • Guarded XSA & Banach Contraction Loops — Module 3 iterates 3 weight-tied Transformer layers up to 6 times with anchor re-injection ($U_0$) and Banach contraction regularization.
  • Strictly Proper Brier Epistemic Calibration — Module 2 detaches backbone features before $q_\phi$ and optimizes Brier loss $(\hat{p}_{\text{conf}} - \mathbb{I}[\text{correct}])^2$ for honest uncertainty-driven routing.

Architecture & Model Family

C-LPNCM Three-Tower Architecture

General C-LPNCM three-tower / four-module architecture (shown with 2.8B dimensions; exact 124M hyperparameters are listed below).

C-LPNCM-Tiny-124M Configuration

Specification Value
Total Parameters 123.6M
Vocabulary Size (vocab_size) 8,192 (custom BPE, bpe_vocab_8192.pickle)
Hidden Dimensions (d_enc / d_core / d_dec) 512 / 768 / 512
Layers (Enc / Core / Loop / Dec) 4 / 8 / 3 (×6 loops) / 4 (+ 2-layer Boundary Refiner)
Attention Heads (Enc / Core / Loop / Dec) 8Q/2KV · 12Q/4KV · 8 · 8
Coarse Concept Codebook ($K_1$) 4,096 Cosine VQ codes with EMA updates
Fine Residual Hierarchy ($L_{\text{FSQ}}$) 3 cascaded FSQ levels ($z^{(2..4)}$) × [8, 8, 8, 6, 5]
Dynamic Concept Patch Size ($S_m$) $1 \le S_m \le 8$ (split at $\tau_H = 2.4\text{ bits}$, window $W=128$)
Epistemic Routing Thresholds $\tau_{\text{high}} = 0.85$ (Fast) · $\tau_{\text{low}} = 0.40$ (Abstain) · 4 deliberative beams

C-LPNCM Family Comparison

Model Variant Checkpoint / Config Total Params $d_{\text{enc}} / d_{\text{core}} / d_{\text{dec}}$ Layers (Enc / Core / Loop / Dec) Coarse VQ Target Hardware
C-LPNCM-Tiny-124M (this repo) entity2260/cl-pncm-tiny-124m 123.6M 512 / 768 / 512 4 / 8 / 3 (×6) / 4 4,096 1× Tesla T4 (16 GB)
C-LPNCM-660M (Base) CLPNCMConfig.base() 659.6M 1024 / 1536 / 1024 4 / 12 / 4 (×6) / 4 4,096 1× L4 / A100
C-LPNCM-1B configs/cl_pncm_1b_a100.yml 1.04B 1024 / 2048 / 1024 6 / 12 / 4 (×6) / 6 16,384 1× A100 (40/80 GB)
C-LPNCM-2.8B (flagship) entity2260/cl-pncm-2.8b 2.89B (3.60B eff.) 1536 / 3072 / 1536 6 / 20 / 4 (×8) / 6 16,384 1× L4 / T4 (inf.) · A100
C-LPNCM-8B (Prod) configs/cl_pncm_8b.yml 7.85B 2048 / 4096 / 2048 6 / 24 / 4 (×8) / 6 16,384 4×–8× A100/H100

Training Metrics & Qualitative Output

Trained on hazyresearch/ncm-datasets (tinystories/train_bpe_8192.bin, 542.7M tokens, custom BPE vocabulary $V=8{,}192$):

Metric Warmup Progression (1,000 steps)
Stage 1 Surface Token Reconstruction CE Loss 9.0726 → 1.2899
Stage 1 Auxiliary Entropy Head Loss 9.1262 → 4.1266
Stage 2 Coarse Next-Concept CE Loss 8.5003 → 6.1182
Stage 2 Multi-Exit FSQ Equilibrium Loss 28.0830 → 2.7818
Stage 2 Banach Contraction Penalty 0.0105 → 0.0014

Qualitative Example (1,000-Step Warmup Checkpoint)

Stage Output
Input Prompt (16 tokens) Once upon a time, there was a boy named Tom and
Concept-Bottleneck Reconstruction Once upon a time, there was a boy very Tom and (15/16 tokens exact, proper noun "Tom" preserved)
Generated Next-Concept Continuation He was happy then found laughing and (6 concepts → variable span lengths [1, 1, 2, 1, 2, 1])

Quickstart Usage

Option A: Load Merged Checkpoint Directly with CLPNCM

import pickle
import torch
from huggingface_hub import hf_hub_download
from looped_dit.cl_pncm import CLPNCM, CLPNCMConfig

repo_id = "entity2260/cl-pncm-tiny-124m"
device = "cuda" if torch.cuda.is_available() else "cpu"

ckpt_file = hf_hub_download(repo_id=repo_id, filename="cl_pncm_merged.pt")
vocab_file = hf_hub_download(repo_id=repo_id, filename="bpe_vocab_8192.pickle")

ckpt = torch.load(ckpt_file, map_location=device, weights_only=True)
with open(vocab_file, "rb") as f:
    bpe_vocab = pickle.load(f)

cfg = CLPNCMConfig.from_dict(ckpt["config"])
model = CLPNCM(cfg).to(device)
model.load_state_dict(ckpt["model"])
model.eval()

prompt_ids = torch.tensor([[1, 412, 895, 19, 334, 28, 12, 601]], device=device)
gen_out = model.generate(prompt_ids, max_concepts=6)
print("Generated Token IDs:", gen_out["tokens"])

Citation

If you use this model or build upon Next-Concept Modeling, please cite the foundational Next-Concept Modeling (NCM) paper and this checkpoint:

Next-Concept Modeling (NCM) Paper

@misc{kumbong2026ncm,
  title         = {Next-Concept Modeling: Fast, Parallel Coarse-to-Fine Text Generation},
  author        = {Hermann Kumbong and Ishan S. Khare and Iddah Mlauzi and Andrew Woen and Charles Ding and E. Kelly Buchanan and Simran Arora and Azalia Mirhoseini and Christopher R{\'e}},
  year          = {2026},
  eprint        = {2610.14162},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/2610.14162}
}

C-LPNCM-Tiny-124M Model

@misc{entity2260_clpncm_tiny_124m_2026,
  title         = {C-LPNCM-Tiny-124M: Calibrated Looped-Pointer Next-Concept Model},
  author        = {Ranjith Kumar (entity2260)},
  year          = {2026},
  publisher     = {Hugging Face},
  howpublished  = {\url{https://huggingface.co/entity2260/cl-pncm-tiny-124m}}
}

License

This model and its accompanying code are released under the MIT License.

Downloads last month
176
Safetensors
Model size
0.1B params
Tensor type
I64
·
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Dataset used to train entity2260/cl-pncm-tiny-124m

Collection including entity2260/cl-pncm-tiny-124m

Paper for entity2260/cl-pncm-tiny-124m

Evaluation results

  • Stage 1 Token Reconstruction CE Loss (1k steps) on TinyStories (hazyresearch/ncm-datasets)
    self-reported
    1.290
  • Stage 2 Multi-Exit FSQ Equilibrium Loss on TinyStories (hazyresearch/ncm-datasets)
    self-reported
    2.782