cl-pncm-tiny-124m / README.md
entity2260's picture
Modernize model card with badges, verified architecture SVG, fixed repo IDs, and NCM paper citation
b442269 verified
|
Raw History Blame Contribute Delete
9.27 kB
metadata
license: mit
library_name: pytorch
pipeline_tag: text-generation
language:
  - en
tags:
  - next-concept-model
  - cl-pncm
  - looped-transformer
  - rvq-fsq
  - finite-scalar-quantization
  - pointer-network
  - calibrated-reasoning
  - grpo
  - pure-transformer
datasets:
  - hazyresearch/ncm-datasets
model-index:
  - name: CL-PNCM-TinyStories-124M
    results:
      - task:
          type: text-generation
          name: Next-Concept Modeling & Autoencoding Reconstruction
        dataset:
          name: TinyStories (hazyresearch/ncm-datasets)
          type: hazyresearch/ncm-datasets
        metrics:
          - type: cross_entropy_loss
            value: 1.2899
            name: Stage 1 Token Reconstruction CE Loss (1k steps)
          - type: fsq_equilibrium_loss
            value: 2.7818
            name: Stage 2 Multi-Exit FSQ Equilibrium Loss

C-LPNCM-Tiny-124M

Calibrated Looped-Pointer Next-Concept Model (123.6M · TinyStories)
Pure-Transformer Hierarchical Concept Planning · Weight-Tied Equilibrium Refinement · Dual-Path Pointer Decoding

License: MIT   Model on Hugging Face   NCM Paper (arXiv:2610.14162)   C-LPNCM Collection   Parameters: 123.6M


entity2260/cl-pncm-tiny-124m is a compact 123.6M-parameter pure-Transformer Calibrated Looped-Pointer Next-Concept Model (C-LPNCM) trained on TinyStories (hazyresearch/ncm-datasets) for fast single-GPU experimentation (fits easily on a single Tesla T4 / Colab GPU).

Looking for the larger flagship checkpoint? See entity2260/cl-pncm-2.8b (2.89B physical / 3.60B effective parameters) in the C-LPNCM Collection.

Highlights

  • Zero Diffusion, 100% Pure Transformer — Shifts generation from flat token-by-token autoregression to discrete multiscale quantization (Cosine VQ + 3× FSQ) and parallel span decoding.
  • Leak-Free Dual-Path Surface Decoding — Path A decodes strictly from quantized representations $E_{\text{quant}} = U_0 + \sum_{\ell=2}^{4} \hat{z}^{(\ell)}$, while Path B queries an unquantized $(K_{\text{raw}}, V_{\text{raw}})$ skip bank strictly at prior tokens $j < t_{\text{start}}(m)$ (with <NULL_PTR>) to preserve proper nouns (e.g., "Tom") verbatim.
  • Guarded XSA & Banach Contraction Loops — Module 3 iterates 3 weight-tied Transformer layers up to 6 times with anchor re-injection ($U_0$) and Banach contraction regularization.
  • Strictly Proper Brier Epistemic Calibration — Module 2 detaches backbone features before $q_\phi$ and optimizes Brier loss $(\hat{p}_{\text{conf}} - \mathbb{I}[\text{correct}])^2$ for honest uncertainty-driven routing.

Architecture & Model Family

C-LPNCM Three-Tower Architecture

General C-LPNCM three-tower / four-module architecture (shown with 2.8B dimensions; exact 124M hyperparameters are listed below).

C-LPNCM-Tiny-124M Configuration

Specification Value
Total Parameters 123.6M
Vocabulary Size (vocab_size) 8,192 (custom BPE, bpe_vocab_8192.pickle)
Hidden Dimensions (d_enc / d_core / d_dec) 512 / 768 / 512
Layers (Enc / Core / Loop / Dec) 4 / 8 / 3 (×6 loops) / 4 (+ 2-layer Boundary Refiner)
Attention Heads (Enc / Core / Loop / Dec) 8Q/2KV · 12Q/4KV · 8 · 8
Coarse Concept Codebook ($K_1$) 4,096 Cosine VQ codes with EMA updates
Fine Residual Hierarchy ($L_{\text{FSQ}}$) 3 cascaded FSQ levels ($z^{(2..4)}$) × [8, 8, 8, 6, 5]
Dynamic Concept Patch Size ($S_m$) $1 \le S_m \le 8$ (split at $\tau_H = 2.4\text{ bits}$, window $W=128$)
Epistemic Routing Thresholds $\tau_{\text{high}} = 0.85$ (Fast) · $\tau_{\text{low}} = 0.40$ (Abstain) · 4 deliberative beams

C-LPNCM Family Comparison

Model Variant Checkpoint / Config Total Params $d_{\text{enc}} / d_{\text{core}} / d_{\text{dec}}$ Layers (Enc / Core / Loop / Dec) Coarse VQ Target Hardware
C-LPNCM-Tiny-124M (this repo) entity2260/cl-pncm-tiny-124m 123.6M 512 / 768 / 512 4 / 8 / 3 (×6) / 4 4,096 1× Tesla T4 (16 GB)
C-LPNCM-660M (Base) CLPNCMConfig.base() 659.6M 1024 / 1536 / 1024 4 / 12 / 4 (×6) / 4 4,096 1× L4 / A100
C-LPNCM-1B configs/cl_pncm_1b_a100.yml 1.04B 1024 / 2048 / 1024 6 / 12 / 4 (×6) / 6 16,384 1× A100 (40/80 GB)
C-LPNCM-2.8B (flagship) entity2260/cl-pncm-2.8b 2.89B (3.60B eff.) 1536 / 3072 / 1536 6 / 20 / 4 (×8) / 6 16,384 1× L4 / T4 (inf.) · A100
C-LPNCM-8B (Prod) configs/cl_pncm_8b.yml 7.85B 2048 / 4096 / 2048 6 / 24 / 4 (×8) / 6 16,384 4×–8× A100/H100

Training Metrics & Qualitative Output

Trained on hazyresearch/ncm-datasets (tinystories/train_bpe_8192.bin, 542.7M tokens, custom BPE vocabulary $V=8{,}192$):

Metric Warmup Progression (1,000 steps)
Stage 1 Surface Token Reconstruction CE Loss 9.0726 → 1.2899
Stage 1 Auxiliary Entropy Head Loss 9.1262 → 4.1266
Stage 2 Coarse Next-Concept CE Loss 8.5003 → 6.1182
Stage 2 Multi-Exit FSQ Equilibrium Loss 28.0830 → 2.7818
Stage 2 Banach Contraction Penalty 0.0105 → 0.0014

Qualitative Example (1,000-Step Warmup Checkpoint)

Stage Output
Input Prompt (16 tokens) Once upon a time, there was a boy named Tom and
Concept-Bottleneck Reconstruction Once upon a time, there was a boy very Tom and (15/16 tokens exact, proper noun "Tom" preserved)
Generated Next-Concept Continuation He was happy then found laughing and (6 concepts → variable span lengths [1, 1, 2, 1, 2, 1])

Quickstart Usage

Option A: Load Merged Checkpoint Directly with CLPNCM

import pickle
import torch
from huggingface_hub import hf_hub_download
from looped_dit.cl_pncm import CLPNCM, CLPNCMConfig

repo_id = "entity2260/cl-pncm-tiny-124m"
device = "cuda" if torch.cuda.is_available() else "cpu"

ckpt_file = hf_hub_download(repo_id=repo_id, filename="cl_pncm_merged.pt")
vocab_file = hf_hub_download(repo_id=repo_id, filename="bpe_vocab_8192.pickle")

ckpt = torch.load(ckpt_file, map_location=device, weights_only=True)
with open(vocab_file, "rb") as f:
    bpe_vocab = pickle.load(f)

cfg = CLPNCMConfig.from_dict(ckpt["config"])
model = CLPNCM(cfg).to(device)
model.load_state_dict(ckpt["model"])
model.eval()

prompt_ids = torch.tensor([[1, 412, 895, 19, 334, 28, 12, 601]], device=device)
gen_out = model.generate(prompt_ids, max_concepts=6)
print("Generated Token IDs:", gen_out["tokens"])

Citation

If you use this model or build upon Next-Concept Modeling, please cite the foundational Next-Concept Modeling (NCM) paper and this checkpoint:

Next-Concept Modeling (NCM) Paper

@misc{kumbong2026ncm,
  title         = {Next-Concept Modeling: Fast, Parallel Coarse-to-Fine Text Generation},
  author        = {Hermann Kumbong and Ishan S. Khare and Iddah Mlauzi and Andrew Woen and Charles Ding and E. Kelly Buchanan and Simran Arora and Azalia Mirhoseini and Christopher R{\'e}},
  year          = {2026},
  eprint        = {2610.14162},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/2610.14162}
}

C-LPNCM-Tiny-124M Model

@misc{entity2260_clpncm_tiny_124m_2026,
  title         = {C-LPNCM-Tiny-124M: Calibrated Looped-Pointer Next-Concept Model},
  author        = {Ranjith Kumar (entity2260)},
  year          = {2026},
  publisher     = {Hugging Face},
  howpublished  = {\url{https://huggingface.co/entity2260/cl-pncm-tiny-124m}}
}

License

This model and its accompanying code are released under the MIT License.