Download README.md from entity2260/cl-pncm-tiny-124m: direct link, hf CLI and curl.
- Browser
- Download file 9.27 kB
-
https://huggingface.co/entity2260/cl-pncm-tiny-124m/resolve/main/README.md
- Command line
-
hf download hf://entity2260/cl-pncm-tiny-124m/README.md
-
curl -L -o README.md https://huggingface.co/entity2260/cl-pncm-tiny-124m/resolve/main/README.md
license: mit
library_name: pytorch
pipeline_tag: text-generation
language:
- en
tags:
- next-concept-model
- cl-pncm
- looped-transformer
- rvq-fsq
- finite-scalar-quantization
- pointer-network
- calibrated-reasoning
- grpo
- pure-transformer
datasets:
- hazyresearch/ncm-datasets
model-index:
- name: CL-PNCM-TinyStories-124M
results:
- task:
type: text-generation
name: Next-Concept Modeling & Autoencoding Reconstruction
dataset:
name: TinyStories (hazyresearch/ncm-datasets)
type: hazyresearch/ncm-datasets
metrics:
- type: cross_entropy_loss
value: 1.2899
name: Stage 1 Token Reconstruction CE Loss (1k steps)
- type: fsq_equilibrium_loss
value: 2.7818
name: Stage 2 Multi-Exit FSQ Equilibrium Loss
C-LPNCM-Tiny-124M
Calibrated Looped-Pointer Next-Concept Model (123.6M · TinyStories)
Pure-Transformer Hierarchical Concept Planning · Weight-Tied Equilibrium Refinement · Dual-Path Pointer Decoding
entity2260/cl-pncm-tiny-124m is a compact 123.6M-parameter pure-Transformer Calibrated Looped-Pointer Next-Concept Model (C-LPNCM) trained on TinyStories (hazyresearch/ncm-datasets) for fast single-GPU experimentation (fits easily on a single Tesla T4 / Colab GPU).
Looking for the larger flagship checkpoint? See
entity2260/cl-pncm-2.8b(2.89Bphysical /3.60Beffective parameters) in the C-LPNCM Collection.
Highlights
- Zero Diffusion, 100% Pure Transformer — Shifts generation from flat token-by-token autoregression to discrete multiscale quantization (
Cosine VQ + 3× FSQ) and parallel span decoding. - Leak-Free Dual-Path Surface Decoding — Path A decodes strictly from quantized representations $E_{\text{quant}} = U_0 + \sum_{\ell=2}^{4} \hat{z}^{(\ell)}$, while Path B queries an unquantized $(K_{\text{raw}}, V_{\text{raw}})$ skip bank strictly at prior tokens $j < t_{\text{start}}(m)$ (with
<NULL_PTR>) to preserve proper nouns (e.g.,"Tom") verbatim. - Guarded XSA & Banach Contraction Loops — Module 3 iterates
3weight-tied Transformer layers up to6times with anchor re-injection ($U_0$) and Banach contraction regularization. - Strictly Proper Brier Epistemic Calibration — Module 2 detaches backbone features before $q_\phi$ and optimizes Brier loss $(\hat{p}_{\text{conf}} - \mathbb{I}[\text{correct}])^2$ for honest uncertainty-driven routing.
Architecture & Model Family
General C-LPNCM three-tower / four-module architecture (shown with 2.8B dimensions; exact 124M hyperparameters are listed below).
C-LPNCM-Tiny-124M Configuration
| Specification | Value |
|---|---|
| Total Parameters | 123.6M |
Vocabulary Size (vocab_size) |
8,192 (custom BPE, bpe_vocab_8192.pickle) |
Hidden Dimensions (d_enc / d_core / d_dec) |
512 / 768 / 512 |
Layers (Enc / Core / Loop / Dec) |
4 / 8 / 3 (×6 loops) / 4 (+ 2-layer Boundary Refiner) |
Attention Heads (Enc / Core / Loop / Dec) |
8Q/2KV · 12Q/4KV · 8 · 8 |
| Coarse Concept Codebook ($K_1$) | 4,096 Cosine VQ codes with EMA updates |
| Fine Residual Hierarchy ($L_{\text{FSQ}}$) | 3 cascaded FSQ levels ($z^{(2..4)}$) × [8, 8, 8, 6, 5] |
| Dynamic Concept Patch Size ($S_m$) | $1 \le S_m \le 8$ (split at $\tau_H = 2.4\text{ bits}$, window $W=128$) |
| Epistemic Routing Thresholds | $\tau_{\text{high}} = 0.85$ (Fast) · $\tau_{\text{low}} = 0.40$ (Abstain) · 4 deliberative beams |
C-LPNCM Family Comparison
| Model Variant | Checkpoint / Config | Total Params | $d_{\text{enc}} / d_{\text{core}} / d_{\text{dec}}$ | Layers (Enc / Core / Loop / Dec) |
Coarse VQ | Target Hardware |
|---|---|---|---|---|---|---|
| C-LPNCM-Tiny-124M (this repo) | entity2260/cl-pncm-tiny-124m |
123.6M | 512 / 768 / 512 |
4 / 8 / 3 (×6) / 4 |
4,096 |
1× Tesla T4 (16 GB) |
| C-LPNCM-660M (Base) | CLPNCMConfig.base() |
659.6M | 1024 / 1536 / 1024 |
4 / 12 / 4 (×6) / 4 |
4,096 |
1× L4 / A100 |
| C-LPNCM-1B | configs/cl_pncm_1b_a100.yml |
1.04B | 1024 / 2048 / 1024 |
6 / 12 / 4 (×6) / 6 |
16,384 |
1× A100 (40/80 GB) |
| C-LPNCM-2.8B (flagship) | entity2260/cl-pncm-2.8b |
2.89B (3.60B eff.) |
1536 / 3072 / 1536 |
6 / 20 / 4 (×8) / 6 |
16,384 |
1× L4 / T4 (inf.) · A100 |
| C-LPNCM-8B (Prod) | configs/cl_pncm_8b.yml |
7.85B | 2048 / 4096 / 2048 |
6 / 24 / 4 (×8) / 6 |
16,384 |
4×–8× A100/H100 |
Training Metrics & Qualitative Output
Trained on hazyresearch/ncm-datasets (tinystories/train_bpe_8192.bin, 542.7M tokens, custom BPE vocabulary $V=8{,}192$):
| Metric | Warmup Progression (1,000 steps) |
|---|---|
| Stage 1 Surface Token Reconstruction CE Loss | 9.0726 → 1.2899 |
| Stage 1 Auxiliary Entropy Head Loss | 9.1262 → 4.1266 |
| Stage 2 Coarse Next-Concept CE Loss | 8.5003 → 6.1182 |
| Stage 2 Multi-Exit FSQ Equilibrium Loss | 28.0830 → 2.7818 |
| Stage 2 Banach Contraction Penalty | 0.0105 → 0.0014 |
Qualitative Example (1,000-Step Warmup Checkpoint)
| Stage | Output |
|---|---|
| Input Prompt (16 tokens) | Once upon a time, there was a boy named Tom and |
| Concept-Bottleneck Reconstruction | Once upon a time, there was a boy very Tom and (15/16 tokens exact, proper noun "Tom" preserved) |
| Generated Next-Concept Continuation | He was happy then found laughing and (6 concepts → variable span lengths [1, 1, 2, 1, 2, 1]) |
Quickstart Usage
Option A: Load Merged Checkpoint Directly with CLPNCM
import pickle
import torch
from huggingface_hub import hf_hub_download
from looped_dit.cl_pncm import CLPNCM, CLPNCMConfig
repo_id = "entity2260/cl-pncm-tiny-124m"
device = "cuda" if torch.cuda.is_available() else "cpu"
ckpt_file = hf_hub_download(repo_id=repo_id, filename="cl_pncm_merged.pt")
vocab_file = hf_hub_download(repo_id=repo_id, filename="bpe_vocab_8192.pickle")
ckpt = torch.load(ckpt_file, map_location=device, weights_only=True)
with open(vocab_file, "rb") as f:
bpe_vocab = pickle.load(f)
cfg = CLPNCMConfig.from_dict(ckpt["config"])
model = CLPNCM(cfg).to(device)
model.load_state_dict(ckpt["model"])
model.eval()
prompt_ids = torch.tensor([[1, 412, 895, 19, 334, 28, 12, 601]], device=device)
gen_out = model.generate(prompt_ids, max_concepts=6)
print("Generated Token IDs:", gen_out["tokens"])
Citation
If you use this model or build upon Next-Concept Modeling, please cite the foundational Next-Concept Modeling (NCM) paper and this checkpoint:
Next-Concept Modeling (NCM) Paper
@misc{kumbong2026ncm,
title = {Next-Concept Modeling: Fast, Parallel Coarse-to-Fine Text Generation},
author = {Hermann Kumbong and Ishan S. Khare and Iddah Mlauzi and Andrew Woen and Charles Ding and E. Kelly Buchanan and Simran Arora and Azalia Mirhoseini and Christopher R{\'e}},
year = {2026},
eprint = {2610.14162},
archivePrefix = {arXiv},
primaryClass = {cs.CL},
url = {https://arxiv.org/abs/2610.14162}
}
C-LPNCM-Tiny-124M Model
@misc{entity2260_clpncm_tiny_124m_2026,
title = {C-LPNCM-Tiny-124M: Calibrated Looped-Pointer Next-Concept Model},
author = {Ranjith Kumar (entity2260)},
year = {2026},
publisher = {Hugging Face},
howpublished = {\url{https://huggingface.co/entity2260/cl-pncm-tiny-124m}}
}
License
This model and its accompanying code are released under the MIT License.