--- license: mit library_name: pytorch pipeline_tag: text-generation language: - en tags: - next-concept-model - cl-pncm - looped-transformer - rvq-fsq - finite-scalar-quantization - pointer-network - calibrated-reasoning - grpo - pure-transformer datasets: - hazyresearch/ncm-datasets model-index: - name: CL-PNCM-TinyStories-124M results: - task: type: text-generation name: Next-Concept Modeling & Autoencoding Reconstruction dataset: name: TinyStories (hazyresearch/ncm-datasets) type: hazyresearch/ncm-datasets metrics: - type: cross_entropy_loss value: 1.2899 name: Stage 1 Token Reconstruction CE Loss (1k steps) - type: fsq_equilibrium_loss value: 2.7818 name: Stage 2 Multi-Exit FSQ Equilibrium Loss ---

C-LPNCM-Tiny-124M

Calibrated Looped-Pointer Next-Concept Model (123.6M · TinyStories)
Pure-Transformer Hierarchical Concept Planning · Weight-Tied Equilibrium Refinement · Dual-Path Pointer Decoding

License: MIT   Model on Hugging Face   NCM Paper (arXiv:2610.14162)   C-LPNCM Collection   Parameters: 123.6M

--- **`entity2260/cl-pncm-tiny-124m`** is a compact **123.6M-parameter** pure-Transformer **Calibrated Looped-Pointer Next-Concept Model (C-LPNCM)** trained on TinyStories ([`hazyresearch/ncm-datasets`](https://huggingface.co/datasets/hazyresearch/ncm-datasets)) for fast single-GPU experimentation (fits easily on a single Tesla T4 / Colab GPU). > Looking for the larger flagship checkpoint? See **[`entity2260/cl-pncm-2.8b`](https://huggingface.co/entity2260/cl-pncm-2.8b)** (`2.89B` physical / `3.60B` effective parameters) in the [C-LPNCM Collection](https://huggingface.co/collections/entity2260/c-lpncm-next-concept-modeling-6aff068d4083c50fbdf92e32). ### Highlights - **Zero Diffusion, 100% Pure Transformer** — Shifts generation from flat token-by-token autoregression to discrete multiscale quantization (`Cosine VQ + 3× FSQ`) and parallel span decoding. - **Leak-Free Dual-Path Surface Decoding** — **Path A** decodes strictly from quantized representations $E_{\text{quant}} = U_0 + \sum_{\ell=2}^{4} \hat{z}^{(\ell)}$, while **Path B** queries an unquantized $(K_{\text{raw}}, V_{\text{raw}})$ skip bank strictly at prior tokens $j < t_{\text{start}}(m)$ (with ``) to preserve proper nouns (e.g., `"Tom"`) verbatim. - **Guarded XSA & Banach Contraction Loops** — Module 3 iterates `3` weight-tied Transformer layers up to `6` times with anchor re-injection ($U_0$) and Banach contraction regularization. - **Strictly Proper Brier Epistemic Calibration** — Module 2 detaches backbone features before $q_\phi$ and optimizes Brier loss $(\hat{p}_{\text{conf}} - \mathbb{I}[\text{correct}])^2$ for honest uncertainty-driven routing. --- ## Architecture & Model Family

C-LPNCM Three-Tower Architecture

General C-LPNCM three-tower / four-module architecture (shown with 2.8B dimensions; exact 124M hyperparameters are listed below).

### C-LPNCM-Tiny-124M Configuration | Specification | Value | | :--- | :--- | | **Total Parameters** | **123.6M** | | **Vocabulary Size (`vocab_size`)** | `8,192` (custom BPE, `bpe_vocab_8192.pickle`) | | **Hidden Dimensions (`d_enc / d_core / d_dec`)** | `512 / 768 / 512` | | **Layers (`Enc / Core / Loop / Dec`)** | `4 / 8 / 3 (×6 loops) / 4` (+ 2-layer Boundary Refiner) | | **Attention Heads (`Enc / Core / Loop / Dec`)** | `8Q/2KV` · `12Q/4KV` · `8` · `8` | | **Coarse Concept Codebook ($K_1$)** | `4,096` Cosine VQ codes with EMA updates | | **Fine Residual Hierarchy ($L_{\text{FSQ}}$)** | `3` cascaded FSQ levels ($z^{(2..4)}$) × `[8, 8, 8, 6, 5]` | | **Dynamic Concept Patch Size ($S_m$)** | $1 \le S_m \le 8$ (split at $\tau_H = 2.4\text{ bits}$, window $W=128$) | | **Epistemic Routing Thresholds** | $\tau_{\text{high}} = 0.85$ (Fast) · $\tau_{\text{low}} = 0.40$ (Abstain) · `4` deliberative beams | ### C-LPNCM Family Comparison | Model Variant | Checkpoint / Config | Total Params | $d_{\text{enc}} / d_{\text{core}} / d_{\text{dec}}$ | Layers (`Enc / Core / Loop / Dec`) | Coarse VQ | Target Hardware | | :--- | :--- | :--- | :--- | :--- | :--- | :--- | | **C-LPNCM-Tiny-124M** *(this repo)* | [`entity2260/cl-pncm-tiny-124m`](https://huggingface.co/entity2260/cl-pncm-tiny-124m) | **123.6M** | `512 / 768 / 512` | `4 / 8 / 3 (×6) / 4` | `4,096` | 1× Tesla T4 (16 GB) | | **C-LPNCM-660M (Base)** | `CLPNCMConfig.base()` | **659.6M** | `1024 / 1536 / 1024` | `4 / 12 / 4 (×6) / 4` | `4,096` | 1× L4 / A100 | | **C-LPNCM-1B** | `configs/cl_pncm_1b_a100.yml` | **1.04B** | `1024 / 2048 / 1024` | `6 / 12 / 4 (×6) / 6` | `16,384` | 1× A100 (40/80 GB) | | **C-LPNCM-2.8B** *(flagship)* | [`entity2260/cl-pncm-2.8b`](https://huggingface.co/entity2260/cl-pncm-2.8b) | **2.89B** (`3.60B` eff.) | `1536 / 3072 / 1536` | `6 / 20 / 4 (×8) / 6` | `16,384` | 1× L4 / T4 (inf.) · A100 | | **C-LPNCM-8B (Prod)** | `configs/cl_pncm_8b.yml` | **7.85B** | `2048 / 4096 / 2048` | `6 / 24 / 4 (×8) / 6` | `16,384` | 4×–8× A100/H100 | --- ## Training Metrics & Qualitative Output Trained on [`hazyresearch/ncm-datasets`](https://huggingface.co/datasets/hazyresearch/ncm-datasets) (`tinystories/train_bpe_8192.bin`, `542.7M` tokens, custom BPE vocabulary $V=8{,}192$): | Metric | Warmup Progression (1,000 steps) | | :--- | :--- | | **Stage 1 Surface Token Reconstruction CE Loss** | `9.0726 → 1.2899` | | **Stage 1 Auxiliary Entropy Head Loss** | `9.1262 → 4.1266` | | **Stage 2 Coarse Next-Concept CE Loss** | `8.5003 → 6.1182` | | **Stage 2 Multi-Exit FSQ Equilibrium Loss** | `28.0830 → 2.7818` | | **Stage 2 Banach Contraction Penalty** | `0.0105 → 0.0014` | ### Qualitative Example (1,000-Step Warmup Checkpoint) | Stage | Output | | :--- | :--- | | **Input Prompt (16 tokens)** | `Once upon a time, there was a boy named Tom and` | | **Concept-Bottleneck Reconstruction** | `Once upon a time, there was a boy very Tom and` *(15/16 tokens exact, proper noun `"Tom"` preserved)* | | **Generated Next-Concept Continuation** | `He was happy then found laughing and` *(6 concepts → variable span lengths `[1, 1, 2, 1, 2, 1]`)* | --- ## Quickstart Usage ### Option A: Load Merged Checkpoint Directly with `CLPNCM` ```python import pickle import torch from huggingface_hub import hf_hub_download from looped_dit.cl_pncm import CLPNCM, CLPNCMConfig repo_id = "entity2260/cl-pncm-tiny-124m" device = "cuda" if torch.cuda.is_available() else "cpu" ckpt_file = hf_hub_download(repo_id=repo_id, filename="cl_pncm_merged.pt") vocab_file = hf_hub_download(repo_id=repo_id, filename="bpe_vocab_8192.pickle") ckpt = torch.load(ckpt_file, map_location=device, weights_only=True) with open(vocab_file, "rb") as f: bpe_vocab = pickle.load(f) cfg = CLPNCMConfig.from_dict(ckpt["config"]) model = CLPNCM(cfg).to(device) model.load_state_dict(ckpt["model"]) model.eval() prompt_ids = torch.tensor([[1, 412, 895, 19, 334, 28, 12, 601]], device=device) gen_out = model.generate(prompt_ids, max_concepts=6) print("Generated Token IDs:", gen_out["tokens"]) ``` --- ## Citation If you use this model or build upon Next-Concept Modeling, please cite the foundational **Next-Concept Modeling (NCM)** paper and this checkpoint: ### Next-Concept Modeling (NCM) Paper ```bibtex @misc{kumbong2026ncm, title = {Next-Concept Modeling: Fast, Parallel Coarse-to-Fine Text Generation}, author = {Hermann Kumbong and Ishan S. Khare and Iddah Mlauzi and Andrew Woen and Charles Ding and E. Kelly Buchanan and Simran Arora and Azalia Mirhoseini and Christopher R{\'e}}, year = {2026}, eprint = {2610.14162}, archivePrefix = {arXiv}, primaryClass = {cs.CL}, url = {https://arxiv.org/abs/2610.14162} } ``` ### C-LPNCM-Tiny-124M Model ```bibtex @misc{entity2260_clpncm_tiny_124m_2026, title = {C-LPNCM-Tiny-124M: Calibrated Looped-Pointer Next-Concept Model}, author = {Ranjith Kumar (entity2260)}, year = {2026}, publisher = {Hugging Face}, howpublished = {\url{https://huggingface.co/entity2260/cl-pncm-tiny-124m}} } ``` --- ## License This model and its accompanying code are released under the [MIT License](https://opensource.org/licenses/MIT).