Modernize model card with badges, verified architecture SVG, fixed repo IDs, and NCM paper citation
b442269 verified |
Download README.md from entity2260/cl-pncm-tiny-124m: direct link, hf CLI and curl.
- Browser
- Download file 9.27 kB
-
https://huggingface.co/entity2260/cl-pncm-tiny-124m/resolve/main/README.md
- Command line
-
hf download hf://entity2260/cl-pncm-tiny-124m/README.md
-
curl -L -o README.md https://huggingface.co/entity2260/cl-pncm-tiny-124m/resolve/main/README.md
9.27 kB
| license: mit | |
| library_name: pytorch | |
| pipeline_tag: text-generation | |
| language: | |
| - en | |
| tags: | |
| - next-concept-model | |
| - cl-pncm | |
| - looped-transformer | |
| - rvq-fsq | |
| - finite-scalar-quantization | |
| - pointer-network | |
| - calibrated-reasoning | |
| - grpo | |
| - pure-transformer | |
| datasets: | |
| - hazyresearch/ncm-datasets | |
| model-index: | |
| - name: CL-PNCM-TinyStories-124M | |
| results: | |
| - task: | |
| type: text-generation | |
| name: Next-Concept Modeling & Autoencoding Reconstruction | |
| dataset: | |
| name: TinyStories (hazyresearch/ncm-datasets) | |
| type: hazyresearch/ncm-datasets | |
| metrics: | |
| - type: cross_entropy_loss | |
| value: 1.2899 | |
| name: Stage 1 Token Reconstruction CE Loss (1k steps) | |
| - type: fsq_equilibrium_loss | |
| value: 2.7818 | |
| name: Stage 2 Multi-Exit FSQ Equilibrium Loss | |
| <h1 align="center">C-LPNCM-Tiny-124M</h1> | |
| <p align="center"> | |
| <strong>Calibrated Looped-Pointer Next-Concept Model (123.6M · TinyStories)</strong><br/> | |
| Pure-Transformer Hierarchical Concept Planning · Weight-Tied Equilibrium Refinement · Dual-Path Pointer Decoding | |
| </p> | |
| <p align="center"> | |
| <a href="https://opensource.org/licenses/MIT"><img src="https://img.shields.io/badge/License-MIT-yellow" alt="License: MIT" /></a> | |
| | |
| <a href="https://huggingface.co/entity2260/cl-pncm-tiny-124m"><img src="https://img.shields.io/badge/Hugging%20Face-Model-FFD21E?logo=huggingface&logoColor=000" alt="Model on Hugging Face" /></a> | |
| | |
| <a href="https://arxiv.org/abs/2610.14162"><img src="https://img.shields.io/badge/Paper-NCM%20(arXiv%3A2610.14162)-b31b1b?logo=arxiv&logoColor=white" alt="NCM Paper (arXiv:2610.14162)" /></a> | |
| | |
| <a href="https://huggingface.co/collections/entity2260/c-lpncm-next-concept-modeling-6aff068d4083c50fbdf92e32"><img src="https://img.shields.io/badge/Collection-C--LPNCM-534AB7" alt="C-LPNCM Collection" /></a> | |
| | |
| <img src="https://img.shields.io/badge/Params-123.6M-0F6E56" alt="Parameters: 123.6M" /> | |
| </p> | |
| --- | |
| **`entity2260/cl-pncm-tiny-124m`** is a compact **123.6M-parameter** pure-Transformer **Calibrated Looped-Pointer Next-Concept Model (C-LPNCM)** trained on TinyStories ([`hazyresearch/ncm-datasets`](https://huggingface.co/datasets/hazyresearch/ncm-datasets)) for fast single-GPU experimentation (fits easily on a single Tesla T4 / Colab GPU). | |
| > Looking for the larger flagship checkpoint? See **[`entity2260/cl-pncm-2.8b`](https://huggingface.co/entity2260/cl-pncm-2.8b)** (`2.89B` physical / `3.60B` effective parameters) in the [C-LPNCM Collection](https://huggingface.co/collections/entity2260/c-lpncm-next-concept-modeling-6aff068d4083c50fbdf92e32). | |
| ### Highlights | |
| - **Zero Diffusion, 100% Pure Transformer** — Shifts generation from flat token-by-token autoregression to discrete multiscale quantization (`Cosine VQ + 3× FSQ`) and parallel span decoding. | |
| - **Leak-Free Dual-Path Surface Decoding** — **Path A** decodes strictly from quantized representations $E_{\text{quant}} = U_0 + \sum_{\ell=2}^{4} \hat{z}^{(\ell)}$, while **Path B** queries an unquantized $(K_{\text{raw}}, V_{\text{raw}})$ skip bank strictly at prior tokens $j < t_{\text{start}}(m)$ (with `<NULL_PTR>`) to preserve proper nouns (e.g., `"Tom"`) verbatim. | |
| - **Guarded XSA & Banach Contraction Loops** — Module 3 iterates `3` weight-tied Transformer layers up to `6` times with anchor re-injection ($U_0$) and Banach contraction regularization. | |
| - **Strictly Proper Brier Epistemic Calibration** — Module 2 detaches backbone features before $q_\phi$ and optimizes Brier loss $(\hat{p}_{\text{conf}} - \mathbb{I}[\text{correct}])^2$ for honest uncertainty-driven routing. | |
| --- | |
| ## Architecture & Model Family | |
| <p align="center"> | |
| <img src="https://huggingface.co/entity2260/cl-pncm-2.8b/resolve/main/assets/a_nseg_llm_style_architecture.svg" alt="C-LPNCM Three-Tower Architecture" width="100%" /> | |
| </p> | |
| <p align="center"><em>General C-LPNCM three-tower / four-module architecture (shown with 2.8B dimensions; exact 124M hyperparameters are listed below).</em></p> | |
| ### C-LPNCM-Tiny-124M Configuration | |
| | Specification | Value | | |
| | :--- | :--- | | |
| | **Total Parameters** | **123.6M** | | |
| | **Vocabulary Size (`vocab_size`)** | `8,192` (custom BPE, `bpe_vocab_8192.pickle`) | | |
| | **Hidden Dimensions (`d_enc / d_core / d_dec`)** | `512 / 768 / 512` | | |
| | **Layers (`Enc / Core / Loop / Dec`)** | `4 / 8 / 3 (×6 loops) / 4` (+ 2-layer Boundary Refiner) | | |
| | **Attention Heads (`Enc / Core / Loop / Dec`)** | `8Q/2KV` · `12Q/4KV` · `8` · `8` | | |
| | **Coarse Concept Codebook ($K_1$)** | `4,096` Cosine VQ codes with EMA updates | | |
| | **Fine Residual Hierarchy ($L_{\text{FSQ}}$)** | `3` cascaded FSQ levels ($z^{(2..4)}$) × `[8, 8, 8, 6, 5]` | | |
| | **Dynamic Concept Patch Size ($S_m$)** | $1 \le S_m \le 8$ (split at $\tau_H = 2.4\text{ bits}$, window $W=128$) | | |
| | **Epistemic Routing Thresholds** | $\tau_{\text{high}} = 0.85$ (Fast) · $\tau_{\text{low}} = 0.40$ (Abstain) · `4` deliberative beams | | |
| ### C-LPNCM Family Comparison | |
| | Model Variant | Checkpoint / Config | Total Params | $d_{\text{enc}} / d_{\text{core}} / d_{\text{dec}}$ | Layers (`Enc / Core / Loop / Dec`) | Coarse VQ | Target Hardware | | |
| | :--- | :--- | :--- | :--- | :--- | :--- | :--- | | |
| | **C-LPNCM-Tiny-124M** *(this repo)* | [`entity2260/cl-pncm-tiny-124m`](https://huggingface.co/entity2260/cl-pncm-tiny-124m) | **123.6M** | `512 / 768 / 512` | `4 / 8 / 3 (×6) / 4` | `4,096` | 1× Tesla T4 (16 GB) | | |
| | **C-LPNCM-660M (Base)** | `CLPNCMConfig.base()` | **659.6M** | `1024 / 1536 / 1024` | `4 / 12 / 4 (×6) / 4` | `4,096` | 1× L4 / A100 | | |
| | **C-LPNCM-1B** | `configs/cl_pncm_1b_a100.yml` | **1.04B** | `1024 / 2048 / 1024` | `6 / 12 / 4 (×6) / 6` | `16,384` | 1× A100 (40/80 GB) | | |
| | **C-LPNCM-2.8B** *(flagship)* | [`entity2260/cl-pncm-2.8b`](https://huggingface.co/entity2260/cl-pncm-2.8b) | **2.89B** (`3.60B` eff.) | `1536 / 3072 / 1536` | `6 / 20 / 4 (×8) / 6` | `16,384` | 1× L4 / T4 (inf.) · A100 | | |
| | **C-LPNCM-8B (Prod)** | `configs/cl_pncm_8b.yml` | **7.85B** | `2048 / 4096 / 2048` | `6 / 24 / 4 (×8) / 6` | `16,384` | 4×–8× A100/H100 | | |
| --- | |
| ## Training Metrics & Qualitative Output | |
| Trained on [`hazyresearch/ncm-datasets`](https://huggingface.co/datasets/hazyresearch/ncm-datasets) (`tinystories/train_bpe_8192.bin`, `542.7M` tokens, custom BPE vocabulary $V=8{,}192$): | |
| | Metric | Warmup Progression (1,000 steps) | | |
| | :--- | :--- | | |
| | **Stage 1 Surface Token Reconstruction CE Loss** | `9.0726 → 1.2899` | | |
| | **Stage 1 Auxiliary Entropy Head Loss** | `9.1262 → 4.1266` | | |
| | **Stage 2 Coarse Next-Concept CE Loss** | `8.5003 → 6.1182` | | |
| | **Stage 2 Multi-Exit FSQ Equilibrium Loss** | `28.0830 → 2.7818` | | |
| | **Stage 2 Banach Contraction Penalty** | `0.0105 → 0.0014` | | |
| ### Qualitative Example (1,000-Step Warmup Checkpoint) | |
| | Stage | Output | | |
| | :--- | :--- | | |
| | **Input Prompt (16 tokens)** | `Once upon a time, there was a boy named Tom and` | | |
| | **Concept-Bottleneck Reconstruction** | `Once upon a time, there was a boy very Tom and` *(15/16 tokens exact, proper noun `"Tom"` preserved)* | | |
| | **Generated Next-Concept Continuation** | `He was happy then found laughing and` *(6 concepts → variable span lengths `[1, 1, 2, 1, 2, 1]`)* | | |
| --- | |
| ## Quickstart Usage | |
| ### Option A: Load Merged Checkpoint Directly with `CLPNCM` | |
| ```python | |
| import pickle | |
| import torch | |
| from huggingface_hub import hf_hub_download | |
| from looped_dit.cl_pncm import CLPNCM, CLPNCMConfig | |
| repo_id = "entity2260/cl-pncm-tiny-124m" | |
| device = "cuda" if torch.cuda.is_available() else "cpu" | |
| ckpt_file = hf_hub_download(repo_id=repo_id, filename="cl_pncm_merged.pt") | |
| vocab_file = hf_hub_download(repo_id=repo_id, filename="bpe_vocab_8192.pickle") | |
| ckpt = torch.load(ckpt_file, map_location=device, weights_only=True) | |
| with open(vocab_file, "rb") as f: | |
| bpe_vocab = pickle.load(f) | |
| cfg = CLPNCMConfig.from_dict(ckpt["config"]) | |
| model = CLPNCM(cfg).to(device) | |
| model.load_state_dict(ckpt["model"]) | |
| model.eval() | |
| prompt_ids = torch.tensor([[1, 412, 895, 19, 334, 28, 12, 601]], device=device) | |
| gen_out = model.generate(prompt_ids, max_concepts=6) | |
| print("Generated Token IDs:", gen_out["tokens"]) | |
| ``` | |
| --- | |
| ## Citation | |
| If you use this model or build upon Next-Concept Modeling, please cite the foundational **Next-Concept Modeling (NCM)** paper and this checkpoint: | |
| ### Next-Concept Modeling (NCM) Paper | |
| ```bibtex | |
| @misc{kumbong2026ncm, | |
| title = {Next-Concept Modeling: Fast, Parallel Coarse-to-Fine Text Generation}, | |
| author = {Hermann Kumbong and Ishan S. Khare and Iddah Mlauzi and Andrew Woen and Charles Ding and E. Kelly Buchanan and Simran Arora and Azalia Mirhoseini and Christopher R{\'e}}, | |
| year = {2026}, | |
| eprint = {2610.14162}, | |
| archivePrefix = {arXiv}, | |
| primaryClass = {cs.CL}, | |
| url = {https://arxiv.org/abs/2610.14162} | |
| } | |
| ``` | |
| ### C-LPNCM-Tiny-124M Model | |
| ```bibtex | |
| @misc{entity2260_clpncm_tiny_124m_2026, | |
| title = {C-LPNCM-Tiny-124M: Calibrated Looped-Pointer Next-Concept Model}, | |
| author = {Ranjith Kumar (entity2260)}, | |
| year = {2026}, | |
| publisher = {Hugging Face}, | |
| howpublished = {\url{https://huggingface.co/entity2260/cl-pncm-tiny-124m}} | |
| } | |
| ``` | |
| --- | |
| ## License | |
| This model and its accompanying code are released under the [MIT License](https://opensource.org/licenses/MIT). | |