cl-pncm-tiny-124m / README.md
entity2260's picture
Modernize model card with badges, verified architecture SVG, fixed repo IDs, and NCM paper citation
b442269 verified
|
Raw History Blame Contribute Delete
9.27 kB
---
license: mit
library_name: pytorch
pipeline_tag: text-generation
language:
- en
tags:
- next-concept-model
- cl-pncm
- looped-transformer
- rvq-fsq
- finite-scalar-quantization
- pointer-network
- calibrated-reasoning
- grpo
- pure-transformer
datasets:
- hazyresearch/ncm-datasets
model-index:
- name: CL-PNCM-TinyStories-124M
results:
- task:
type: text-generation
name: Next-Concept Modeling & Autoencoding Reconstruction
dataset:
name: TinyStories (hazyresearch/ncm-datasets)
type: hazyresearch/ncm-datasets
metrics:
- type: cross_entropy_loss
value: 1.2899
name: Stage 1 Token Reconstruction CE Loss (1k steps)
- type: fsq_equilibrium_loss
value: 2.7818
name: Stage 2 Multi-Exit FSQ Equilibrium Loss
---
<h1 align="center">C-LPNCM-Tiny-124M</h1>
<p align="center">
<strong>Calibrated Looped-Pointer Next-Concept Model (123.6M · TinyStories)</strong><br/>
Pure-Transformer Hierarchical Concept Planning · Weight-Tied Equilibrium Refinement · Dual-Path Pointer Decoding
</p>
<p align="center">
<a href="https://opensource.org/licenses/MIT"><img src="https://img.shields.io/badge/License-MIT-yellow" alt="License: MIT" /></a>
&nbsp;
<a href="https://huggingface.co/entity2260/cl-pncm-tiny-124m"><img src="https://img.shields.io/badge/Hugging%20Face-Model-FFD21E?logo=huggingface&logoColor=000" alt="Model on Hugging Face" /></a>
&nbsp;
<a href="https://arxiv.org/abs/2610.14162"><img src="https://img.shields.io/badge/Paper-NCM%20(arXiv%3A2610.14162)-b31b1b?logo=arxiv&logoColor=white" alt="NCM Paper (arXiv:2610.14162)" /></a>
&nbsp;
<a href="https://huggingface.co/collections/entity2260/c-lpncm-next-concept-modeling-6aff068d4083c50fbdf92e32"><img src="https://img.shields.io/badge/Collection-C--LPNCM-534AB7" alt="C-LPNCM Collection" /></a>
&nbsp;
<img src="https://img.shields.io/badge/Params-123.6M-0F6E56" alt="Parameters: 123.6M" />
</p>
---
**`entity2260/cl-pncm-tiny-124m`** is a compact **123.6M-parameter** pure-Transformer **Calibrated Looped-Pointer Next-Concept Model (C-LPNCM)** trained on TinyStories ([`hazyresearch/ncm-datasets`](https://huggingface.co/datasets/hazyresearch/ncm-datasets)) for fast single-GPU experimentation (fits easily on a single Tesla T4 / Colab GPU).
> Looking for the larger flagship checkpoint? See **[`entity2260/cl-pncm-2.8b`](https://huggingface.co/entity2260/cl-pncm-2.8b)** (`2.89B` physical / `3.60B` effective parameters) in the [C-LPNCM Collection](https://huggingface.co/collections/entity2260/c-lpncm-next-concept-modeling-6aff068d4083c50fbdf92e32).
### Highlights
- **Zero Diffusion, 100% Pure Transformer** — Shifts generation from flat token-by-token autoregression to discrete multiscale quantization (`Cosine VQ + 3× FSQ`) and parallel span decoding.
- **Leak-Free Dual-Path Surface Decoding** — **Path A** decodes strictly from quantized representations $E_{\text{quant}} = U_0 + \sum_{\ell=2}^{4} \hat{z}^{(\ell)}$, while **Path B** queries an unquantized $(K_{\text{raw}}, V_{\text{raw}})$ skip bank strictly at prior tokens $j < t_{\text{start}}(m)$ (with `<NULL_PTR>`) to preserve proper nouns (e.g., `"Tom"`) verbatim.
- **Guarded XSA & Banach Contraction Loops** — Module 3 iterates `3` weight-tied Transformer layers up to `6` times with anchor re-injection ($U_0$) and Banach contraction regularization.
- **Strictly Proper Brier Epistemic Calibration** — Module 2 detaches backbone features before $q_\phi$ and optimizes Brier loss $(\hat{p}_{\text{conf}} - \mathbb{I}[\text{correct}])^2$ for honest uncertainty-driven routing.
---
## Architecture & Model Family
<p align="center">
<img src="https://huggingface.co/entity2260/cl-pncm-2.8b/resolve/main/assets/a_nseg_llm_style_architecture.svg" alt="C-LPNCM Three-Tower Architecture" width="100%" />
</p>
<p align="center"><em>General C-LPNCM three-tower / four-module architecture (shown with 2.8B dimensions; exact 124M hyperparameters are listed below).</em></p>
### C-LPNCM-Tiny-124M Configuration
| Specification | Value |
| :--- | :--- |
| **Total Parameters** | **123.6M** |
| **Vocabulary Size (`vocab_size`)** | `8,192` (custom BPE, `bpe_vocab_8192.pickle`) |
| **Hidden Dimensions (`d_enc / d_core / d_dec`)** | `512 / 768 / 512` |
| **Layers (`Enc / Core / Loop / Dec`)** | `4 / 8 / 3 (×6 loops) / 4` (+ 2-layer Boundary Refiner) |
| **Attention Heads (`Enc / Core / Loop / Dec`)** | `8Q/2KV` · `12Q/4KV` · `8` · `8` |
| **Coarse Concept Codebook ($K_1$)** | `4,096` Cosine VQ codes with EMA updates |
| **Fine Residual Hierarchy ($L_{\text{FSQ}}$)** | `3` cascaded FSQ levels ($z^{(2..4)}$) × `[8, 8, 8, 6, 5]` |
| **Dynamic Concept Patch Size ($S_m$)** | $1 \le S_m \le 8$ (split at $\tau_H = 2.4\text{ bits}$, window $W=128$) |
| **Epistemic Routing Thresholds** | $\tau_{\text{high}} = 0.85$ (Fast) · $\tau_{\text{low}} = 0.40$ (Abstain) · `4` deliberative beams |
### C-LPNCM Family Comparison
| Model Variant | Checkpoint / Config | Total Params | $d_{\text{enc}} / d_{\text{core}} / d_{\text{dec}}$ | Layers (`Enc / Core / Loop / Dec`) | Coarse VQ | Target Hardware |
| :--- | :--- | :--- | :--- | :--- | :--- | :--- |
| **C-LPNCM-Tiny-124M** *(this repo)* | [`entity2260/cl-pncm-tiny-124m`](https://huggingface.co/entity2260/cl-pncm-tiny-124m) | **123.6M** | `512 / 768 / 512` | `4 / 8 / 3 (×6) / 4` | `4,096` | 1× Tesla T4 (16 GB) |
| **C-LPNCM-660M (Base)** | `CLPNCMConfig.base()` | **659.6M** | `1024 / 1536 / 1024` | `4 / 12 / 4 (×6) / 4` | `4,096` | 1× L4 / A100 |
| **C-LPNCM-1B** | `configs/cl_pncm_1b_a100.yml` | **1.04B** | `1024 / 2048 / 1024` | `6 / 12 / 4 (×6) / 6` | `16,384` | 1× A100 (40/80 GB) |
| **C-LPNCM-2.8B** *(flagship)* | [`entity2260/cl-pncm-2.8b`](https://huggingface.co/entity2260/cl-pncm-2.8b) | **2.89B** (`3.60B` eff.) | `1536 / 3072 / 1536` | `6 / 20 / 4 (×8) / 6` | `16,384` | 1× L4 / T4 (inf.) · A100 |
| **C-LPNCM-8B (Prod)** | `configs/cl_pncm_8b.yml` | **7.85B** | `2048 / 4096 / 2048` | `6 / 24 / 4 (×8) / 6` | `16,384` | 4×–8× A100/H100 |
---
## Training Metrics & Qualitative Output
Trained on [`hazyresearch/ncm-datasets`](https://huggingface.co/datasets/hazyresearch/ncm-datasets) (`tinystories/train_bpe_8192.bin`, `542.7M` tokens, custom BPE vocabulary $V=8{,}192$):
| Metric | Warmup Progression (1,000 steps) |
| :--- | :--- |
| **Stage 1 Surface Token Reconstruction CE Loss** | `9.0726 → 1.2899` |
| **Stage 1 Auxiliary Entropy Head Loss** | `9.1262 → 4.1266` |
| **Stage 2 Coarse Next-Concept CE Loss** | `8.5003 → 6.1182` |
| **Stage 2 Multi-Exit FSQ Equilibrium Loss** | `28.0830 → 2.7818` |
| **Stage 2 Banach Contraction Penalty** | `0.0105 → 0.0014` |
### Qualitative Example (1,000-Step Warmup Checkpoint)
| Stage | Output |
| :--- | :--- |
| **Input Prompt (16 tokens)** | `Once upon a time, there was a boy named Tom and` |
| **Concept-Bottleneck Reconstruction** | `Once upon a time, there was a boy very Tom and` *(15/16 tokens exact, proper noun `"Tom"` preserved)* |
| **Generated Next-Concept Continuation** | `He was happy then found laughing and` *(6 concepts → variable span lengths `[1, 1, 2, 1, 2, 1]`)* |
---
## Quickstart Usage
### Option A: Load Merged Checkpoint Directly with `CLPNCM`
```python
import pickle
import torch
from huggingface_hub import hf_hub_download
from looped_dit.cl_pncm import CLPNCM, CLPNCMConfig
repo_id = "entity2260/cl-pncm-tiny-124m"
device = "cuda" if torch.cuda.is_available() else "cpu"
ckpt_file = hf_hub_download(repo_id=repo_id, filename="cl_pncm_merged.pt")
vocab_file = hf_hub_download(repo_id=repo_id, filename="bpe_vocab_8192.pickle")
ckpt = torch.load(ckpt_file, map_location=device, weights_only=True)
with open(vocab_file, "rb") as f:
bpe_vocab = pickle.load(f)
cfg = CLPNCMConfig.from_dict(ckpt["config"])
model = CLPNCM(cfg).to(device)
model.load_state_dict(ckpt["model"])
model.eval()
prompt_ids = torch.tensor([[1, 412, 895, 19, 334, 28, 12, 601]], device=device)
gen_out = model.generate(prompt_ids, max_concepts=6)
print("Generated Token IDs:", gen_out["tokens"])
```
---
## Citation
If you use this model or build upon Next-Concept Modeling, please cite the foundational **Next-Concept Modeling (NCM)** paper and this checkpoint:
### Next-Concept Modeling (NCM) Paper
```bibtex
@misc{kumbong2026ncm,
title = {Next-Concept Modeling: Fast, Parallel Coarse-to-Fine Text Generation},
author = {Hermann Kumbong and Ishan S. Khare and Iddah Mlauzi and Andrew Woen and Charles Ding and E. Kelly Buchanan and Simran Arora and Azalia Mirhoseini and Christopher R{\'e}},
year = {2026},
eprint = {2610.14162},
archivePrefix = {arXiv},
primaryClass = {cs.CL},
url = {https://arxiv.org/abs/2610.14162}
}
```
### C-LPNCM-Tiny-124M Model
```bibtex
@misc{entity2260_clpncm_tiny_124m_2026,
title = {C-LPNCM-Tiny-124M: Calibrated Looped-Pointer Next-Concept Model},
author = {Ranjith Kumar (entity2260)},
year = {2026},
publisher = {Hugging Face},
howpublished = {\url{https://huggingface.co/entity2260/cl-pncm-tiny-124m}}
}
```
---
## License
This model and its accompanying code are released under the [MIT License](https://opensource.org/licenses/MIT).