C-LPNCM-2.8B Cover Banner

C-LPNCM-2.8B

Calibrated Looped-Pointer Next-Concept Model
Pure-Transformer Hierarchical Concept Planning Β· Weight-Tied Equilibrium Refinement Β· Dual-Path Pointer Decoding

License: MIT   Model on Hugging Face   NCM Paper (arXiv:2610.14162)   Parameters: 2.89B physical / 3.60B effective   Precision: bfloat16


entity2260/cl-pncm-2.8b is a 2.89B physical parameter (~3.60B effective compute parameter via 8Γ— weight-tied equilibrium loops) pure-Transformer Calibrated Looped-Pointer Next-Concept Model (C-LPNCM) built on the Next-Concept Modeling (NCM) paradigm.

Rather than generating one flat token at a time or relying on continuous diffusion noise schedules, C-LPNCM compresses tokens into dynamic semantic patches via Shannon Entropy Spike Detection, quantizes them into a Multiscale Cosine-VQ + 4Γ— FSQ Concept Hierarchy, reasons autoregressively in concept space with a Brier-Calibrated Epistemic Decision Router, iteratively refines latent states to a Banach fixed-point equilibrium through weight-tied loops, and decodes surface tokens in parallel with a Dual-Path Concept + Causal Pointer Decoder.

Highlights

  • Zero Diffusion, 100% Pure Transformer β€” Eliminates continuous noise schedules and iterative denoising in favor of discrete multiscale quantization (Cosine VQ + 4Γ— FSQ) and parallel span decoding.
  • Adaptive Test-Time Compute (K* ∈ 1..8) β€” Dynamically routes easy spans through a fast 1–2 loop path, hard spans through 4-beam deliberative verification with up to 8 equilibrium loops, and OOD/unverifiable inputs to immediate epistemic abstention (100% OOD abstention, 0.0216 Brier error).
  • Zero-Blur Verbatim Copying β€” Preserves an unquantized (K_raw, V_raw) skip memory bank queried by a strictly causal prior-span pointer (Path B) with <NULL_PTR> fallback and dynamic copy gate g ∈ [0, 1].
  • Single-GPU Friendly β€” Ships as 5.38 GB sharded bfloat16 safetensors requiring only 5.88 GB VRAM at inference (runs natively on a single T4, L4, or A100).

Architecture

C-LPNCM-2.8B Three-Tower LLM-Style Architecture

Global System Specifications

Specification Value
Model Class CLPNCMForCausalLM (4-Module / 3-Tower Pure Transformer, Zero Diffusion)
Physical Parameters 2,887,665,414 (2.89B)
Effective Compute Parameters ~3.60B (with up to 8Γ— weight-tied equilibrium refinement loops)
Weight Format & Size bfloat16 Β· 2 sharded .safetensors files (5.38 GB on disk)
Inference VRAM Footprint 5.88 GB (bfloat16 on L4 / A100) Β· ~6.1 GB (float16 on T4)
Vocabulary Size (vocab_size) 30,522
Coarse Concept Codebook ($K_1$) 16,384 unit-sphere Cosine VQ codes with EMA updates
Fine Residual Hierarchy ($L_{\text{FSQ}}$) 4 cascaded FSQ levels ($z^{(2..5)}$) Γ— [8, 8, 8, 6, 5] (15,360 states/level)
Dynamic Concept Patch Size ($S_m$) $1 \le S_m \le 8$ tokens per concept (split at $\tau_H = 2.4\text{ bits}$)
Attention & Normalization GQA (24Q / 6KV), Guarded Exclusive Self-Attention (XSA), Sigmoid Attn Gates, QK-RMSNorm, Pre-RMSNorm, SwiGLU
Positional Encoding 1D Rotary Position Embeddings (RoPE, $\theta = 10{,}000$)

Module-by-Module Structural Breakdown

Module Role Layers Hidden Dim ($d$) Heads ($Q / KV$) Key Sub-Components
Module 1: Token & Patch Encoder Causal Local-Window Encoder + Entropy Boundary Detector + RVQ-FSQ 6 1536 24 / 6 (GQA) Window $W=128$, unquantized $(K_{\text{raw}}, V_{\text{raw}})$ skip bank, intra-patch bi-dir attn + query pooler, Cosine VQ + 4Γ— FSQ
VQ-to-Core Adapter Isometric Coordinate Bridge ($d_{\text{enc}} \to d_{\text{core}}$) 1 1536 β†’ 3072 β€” Anchors Stage 1 VQ codebook coordinates inside Stage 2 latent space
Module 2: Calibrated Latent Core Causal Next-Concept Transformer + Epistemic Decision Router 20 3072 24 / 6 (GQA) <BOS_CONCEPT>, Policy ($\pi_\theta$), Span Length ($P_{\text{len}}$), Brier Calibration ($q_\phi$), Value Verifier ($V_{\text{verify}}$)
Module 3: Looped Equilibrium Refiner Anchor-Injected Weight-Tied Fixed-Point Refiner 4 Γ— 8 loops (32 eff.) 1536 24 / 24 Span-Query Upsampler ($U_0$), Anchor Injection, Guarded XSA + Sigmoid Gate, Causal Concept Cross-Attn ($H_{\text{core}}$), ACT Halting ($\lambda_k$), 4Γ— FSQ bottleneck
Module 4: Dual-Path Surface Decoder Parallel Concept Decoder + Prior-Span Pointer + Boundary Refiner 6 + 2 1536 24 (12 refiner) Path A (6L Concept Transformer β†’ Linear(V)), Path B (Prior-Span Pointer + <NULL_PTR>), Dynamic Copy Gate ($g$), optional Trie/FSM bitmask, 2L Boundary Refiner (conf < 0.75)

Core Mathematical Formulation

  1. Entropy-Gated Dynamic Concept Patching (Module 1): Local causal Shannon entropy $H_t = -\sum_{v} P(w_t = v \mid w_{<t}) \log_2 P(w_t = v \mid w_{<t})$ triggers a new concept boundary whenever $H_t > 2.4\text{ bits}$ (and above sequence mean) or span length reaches $S_{\max} = 8$.
  2. Epistemic Routing & Brier Calibration (Module 2): The detached calibration head $q_\phi\big([H_m ,|, e(\hat{z}m^{(1)}) ,|, \mathcal{H}(\pi_\theta)]\big)$ predicts $\hat{p}{\text{conf}} \in [0, 1]$ under the strictly proper Brier score:
    • $\hat{p}_{\text{conf}} \ge 0.85 \implies$ Fast Path ($K^* \in {1, 2}$)
    • $0.40 \le \hat{p}{\text{conf}} < 0.85 \implies$ Deliberative Path (top-$4$ beams scored by $\hat{p}{\text{conf}} \cdot V_{\text{verify}}$, $K^* = 8$)
    • $\hat{p}_{\text{conf}} < 0.40 \implies$ *Epistemic Abstention** (<ABSTAIN_OR_RETRIEVE>, $K^ = 0$)
  3. Anchor-Injected Banach Fixed-Point Loops (Module 3): For loop iteration $k \in {1, \dots, 8}$: $$H^{(k)} = f_{\text{loop}}\Big(\text{RMSNorm}\big(H^{(k-1)} + W_{\text{inj}} U_0\big),; H_{\text{core}}\Big)$$ regularized by the Banach contraction penalty $\mathcal{L}_{\text{contract}} = \sum_k \max\big(0,, \Delta^{(k)} - \rho \cdot \text{sg}[\Delta^{(k-1)}]\big)$ ($\rho = 0.7$), reaching empirical contraction violation 0.0000.
  4. Dual-Path Pointer + Concept Surface Decoding (Module 4): $$P(w_i) = (1 - g_i),P_{\text{concept}}(w_i \mid E_{\text{quant}}) + g_i,P_{\text{copy}}\big(w_i \mid H^{(K^*)}, K_{\text{raw}}, V_{\text{raw}}\big)$$

Quickstart Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "entity2260/cl-pncm-2.8b"

tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
model.eval()

prompt = "the government announced that the new economic plan will"
inputs = tokenizer(prompt, add_special_tokens=False, return_tensors="pt").to(model.device)

with torch.no_grad():
    output_ids = model.generate(
        **inputs,
        max_new_tokens=8,
        temperature=0.7,
        top_k=10,
        repetition_penalty=8.0,
    )

print(tokenizer.decode(output_ids[0], skip_special_tokens=True))

Note: Because C-LPNCM uses a custom pure-Transformer multi-module architecture (CLPNCMForCausalLM), pass trust_remote_code=True when loading via transformers.


Training Curriculum & Verified Benchmarks

Trained across the 3-Stage C-LPNCM Curriculum on an NVIDIA A100-SXM4-80GB (bfloat16 + TF32 + Fused AdamW):

Curriculum Stage Active Params Steps / Budget Key Verified Metrics Final Validation / Eval
Stage 1: Multiscale RVQ-FSQ + Pointer Autoencoding 786,152,700 (786M) 2,000 steps (131.1M tokens) Total Loss: 1.9731
Surface CE Loss: 1.4390
VQ Commitment: 0.0002
Pointer Copy Loss: 0.0000
Val Loss: 1.9649
Stage 2: Coarse Latent Core + 8-Loop Equilibrium 2,278,851,275 (2.28B) 4,000 steps (131.1M tokens) Total Loss: 14.0389
Coarse Concept CE: 7.0876
Span Length CE: 1.1000
FSQ Deep Supervision: 5.4191
Banach Contraction: 0.0000
Val Loss: 11.7565
(2.65 pts below train)
Stage 3: Calibrated GRPO RL & Epistemic Alignment 2,278,851,275 (2.28B) 500 steps ($G=4$ rollouts) Policy Gradient Loss: -0.2656 β†’ 0.0000
Brier Calibration Error: 0.0216
Mean Equilibrium Loops: 1.12 – 1.37
OOD / Gibberish Abstention: 100%

Hardware & Runtime Efficiency

Metric NVIDIA A100-80GB (Training) NVIDIA L4-24GB (Inference) NVIDIA T4-16GB (Inference)
Precision bfloat16 + TF32 bfloat16 float16
Peak VRAM Allocated 71.3 GB / 79.3 GB 5.88 GB / 24.0 GB ~6.1 GB / 15.0 GB
Adaptive Loops Run ($K^$)* 1 – 8 (Deep Supervision) 1 – 7 dynamic (mean ~1.8) 1 – 7 dynamic (mean ~1.8)

Verified Sample Generations & Adaptive Loop Traces

Input Prompt Generated Next-Concept Continuation Predicted Coarse Concepts ($\hat{z}^{(1)}$) Adaptive Equilibrium Loops ($K^*$)
the government announced that the new economic plan will fund would no longer be considered as a 8 distinct coarse concepts [1, 1, 1, 1, 2, 1, 1, 3]
the government announced that the new economic plan will to backed company was in process of selling 8 distinct coarse concepts [1, 2, 1, 1, 4, 1, 5, 1]
... the same day as the filing deadline for the senate seat. like a new study published based [3491, 1044, 531, 7484, 6048, 4074] [1, 2, 1, 1, 1, 3]
... but she is not the only one to [8429, 4891, 7102, 4317, 6081, 11979] [2, 1, 7, 3, 2, 1]

Repository Contents

File Description
assets/a_nseg_llm_style_architecture.svg Verified LLM-style Three-Tower architecture diagram
banner.gif Animated cover banner
config.json Model hyperparameters and AutoConfig / AutoModelForCausalLM registration
generation_config.json Default sampling, temperature, top-k, and repetition penalty settings
modeling_cl_pncm.py Self-contained PyTorch implementation (trust_remote_code=True)
model-00001-of-00002.safetensors & model-00002-of-00002.safetensors Sharded bfloat16 model weights (5.38 GB total)
model.safetensors.index.json Sharded weight map and byte-size metadata
tokenizer.json, tokenizer_config.json Fast tokenizer configuration and vocabulary

Citation

If you use this model or build upon Next-Concept Modeling, please cite the foundational Next-Concept Modeling (NCM) paper and this checkpoint:

Next-Concept Modeling (NCM) Paper

@misc{kumbong2026ncm,
  title         = {Next-Concept Modeling: Fast, Parallel Coarse-to-Fine Text Generation},
  author        = {Hermann Kumbong and Ishan S. Khare and Iddah Mlauzi and Andrew Woen and Charles Ding and E. Kelly Buchanan and Simran Arora and Azalia Mirhoseini and Christopher R{\'e}},
  year          = {2026},
  eprint        = {2610.14162},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/2610.14162}
}

C-LPNCM-2.8B Model

@misc{entity2260_clpncm_2_8b_2026,
  title         = {C-LPNCM-2.8B: Calibrated Looped-Pointer Next-Concept Model},
  author        = {Ranjith Kumar (entity2260)},
  year          = {2026},
  publisher     = {Hugging Face},
  howpublished  = {\url{https://huggingface.co/entity2260/cl-pncm-2.8b}}
}

License

This model and its accompanying code are released under the MIT License.

Downloads last month
251
Safetensors
Model size
3B params
Tensor type
BF16
Β·
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Collection including entity2260/cl-pncm-2.8b

Paper for entity2260/cl-pncm-2.8b