Next-Concept Modeling (NCM v2) — 100M SFT

Official model weights and configuration for Next-Concept Modeling (NCM v2): a hierarchical language model architecture combining Residual Vector Quantization (RVQ-4) in concept space with Semi-Autoregressive Surface Realization.

Codebase & Full Implementation: GitHub (Kanz4kii/Next-Concept-Modeling)
Architecture: 5-layer ConceptPlanner ($D_c=384$) + 5-layer SurfaceDecoder ($D=768$)
Total Active Parameters: 101.88M
Compression Rate: $r = 4$ tokens per concept (16x high-level self-attention FLOP reduction)
Pretraining Data: 1 Billion tokens of HuggingFaceFW/fineweb-edu
Instruction Tuning: 10,000 dialogues from yahma/alpaca-cleaned

Benchmark: NCM v2 vs. Standard Transformer (GPT-2 124M)

Measured on a consumer NVIDIA GeForce RTX 5050 (8GB VRAM) with PyTorch bfloat16 and FlashAttention SDPA:

Metric / Regime	Standard Transformer (GPT-2)	NCM v2 (Hierarchical)	NCM Advantage
Model Parameters	124.48M (12 layers, $D=768$)	101.88M (5 planner + 5 decoder)	18.2% leaner
Train Throughput ($B=4, T=1024$)	6,667 tok/s (`7,055 MB`)	23,359 tok/s (`3,233 MB`)	`3.50x` faster & `2.18x` less VRAM
Train Throughput ($B=8, T=1024$)	455 tok/s (`13,063 MB` — RAM paging)	27,652 tok/s (`5,583 MB`)	`60.8x` faster (zero VRAM spillover)
Single-Stream Inference ($B=1, 128 ext{ tok}$)	67.1 tok/s (`1,908 ms`)	122.4 tok/s (`1,045 ms`)	`1.82x` faster latency
High-Batch Inference ($B=64, 8192 ext{ tok}$)	3,840.2 tok/s	7,357.2 tok/s (`1,113 ms`)	`1.92x` higher throughput
High-Level Attention Complexity	$O(T^2) = 1,048,576$ pairs	$O((T/4)^2) = 65,536$ pairs	`16x` fewer attention pairs

Quick Start & Inference

Clone the official repository to run one-line inference with automatic checkpoint resolution:

git clone https://github.com/Kanz4kii/Next-Concept-Modeling.git
cd Next-Concept-Modeling
pip install -r requirements.txt

Interactive Chat Mode

python generate.py \
  --hf-repo Kanz4kii/Next-Concept-Modeling \
  --chat \
  --prompt "What are three key principles of effective software engineering?" \
  --max-new-tokens 128 \
  --token-temperature 0.5 \
  --min-p 0.05 \
  --repetition-penalty 1.25

Python API

import torch
from model import NCMModel
from interfaces import ModelArgs

# Loads model directly from Hugging Face Hub checkpoint
model = NCMModel.from_safetensors("model.safetensors", args=ModelArgs.from_json("config.json"))
model.to("cuda" if torch.cuda.is_available() else "cpu")

Citation

@software{{ncm_v2_2026,
  author = {{Kanz4kii}},
  title = {{Next-Concept Modeling (NCM v2): Semi-Autoregressive Hierarchical Language Modeling}},
  year = {{2026}},
  publisher = {{Hugging Face & GitHub}},
  url = {{https://huggingface.co/Kanz4kii/Next-Concept-Modeling}}
}}

Downloads last month: 31

Safetensors

Model size

0.1B params

Tensor type

F32

Kanz4kii
/

Next-Concept-Modeling