NGen-5-20B-A3B

Open-weight release in the NGen-5 series by TNSA. This open series is separate from TNSA's closed-source NGen-5.

Specifications

Total parameters	20.5B
Active parameters per token	~3B
Architecture	Mixture-of-Experts decoder, hybrid attention (3 Gated DeltaNet linear-attention layers per 1 gated full-attention layer)
Layers	40
Hidden size	2048
Experts per layer	136 routed (top-8) + 1 shared
Expert hidden size	512
Context length	262,144 tokens
Precision	FP8 (e4m3, 128x128 block scales); routers, norms, embeddings and output head in BF16
Size on disk	about 22 GB
Config	model_type: ngen5_moe

Benchmarks

Benchmark	NGen-5-20B-A3B
GPQA Diamond (198 questions)	72.7
MMLU-Pro (300-question subset)	73.7

Evaluated by TNSA in thinking mode with a 28K-token budget (temperature 1.0, top-p 0.95, top-k 20, presence penalty 1.5), single run. Differences of about 3 points are within noise. Full results and settings: benchmark_results.json.

How to run

The model code is in modeling_ngen5_moe.py, written for OpenArchX. It loads the FP8 weights in this repository directly.

pip install -U "openarchx[cuda]" numpy tokenizers huggingface_hub
hf download TNSA/NGen-5-20B-A3B --local-dir NGen-5-20B-A3B

import sys
sys.path.insert(0, "NGen-5-20B-A3B")
from modeling_ngen5_moe import NGen5MoeForCausalLM
from tokenizers import Tokenizer

tok = Tokenizer.from_file("NGen-5-20B-A3B/tokenizer.json")
model = NGen5MoeForCausalLM.from_pretrained("NGen-5-20B-A3B")

prompt = "<|im_start|>user\nExplain why the sky is blue.<|im_end|>\n<|im_start|>assistant\n<think>\n"
ids = tok.encode(prompt).ids
out = model.generate([ids], max_new_tokens=512, temperature=1.0, top_k=20)
print(tok.decode(out[0][len(ids):]))

For the full chat format (system prompts, tools, thinking), use chat_template.jinja from this repository. Recommended sampling: temperature 1.0, top-p 0.95, top-k 20, presence penalty 1.5.

Identity

This model is NGen, developed by TNSA, but it may often fail to identify itself correctly. Because of the upstream models it learned from, it may sometimes say that it is Qwen, Kimi, GLM or DeepSeek, or name their developers as its creator. These statements are incorrect.

License

Released under the TNSA open-weight model license. Includes third-party material under Apache-2.0; see NOTICE and LICENSE-APACHE-2.0.