🤗 Hugging Face   |   🤖 ModelScope

Ling-3.1-flash-base

Introduction

We introduce Ling-3.1-flash-base, the base checkpoint of Ling-3.1-flash, released to support research and community-driven innovation. With 560B total parameters and 25B active parameters per token, the model provides a foundation for continued pretraining, domain adaptation, and post-training research.

Ling-3.1-flash-base shares the same hybrid-linear Mixture-of-Experts (MoE) architecture as Ling-3.1-flash, combining KDA with Gated MLA. As a base checkpoint, it is intended for further training and adaptation.

The model was trained using FP8. This release provides the BF16 base checkpoint produced after Warmup-Stable and Merge (WSM) checkpoint merging. BF16 refers to the precision of the released weights, while FP8 refers to the precision used during training.

Model Overview

Key Features

• Sparse MoE architecture: 512 routed experts, with 8 routed experts and 1 shared expert activated per token, for a total of 25B active parameters per token.
• Hybrid linear attention: 63 KDA layers and 9 Gated MLA layers combine linear attention with Gated MLA at a 7:1 ratio.

Model Specifications

Specification	Value
Model Type	Base checkpoint
Architecture	Hybrid-linear MoE
Parameter Scale	Total 560B, Activated 25B
Transformer Layers	63 KDA + 9 Gated MLA (7:1)
Number of Dense Layers	4
Number of Routed Experts	512
Number of Shared Experts	1
Number of Activated Routed Experts	8
Attention Heads	64
Hidden Size	4096
Expert Intermediate Size	1280
Dense Intermediate Size	10240
Vocabulary Size	157184

Architecture Diagram

Base Model Evaluation

We evaluate Ling-3.1-flash-base on 47 benchmarks across eight domains: language understanding, knowledge, code, mathematics, reasoning, multilingual capabilities, medical knowledge, and long-context understanding. The results below compare the model with DeepSeek-V4-Pro-Base and MiMo-V2.5-Pro-Base under the shot configurations shown.

Bold indicates the highest score in each row; underlining indicates the second-highest distinct score. Tied scores receive the same formatting. EM denotes exact match, and Acc denotes accuracy. Parameter scales are shown as total parameters and active parameters per token (A).

Domain	Benchmark	Shot Config	Ling-3.1-flash-base	DeepSeek-V4-Pro-Base	MiMo-V2.5-Pro-Base
			560B A25B	1.6T A49B	1.02T A42B
Language Understanding	Race-High(EM)	0-shot	93.88	93.37	93.60
	SquadV2(Acc)	1-shot	92.79	87.70	88.93
	Belebele(EM)	0-shot	94.67	93.83	92.72
	C3(EM)	0-shot	83.34	81.64	81.53
	CCPM(EM)	0-shot	91.88	93.31	90.18
Knowledge	ARC-c(EM)	0-shot	97.97	96.61	96.95
	AGIEval(Acc)	few-shot(0~5)	82.59	81.26	79.90
	SimpleQA-Verified(Acc)	5-shot	52.40	52.10	32.60
	C-SimpleQA(Acc)	5-shot	77.43	77.60	70.93
	AA-Omniscience(Acc)	3-shot	37.67	48.16	36.45
	MMLU-Pro(EM)	5-shot	76.71	75.28	71.26
	CEval(EM)	5-shot	92.36	93.76	92.08
	CMMLU(EM)	5-shot	91.60	90.52	88.33
	GPQA-Diamond(EM)	0-shot	56.57	52.02	54.04
	SuperGPQA(EM)	5-shot	55.84	53.19	48.65
Code	HumanEval-Plus(Pass@1)	0-shot	79.88	57.93	82.93
	CruxEval(Pass@1)	1-shot	80.50	69.06	71.25
	MultiPL-E(Pass@1)	0-shot	70.55	55.94	64.74
	MBPP-Plus(Pass@1)	3-shot	76.19	73.81	77.51
	BigCodeBench(Pass@1)	0-shot	53.60	51.49	66.75
	LCBench1(Pass@1)	3-shot	68.48	59.14	67.58
	LCBench-cn1(Pass@1)	3-shot	60.83	57.35	63.36
Math	MATH500(Acc)	4-shot	79.40	68.20	57.20
	CollegeMath(Acc)	4-shot	73.41	63.15	66.61
	FinanceReasoning(Acc)	3-shot	81.76	82.16	80.89
	OlympiadBench(Acc)	3-shot	60.24	46.39	48.04
	TheoremQA(Acc)	5-shot	63.56	61.92	60.27
	OmniMath(Acc)	3-shot	65.31	34.89	40.13
Reasoning	CommonSenseQA(EM)	5-shot	92.22	86.49	88.53
	BBH(EM)	3-shot	90.25	89.98	89.80
	KorBench(Acc)	3-shot	54.96	54.56	53.20
	AutoLogi(Acc)	3-shot	67.13	64.81	66.23
	ZebraLogic(Acc)	1-shot	30.80	20.30	28.90
Multilingual	mARC(EM)	0-shot	92.98	89.10	88.55
	MMMLU(EM)	0-shot	77.56	68.81	67.15
	MultiGSM(Acc)	4-shot	87.66	85.87	86.33
	HumanEvalXL(Pass@1)	0-shot	83.25	70.31	87.34
Med	DiagnosisArena(EM)	0-shot	62.20	55.50	52.70
	medmcqa(EM)	5-shot	78.08	80.33	77.50
	medqa(EM)	5-shot	87.82	87.75	85.55
Long-context	LongBench(Acc)	0-shot	51.90	17.70	15.66
	LEval(Acc)	0-shot	85.09	58.34	61.49
	ruler_16k(Acc)	1-shot	96.90	97.00	90.61
	ruler_32k(Acc)	1-shot	95.87	95.57	89.10
	ruler_64k(Acc)	1-shot	93.81	95.56	87.19
	ruler_128k(Acc)	1-shot	88.64	95.06	85.79
	ruler_256k(Acc)	1-shot	86.64	94.37	86.00
Note: 1 LCBench and LCBench-cn: January 2023–February 2025 (2301-2502).

Intended Use

Recommended use cases:

• Continued pretraining.
• Mid-training.
• Supervised fine-tuning for domain adaptation.
• Preference optimization and reinforcement-learning post-training.
• Distillation research.
• Long-context and MoE systems research.

Not recommended as-is for:

• Direct end-user chat deployment.
• Safety-critical applications without additional alignment and evaluation.
• Production use without post-training and task-specific validation.

Usage

For fine-tuning examples, please refer to our ling-cookbook.

FAQ

For questions or feedback, please open a discussion in the model repository.

License

This model is released under the MIT License.