Yunmo-Next-1B-Base

Yunmo-Next-1B-Base is a 1.008B-parameter decoder-only language model trained from scratch on 19.76B tokens. It is Traditional-Chinese-first, with English, code and mathematics in the pretraining mixture. The model interleaves one sliding-window/global grouped-query attention layer with three Kimi Delta Attention (KDA) linear-recurrent layers, routes information across depth with low-rank Attention Residuals, and was grown in depth from 24 to 60 layers during training.

This is a raw pretrained model. It has not been instruction-tuned or aligned, and it is not a chat assistant. The instruction-tuned model built from this checkpoint is Yunmo-Next-1B-Instruct.

• Weights, tokenizer and inference code: Apache-2.0
• Tokenizer: yuhuanstudio/Yunmo-Next-Tokenizer
• Technical reports (architecture, training, data, tokenizer, evaluation): YuhuanStudio/Yunmo-Next-1B
• Training data is not redistributed.

At a glance

Parameters	1,008,058,231 (embedding tied with the output head)
Layers / width	60 layers, hidden size 1,024
Token mixers	15 GQA layers + 45 KDA layers, repeating GQA → KDA → KDA → KDA
Vocabulary	65,536 (65,280 text IDs + 256 protocol IDs)
Training tokens	19,761,135,616
Training context	8K → 32K → 16K by stage; validated inference length 16K
Precision	trained in BF16; model.safetensors holds the BF16 training weights bit for bit, model.fp32.safetensors their exact FP32 upcast
Hardware	one GPU at a time: RTX 4070 Ti 12 GB for ~10.3B tokens (K24, K36, end of the capability stage, extension, anneal); a rented RTX 5090 for ~9.4B (K48, K60 16K, start of the capability stage)

Architecture

Hybrid cell

Every block of four layers is one cell:

GQA → KDA → KDA → KDA

Each layer is pre-norm (RMSNorm) with a SwiGLU feed-forward network.

Grouped-query attention. 8 query heads and 2 key/value heads of dimension 128, with RMSNorm on queries and keys and an elementwise sigmoid output gate (o_proj(attn ⊙ σ(gate)), the gate projected from the same input as the queries). Ten of the fifteen GQA layers use a causal sliding window of 2,048 tokens; every third GQA layer (layers 8, 20, 32, 44, 56) attends globally. Rotary position embedding is applied to 32 of the 128 dimensions of each head (25% partial RoPE) with θ = 10⁷.

Kimi Delta Attention. The KDA layers use the gated delta-rule recurrence of Kimi Linear (flash-linear-attention implementation, chunked mode) with 8 heads, key and value dimension 128, and a width-4 short convolution. KDA layers receive no explicit position signal. At inference time each KDA layer carries a fixed-size state that does not grow with sequence length; only the 15 GQA layers keep a KV cache, and 10 of those are capped at a 2,048-token window.

Depth routing: low-rank Attention Residuals with Cell-Echo

The residual stream is replaced by Block Attention Residuals. Sublayers are grouped into blocks of 6; before each sublayer reads its input, it attends with a learned query over the token embedding, the output of every completed block, and the running sum of the current block. To keep memory and compute small, keys are the RMS-normalized last 64 channels of each source (rank 64), while values remain the full 1,024-dimensional source. Block keys are materialized once per block in the forward pass and recomputed blockwise in the backward pass.

In each four-layer cell, the read that feeds the last layer's mixer also has a zero-initialized scalar bias on the completed block holding that cell's GQA output (Cell-Echo; 15 scalars in total). It changes one routing logit before the softmax and nothing else.

Cell-phase SwiGLU widths

Feed-forward width varies with a layer's position r ∈ {0,1,2,3} inside its cell:

w_r = round_16( d · (1 + ½ cos(π r / 3)) ),   d = 3,648
    = [5472, 4560, 2736, 1824]

The four widths sum to exactly 4d, so each cell has the same parameters and MLP FLOPs as four uniform layers of width 3,648. The widest MLP sits on the GQA layer. Because depth grows in steps of 12 layers (three whole cells), every existing layer keeps its phase and tensor shapes when the model grows.

Training

Depth-grown route

Training started from a 24-layer model and inserted 12 layers at each growth step. Existing layers and their optimizer state were copied unchanged; the 12 new layers were initialized from the first three cells of the source model (for K48→K60, target layers come from source layers [0..47] + [0..11]). This mapping does not preserve the source function, so the loss rises immediately after growth and is recovered by training.

All numbers below use the same 125-document, 16-stratum held-out set (1,023,044 scored tokens), scored at 8K context as bits per UTF-8 byte (lower is better).

Stage	Layers	Parameters	Context	Tokens in stage	Cumulative	Held-out BPB
K24	24	443,489,302	8,192	2.000B	2.000B	0.7956
K36	36	631,678,945	8,192	2.000B	4.000B	0.7589
K48	48	819,868,588	8,192	2.500B	6.500B	0.7385
K48 long context	48	819,868,588	32,768	0.500B	7.000B	0.8339
K60	60	1,008,058,231	16,384	5.000B	12.000B	0.7190
K60 capability mixture	60	1,008,058,231	16,384	2.400B	14.400B	0.7160
K60 extension	60	1,008,058,231	16,384	4.161B	18.561B	0.6860
K60 anneal (this checkpoint)	60	1,008,058,231	16,384	1.200B	19.761B	0.6706

The 32K continuation of the 48-layer model improved synthetic retrieval signals but raised short-context BPB by 12.9%; the following 60-layer 16K stage recovered it and ended 2.6% below the 48-layer 8K checkpoint.

The extension and anneal stages draw from one 16K data pack of 3.378B tokens. The extension consumed 4.161B tokens, so its second pass over the pack began at 17.78B cumulative tokens, where training loss drops by about 0.1 nats; the anneal drew a further 1.200B tokens from the same pack. In total the released checkpoint has seen it about 1.59 times (4.161B tokens in the extension plus 1.200B in the anneal up to this checkpoint). Held-out BPB improved through the extension (0.716 → 0.686), but the training-loss drop at the start of the second pass reflects repeated data and is not evidence of that improvement.

Why this checkpoint

The anneal ran to 20.16B tokens (held-out BPB 0.6654). We release the 19.76B checkpoint from the same anneal instead. Both checkpoints received an identical short instruction-tuning probe (21 updates, 66,430 supervised tokens), and the 19.76B parent scored higher afterwards on MMLU gold-answer log-likelihood, DRCD and long-context (InfiniteBench) likelihood, while the 20.16B checkpoint was better on broad held-out BPB (0.8%) and GSM8K answer likelihood (1.4%). The 19.76B checkpoint is also the parent of the instruction-tuned model.

Objective, batching and packing

Causal next-token cross-entropy. Every optimizer update covers 65,536 packed input tokens; micro-batch and gradient-accumulation counts change with context length but the update size never does. An early batch screen at 24 layers compared 65,536, 262,144 and 1,048,576 tokens per update: larger batches gained only 3.6–4.6% throughput and reached worse held-out BPB at equal tokens, so 65,536 was used throughout.

Documents are packed into fixed-length rows with explicit boundaries. Attention masks, KDA recurrent state and position IDs all reset at document boundaries, and cross-document next-token targets are excluded from the loss.

Optimizer and schedule

Parameters are split by role:

• SwiGLU up/gate projections: U-NorMuon;
• KDA query/key/value projections: Muon with Newton–Schulz orthogonalization done independently per recurrent head;
• other eligible 2-D matrices: Muon with reliability-scaled momentum;
• embeddings, norms, biases and scalars: AdamW.

Muon and AdamW share one base learning rate: 3×10⁻⁴ through the capability stage, with a single 160M-token warmup at the start of K24 only. Muon matrices use weight decay 0.1; AdamW parameters use none. The extension stage ran at a constant 1×10⁻⁴, and the anneal decayed from 1×10⁻⁴ with a cosine schedule (this checkpoint is part-way through that decay).

Precision and memory

Weights, activations and the main optimizer states are BF16. Losses, normalizations and reductions run in FP32; there are no FP32 master weights. Updates smaller than the BF16 spacing are accumulated with FP16 Kahan compensation. Activations are recomputed per layer, and the depth router rebuilds its intermediate values per block. For the 32K stage, the MLP was executed in sequence chunks and Muon state was streamed between GPU and pinned CPU memory. The 60-layer model trains at 16K context in about 9.8 GB (peak reserved) on a 12 GB RTX 4070 Ti at a median 3,920 tokens/s during the extension stage (29.6% MFU, lower bound from 6N FLOPs per token).

Pretraining data

The data is not released; this section describes it.

• Language policy. Chinese text is predominantly Traditional. Simplified Chinese sources are converted with OpenCC (s2twp, which also maps vocabulary to the Taiwan standard, e.g. 軟件 → 軟體), with code, URLs and other machine text protected from conversion. A small share of original Simplified text (about 0.6% of the late-stage pack) is kept for reading.
• Sources. Natural web text, educational text including synthetic rewrites, Traditional-Chinese Wikipedia (yuhuanstudio/wikipedia-pretrain-zh-tw), mathematics, code (source files, documentation, configuration and notebooks), classical Chinese texts and public government documents.
• Domain mix of the main 8K and 16K stages (share of tokens): general web 40.2%, general knowledge/education 24.9%, code and computer science 10.4%, mathematics and physical science 10.3%, books and humanities 5.5%, education 4.1%, encyclopedia 4.1%, law/economics/society 0.5%. The 32K stage shifted toward documents long enough to fill the window (general web 64.2%, books 18.6%).
• Late-stage language mix. The extension/anneal pack is 58.0% Chinese, 31.3% English and 9.7% code by document-level language identification.
• Hygiene. Exact content-hash deduplication, re-deduplication after Traditional conversion, removal of known-bad and templated text, and substring checks against evaluation sets. These do not rule out paraphrased or otherwise semantic contamination.
• Licenses of training sources. Sources include ODC-By web corpora, CC BY-SA Wikipedia/Wikisource text, Taiwan Open Government Data, Apache-2.0 datasets, code from The Stack and Kaggle, and two sources with additional terms: OpenCSG FineWeb-Edu-zh (its community license requires OpenCSG approval for commercial use of the dataset) and the MiniMind pretraining corpus (mixed upstream licenses). The full table is in the data report.

Tokenizer

65,536-entry byte-level BPE trained in two phases (word-bounded merges, then up to 5,000 cross-boundary "superword" merges), with byte fallback, no UNK and single-digit splitting. Among 18 tokenizers measured on 12 domains, it needs the fewest tokens on 9, including every Traditional Chinese domain except patents (second), English and mathematics; it ranks 9th on code. Those domains come from the same source families as its training corpus, which favors it. See the tokenizer card for the corpus, method and comparisons.

Evaluation

All results are for this base checkpoint without instruction tuning.

Comparison with 2026 open base models

lm-evaluation-harness 0.4.13, identical tasks and settings for every model; 0-shot except MMLU (5-shot); normalized accuracy for ARC, HellaSwag, PIQA, OpenBookQA and SciQ. Comparison models run in BF16 through transformers; Yunmo-Next runs in FP32 through a small adapter over this package (see the evaluation report for why).

Task	Yunmo-Next 1B	Qwen3.5 0.8B	LFM2.5 1.2B	MiniCPM5 1B	Qwen3.5 2B	MiniCPM5 2B
Pretraining tokens	0.0198T	not stated	28T	not stated	not stated	not stated
ARC-Easy	50.3	67.2	77.7	65.6	71.4	74.7
ARC-Challenge	29.2	40.1	51.3	37.3	45.9	50.5
HellaSwag	41.3	54.9	59.2	56.8	66.0	67.3
PIQA	65.9	71.8	73.3	72.7	74.8	77.0
Winogrande	56.1	60.2	58.7	58.2	64.6	63.5
OpenBookQA	34.4	35.4	39.0	33.8	37.6	41.8
SciQ	85.2	91.7	93.3	92.3	93.2	93.9
LAMBADA	44.5	50.9	46.6	55.4	58.4	61.2
MMLU (5-shot)	22.9	48.4	55.0	43.5	54.0	65.3
TMMLU+	25.0	31.2	31.2	28.4	42.1	44.1
XStoryCloze-zh	57.5	58.3	58.5	58.0	62.3	63.6
XCOPA-zh	61.8	63.4	60.8	62.4	67.8	67.0
XWinograd-zh	67.5	68.8	63.1	71.6	77.4	74.6

Yunmo-Next was trained on roughly three orders of magnitude fewer tokens than these models. It trails on English reasoning and knowledge, is at chance on letter-scored multiple choice (MMLU, TMMLU+), and on the three Chinese cloze tasks is within about 1–4 points of the 1B models (ahead of LFM2.5-1.2B on XCOPA and XWinograd) and 4–10 points behind the 2B models. Differences of 2–3 points on the 500-item sets are within noise.

Internal evaluations

Benchmark	Protocol	Result
TMMLU+ (20,160 questions, 67 subjects)	5-shot, next-token letter A–D, subject macro-average	25.1%
MMLU (14,042 questions, 57 subjects)	same	24.2%
DRCD (3,524)	bits per byte of the gold answer	0.445
GSM8K (1,319)	bits per byte of the gold solution	0.633

Letter-scored multiple choice is at the 25% chance level in this internal 5-shot protocol as well (ties, broken A→B→C→D, occur on 2,481 TMMLU+ and 2,096 MMLU questions).

Long context

Probe	8K	16K	32K
Held-out BPB	0.671	0.637	—
MQAR accuracy (synthetic)	40.6%	6.3%	0%
Multi-needle retrieval, 4 needles	25%	0%	0%

Retrieval degrades sharply beyond 8K and fails at 32K. Answer likelihoods on InfiniteBench documents from 64K to 256K (24 items, 59 variants) were measured as a diagnostic only and do not indicate usable capability at those lengths.

Usage

transformers

The architecture is not part of transformers; the repository ships it as remote code.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "yuhuanstudio/Yunmo-Next-1B-Base"
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    repo, trust_remote_code=True, dtype=torch.float32
).cuda().eval()

ids = tokenizer("臺灣最高的山是", return_tensors="pt").input_ids.cuda()
print(tokenizer.decode(model.generate(ids, max_new_tokens=64)[0], skip_special_tokens=True))

The tokenizer prepends <|bos|>, as in training. forward returns logits (and loss with labels) for right-padded batches; FP32 logits are bit-identical to the package runtime's. generate is a batch-1 continuation loop (greedy or temperature/top-p) with CUDA-graph decoding, not transformers.GenerationMixin. Requires transformers > 4.53.3.

Package CLI

The repository also ships the reference runtime (code/yunmo_next, bin/yunmo-next).

git lfs install
git clone https://huggingface.co/yuhuanstudio/Yunmo-Next-1B-Base
cd Yunmo-Next-1B-Base
python bin/yunmo-next --mode completion --prompt "臺灣最高的山是" \
  --max-new-tokens 128 --device cuda --dtype float32 --static-decode

• --static-decode replays each decoding step as one CUDA graph: about 66 tokens/s instead of 9 on an RTX 4070 Ti in FP32, with logits within 3×10⁻⁵ of regular decoding.

• Use --dtype float32 when scores or exact outputs matter. In BF16 a short answer's log-likelihood moved by up to 0.15 nats with batch padding width alone (0.007 in FP32); FP16, the CLI default, has not been measured.

• Use --mode completion for this base model. The default chat mode is for the instruction-tuned model.

• Requirements: PyTorch with CUDA, tokenizers, safetensors, flash-linear-attention with Triton for the KDA kernels, and transformers > 4.53.3 (older versions give flash-linear-attention a cache that misplaces KDA states during generation; the runtime refuses them).

• Prompt plus generation is limited to 16,384 tokens by default; --allow-unvalidated-long-context lifts the limit without implying the model works at longer lengths.

• On load, every runtime file is checked against release-manifest.json (size and SHA-256).

Limitations

• Not instruction-tuned or aligned: it continues text and does not reliably follow instructions or hold a conversation. The tokenizer reserves IDs for chat roles, reasoning channels and tool calls, but this checkpoint has not been trained on those formats.
• Generations frequently contain factual errors, repetition and unterminated text.
• Multiple-choice knowledge benchmarks are at chance level; do not use it where correctness matters, including legal, medical or financial questions.
• No safety tuning.
• Long-context retrieval is weak beyond 8K and fails at 32K, even though cached inference can run longer inputs.

License

The weights, tokenizer and bundled inference code are released under Apache-2.0. The training data is not distributed; each source remains under its own terms (see Pretraining data above). If you plan commercial use, review those terms yourself; this card is not legal advice.