Concertina-2.27B

This is the main model of Folding Language Models into Themselves. It is built from the weights of Qwen3-4B with Concertina and then distilled from Qwen3-4B on 22B tokens.

Paper: arXiv:2611.18113 ยท Code: github.com/myboker/concertina

Model

Concertina folds every two adjacent layers of Qwen3-4B into one shared block that is executed twice.

  • Shared weight. Each linear map of a shared block starts from the weight of the first layer of its pair.
  • Step corrections. The second loop step adds a rank-64 correction initialized from the truncated SVD of the difference between the two original layers.
  • Per-token gate. A gate scales the correction separately for every token.
  • Per-step norms. Each loop step keeps the RMSNorm weights of the layer it replaces.
Qwen3-4B Concertina-2.27B
Parameters 4.02B 2.27B (57%)
Unique blocks ร— loop steps 36 ร— 1 18 ร— 2
Effective depth 36 36
KV cache same same
Weights (bf16) 8.0 GB 4.5 GB

Results

The scores are from Table 1 of the paper.

Model Params Tokens base-7 MMLU Redux Pro GPQA BBH GSM8K MATH
Qwen3-4B (teacher) 4.02B 36T 0.630 0.671 0.720 0.404 0.379 0.505 0.879 0.566
Qwen3-1.7B 1.72B 36T 0.562 0.557 0.591 0.297 0.293 0.363 0.710 0.460
Concertina-2.27B 2.27B 22B 0.580 0.556 0.587 0.301 0.308 0.400 0.788 0.498

base-7 is the mean of ARC-Easy, ARC-Challenge, HellaSwag, PIQA, OpenBookQA, WinoGrande and LAMBADA. Tokens are pretraining tokens for the Qwen3 models and distillation tokens for Concertina.

Usage

The folded architecture is defined in the files of this repository, so loading the model and the tokenizer needs trust_remote_code=True.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "myboker/Concertina-2.27B"
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True, dtype=torch.bfloat16, device_map="auto")

inputs = tokenizer("The capital of France is", return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=32, do_sample=False)
print(tokenizer.decode(out[0], skip_special_tokens=True))

Versions of transformers older than 4.56 take torch_dtype instead of dtype. Generation with the KV cache enabled is supported, since every loop step keeps the cache slot of the layer it replaces. It is tested with transformers 4.57 and 5.6.

Citation

@article{zhang2026folding,
  title   = {Folding Language Models into Themselves},
  author  = {Zhang, Ziyang and Jing, Yubin and Li, Yuyao and Wang, Haofan and Gong, Yichen},
  journal = {arXiv preprint arXiv:2611.18113},
  year    = {2026}
}

License

Apache License 2.0, the same as Qwen3-4B.

Downloads last month
-
Safetensors
Model size
2B params
Tensor type
BF16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for myboker/Concertina-2.27B

Finetuned
Qwen/Qwen3-4B
Finetuned
(1,276)
this model

Paper for myboker/Concertina-2.27B