Qwen3-4B-MathRL-Multilingual-Additive

Qwen3-4B that reasons in the language of the problem, and finishes

_{Mean over Chinese, Spanish, French, Arabic and Russian · 70 problems × 16 samples per language}

• What it is. Qwen3-4B-SFT-Math-Multilingual after 200 PPO steps on localized math prompts with the additive reward R = 0.7 · C + 0.3 · ℓ: correctness C plus a language score ℓ that is paid whether or not the answer is right.
• What it fixes. The SFT model reasons in the right language but 21.1% of its non-English responses never finish. Here that is 1.2%, the reasoning stays in the requested language (99.6%), and the median response shrinks from 7,673 to 4,072 tokens.
• What to know. It is one of two matched runs that differ only in reward; Qwen3-4B-MathRL-Multilingual-Gated (gated reward) performs comparably. A third run rewarded for correctness alone also stops the looping but drifts back to English.

Results

	Released	SFT	RL additive	RL gated
Answer correct	76.2%	61.4%	60.1%	61.7%
Reasoning in the requested language	39.8%	99.4%	99.6%	99.7%
Did not finish	4.7%	21.1%	1.2%	1.8%
Delivered (J@1)	28.6%	60.2%	59.8%	61.4%
Delivered, best of 16 (J@16)	35.7%	84.6%	83.1%	83.7%
Median response, tokens	5,824	7,673	4,072	4,463

70 competition problems (30 AIME-2026 + 40 AMC-2023) per language from Multilingual-Math-Reasoning-Eval, 16 samples each at temperature 0.7, 24,576-token budget. A response is delivered when the answer is correct, the visible reasoning is in the requested language, and it terminates (J = C ∧ L ∧ T, scored per response). Tables show the mean over Chinese, Spanish, French, Arabic and Russian.

Per language · other benchmarks

	Correct	In language	Did not finish	J@1	J@16	Median tokens
English	65.7%	100.0%	0.2%	65.7%	85.7%	3,510
Chinese	58.7%	99.7%	0.6%	58.5%	81.4%	3,596
Spanish	61.5%	99.8%	0.8%	61.4%	81.4%	3,856
French	61.3%	99.8%	0.3%	61.3%	85.7%	3,818
Arabic	57.3%	98.6%	2.1%	56.1%	82.9%	4,566
Russian	61.7%	99.9%	2.0%	61.7%	84.3%	4,524

Delivered (J), non-English mean	Released	SFT	RL additive	RL gated
PolyMath, 444 problems	16.8%	34.1%	35.4%	36.8%
MMLU-ProX-Lite, 588 questions	24.0%	49.0%	56.2%	55.5%

Quick start

from datasets import load_dataset
from transformers import AutoModelForCausalLM, AutoTokenizer

name = "96kevinli29/Qwen3-4B-MathRL-Multilingual-Additive"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForCausalLM.from_pretrained(name, torch_dtype="auto", device_map="auto")

# a French problem with a French instruction: "…en raisonnant entièrement en français."
prompt = load_dataset("96kevinli29/Multilingual-Math-Reasoning-Eval", "fr")["amc23"][0]["prompt"]
text = tok.apply_chat_template([{"role": "user", "content": prompt}], tokenize=False, add_generation_prompt=True)
inputs = tok(text, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=24576, do_sample=True, temperature=0.7)
print(tok.decode(out[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))

Qwen3 chat template with thinking on; the reasoning appears inside <think> … </think>. Give hard problems a large max_new_tokens. Loads with transformers 4.x and 5.x.

Prompt format in six languages

	Instruction that opens the user turn
English	Solve the following math problem step by step, reasoning entirely in English.
Chinese	请全程用中文逐步推理并解决以下数学问题。
Spanish	Resuelve el siguiente problema matemático paso a paso, razonando enteramente en español.
French	Résous le problème de mathématiques suivant étape par étape, en raisonnant entièrement en français.
Arabic	حل المسألة الرياضية التالية خطوة بخطوة مع التفكير بالكامل باللغة العربية.
Russian	Решите следующую математическую задачу шаг за шагом, рассуждая полностью на русском языке.

The full prompts (instruction, problem and answer format, all in the target language) are in the evaluation set.

Model family

Stage	Checkpoint
I · Release	Qwen/Qwen3-4B · Qwen/Qwen3-8B	solves the problem, reasons in English
II · SFT	Qwen3-4B-SFT-Math-Multilingual	reasons in the requested language, sometimes loops
II · SFT	Qwen3-8B-SFT-Math-Multilingual	the same result at 8B
III · RL	Qwen3-4B-MathRL-Multilingual-Gated	keeps the language, stops the looping
III · RL	Qwen3-4B-MathRL-Multilingual-Additive ← this model	keeps the language, stops the looping

Data: SFT-Math-Multilingual-43k · SFT-Math-Multilingual-56k · DAPO-Math-Multilingual-6Lang · Multilingual-Math-Reasoning-Eval

Training

Algorithm	PPO, GAE with γ = λ = 1, clip ratio 0.2, no KL term
Reward	0.7 · C + 0.3 · ℓ, plus a soft penalty for overlong responses
Prompts	DAPO-Math-Multilingual-6Lang, 16,754 localized DAPO-Math-17K prompts
Batch	32 prompts × 8 rollouts per step, 16,384-token response cap
Learning rate	1e-6 actor, 1e-5 critic
Steps	200
Hardware · framework	8 GPUs on one node · verl, FSDP

C is rule-based correctness of the final answer; ℓ ∈ [0, 1] is a language-identification score of the response against the language of the prompt.

Limitations

• Accuracy stays below the released Qwen3-4B on the same problems (76.2% → 60.1%). The cost enters with SFT; RL does not recover it, and it is now what limits delivery.
• 200 PPO steps, one seed, one model size.
• Mathematics and six languages only.

Citation

@article{li2026posttraining,
  title   = {What Does Post-Training Change in Multilingual Reasoning?},
  author  = {Li, Hongyang and Li, Xiao and Wu, Caesar and Danoy, Gr{\'e}goire and Bouvry, Pascal},
  journal = {arXiv preprint arXiv:2609.37104},
  year    = {2026}
}

@misc{qwen3-4b-mathrl-multilingual-additive-2026,
  title        = {{Qwen3-4B-MathRL-Multilingual-Additive}: PPO from multilingual-SFT {Qwen3-4B} with a additive language reward},
  author       = {Li, Hongyang and Li, Xiao and Wu, Caesar and Danoy, Gr{\'e}goire and Bouvry, Pascal and {Sea-Fill Community}},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/96kevinli29/Qwen3-4B-MathRL-Multilingual-Additive}},
  note         = {Maintained by Sea-Fill Community.}
}

Built and released by the authors together with the Sea-Fill Community, which maintains this release.