---
language:
- en
- zh
- es
- fr
- ar
- ru
license: apache-2.0
library_name: transformers
pipeline_tag: text-generation
tags:
- qwen3
- causal-lm
- reinforcement-learning
- ppo
- rlvr
- math
- reasoning
- multilingual
- multilingual-reasoning
- chain-of-thought
- verl
base_model: 96kevinli29/Qwen3-4B-SFT-Math-Multilingual
datasets:
- 96kevinli29/DAPO-Math-Multilingual-6Lang
model-index:
- name: Qwen3-4B-MathRL-Multilingual-Additive
results:
- task:
type: text-generation
dataset:
name: AIME-2026 + AMC-2023, five non-English languages (zh, es, fr, ar, ru)
type: 96kevinli29/Multilingual-Math-Reasoning-Eval
metrics:
- name: Delivered J@1 (correct, in the requested language, terminated)
type: accuracy
value: 59.8
- name: Answer accuracy
type: accuracy
value: 60.1
- name: Reasoning in the requested language
type: accuracy
value: 99.6
---
Mean over Chinese, Spanish, French, Arabic and Russian · 70 problems × 16 samples per language
- **What it is.** [Qwen3-4B-SFT-Math-Multilingual](https://huggingface.co/96kevinli29/Qwen3-4B-SFT-Math-Multilingual) after 200 PPO steps on localized math prompts with the **additive reward `R = 0.7 · C + 0.3 · ℓ`**: correctness `C` plus a language score `ℓ` that is paid whether or not the answer is right.
- **What it fixes.** The SFT model reasons in the right language but 21.1% of its non-English responses never finish. Here that is **1.2%**, the reasoning stays in the requested language (99.6%), and the median response shrinks from 7,673 to 4,072 tokens.
- **What to know.** It is one of two matched runs that differ only in reward; [Qwen3-4B-MathRL-Multilingual-Gated](https://huggingface.co/96kevinli29/Qwen3-4B-MathRL-Multilingual-Gated) (gated reward) performs comparably. A third run rewarded for correctness alone also stops the looping but drifts back to English.
## Results
| | Released | SFT | **RL additive** | RL gated |
| :--- | :---: | :---: | :---: | :---: |
| Answer correct | 76.2% | 61.4% | **60.1%** | 61.7% |
| Reasoning in the requested language | 39.8% | 99.4% | **99.6%** | 99.7% |
| Did not finish | 4.7% | 21.1% | **1.2%** | 1.8% |
| **Delivered** (J@1) | 28.6% | 60.2% | **59.8%** | 61.4% |
| **Delivered**, best of 16 (J@16) | 35.7% | 84.6% | **83.1%** | 83.7% |
| Median response, tokens | 5,824 | 7,673 | **4,072** | 4,463 |
70 competition problems (30 AIME-2026 + 40 AMC-2023) per language from [Multilingual-Math-Reasoning-Eval](https://huggingface.co/datasets/96kevinli29/Multilingual-Math-Reasoning-Eval), 16 samples each at temperature 0.7, 24,576-token budget. A response is **delivered** when the answer is correct, the visible reasoning is in the requested language, and it terminates (`J = C ∧ L ∧ T`, scored per response). Tables show the mean over Chinese, Spanish, French, Arabic and Russian.
Per language · other benchmarks
| | Correct | In language | Did not finish | J@1 | J@16 | Median tokens |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: |
| English | 65.7% | 100.0% | 0.2% | 65.7% | 85.7% | 3,510 |
| Chinese | 58.7% | 99.7% | 0.6% | 58.5% | 81.4% | 3,596 |
| Spanish | 61.5% | 99.8% | 0.8% | 61.4% | 81.4% | 3,856 |
| French | 61.3% | 99.8% | 0.3% | 61.3% | 85.7% | 3,818 |
| Arabic | 57.3% | 98.6% | 2.1% | 56.1% | 82.9% | 4,566 |
| Russian | 61.7% | 99.9% | 2.0% | 61.7% | 84.3% | 4,524 |
| Delivered (J), non-English mean | Released | SFT | **RL additive** | RL gated |
| :--- | :---: | :---: | :---: | :---: |
| PolyMath, 444 problems | 16.8% | 34.1% | **35.4%** | 36.8% |
| MMLU-ProX-Lite, 588 questions | 24.0% | 49.0% | **56.2%** | 55.5% |
## Quick start
```python
from datasets import load_dataset
from transformers import AutoModelForCausalLM, AutoTokenizer
name = "96kevinli29/Qwen3-4B-MathRL-Multilingual-Additive"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForCausalLM.from_pretrained(name, torch_dtype="auto", device_map="auto")
# a French problem with a French instruction: "…en raisonnant entièrement en français."
prompt = load_dataset("96kevinli29/Multilingual-Math-Reasoning-Eval", "fr")["amc23"][0]["prompt"]
text = tok.apply_chat_template([{"role": "user", "content": prompt}], tokenize=False, add_generation_prompt=True)
inputs = tok(text, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=24576, do_sample=True, temperature=0.7)
print(tok.decode(out[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))
```
Qwen3 chat template with thinking on; the reasoning appears inside ` … `. Give hard problems a large `max_new_tokens`. Loads with transformers 4.x and 5.x.
Prompt format in six languages
| | Instruction that opens the user turn |
| :--- | :--- |
| English | Solve the following math problem step by step, reasoning entirely in English. |
| Chinese | 请全程用中文逐步推理并解决以下数学问题。 |
| Spanish | Resuelve el siguiente problema matemático paso a paso, razonando enteramente en español. |
| French | Résous le problème de mathématiques suivant étape par étape, en raisonnant entièrement en français. |
| Arabic | حل المسألة الرياضية التالية خطوة بخطوة مع التفكير بالكامل باللغة العربية. |
| Russian | Решите следующую математическую задачу шаг за шагом, рассуждая полностью на русском языке. |
The full prompts (instruction, problem and answer format, all in the target language) are in the evaluation set.
## Model family
| Stage | Checkpoint | |
| :--- | :--- | :--- |
| I · Release | [Qwen/Qwen3-4B](https://huggingface.co/Qwen/Qwen3-4B) · [Qwen/Qwen3-8B](https://huggingface.co/Qwen/Qwen3-8B) | solves the problem, reasons in English |
| II · SFT | [Qwen3-4B-SFT-Math-Multilingual](https://huggingface.co/96kevinli29/Qwen3-4B-SFT-Math-Multilingual) | reasons in the requested language, sometimes loops |
| II · SFT | [Qwen3-8B-SFT-Math-Multilingual](https://huggingface.co/96kevinli29/Qwen3-8B-SFT-Math-Multilingual) | the same result at 8B |
| III · RL | [Qwen3-4B-MathRL-Multilingual-Gated](https://huggingface.co/96kevinli29/Qwen3-4B-MathRL-Multilingual-Gated) | keeps the language, stops the looping |
| III · RL | [Qwen3-4B-MathRL-Multilingual-Additive](https://huggingface.co/96kevinli29/Qwen3-4B-MathRL-Multilingual-Additive) ← this model | keeps the language, stops the looping |
Data: [SFT-Math-Multilingual-43k](https://huggingface.co/datasets/96kevinli29/SFT-Math-Multilingual-43k) · [SFT-Math-Multilingual-56k](https://huggingface.co/datasets/96kevinli29/SFT-Math-Multilingual-56k) · [DAPO-Math-Multilingual-6Lang](https://huggingface.co/datasets/96kevinli29/DAPO-Math-Multilingual-6Lang) · [Multilingual-Math-Reasoning-Eval](https://huggingface.co/datasets/96kevinli29/Multilingual-Math-Reasoning-Eval)
## Training
| | |
| :--- | :--- |
| Algorithm | PPO, GAE with γ = λ = 1, clip ratio 0.2, no KL term |
| Reward | `0.7 · C + 0.3 · ℓ`, plus a soft penalty for overlong responses |
| Prompts | [DAPO-Math-Multilingual-6Lang](https://huggingface.co/datasets/96kevinli29/DAPO-Math-Multilingual-6Lang), 16,754 localized DAPO-Math-17K prompts |
| Batch | 32 prompts × 8 rollouts per step, 16,384-token response cap |
| Learning rate | 1e-6 actor, 1e-5 critic |
| Steps | 200 |
| Hardware · framework | 8 GPUs on one node · [verl](https://github.com/volcengine/verl), FSDP |
`C` is rule-based correctness of the final answer; `ℓ ∈ [0, 1]` is a language-identification score of the response against the language of the prompt.
## Limitations
- **Accuracy** stays below the released Qwen3-4B on the same problems (76.2% → 60.1%). The cost enters with SFT; RL does not recover it, and it is now what limits delivery.
- 200 PPO steps, one seed, one model size.
- Mathematics and six languages only.
## Citation
```bibtex
@article{li2026posttraining,
title = {What Does Post-Training Change in Multilingual Reasoning?},
author = {Li, Hongyang and Li, Xiao and Wu, Caesar and Danoy, Gr{\'e}goire and Bouvry, Pascal},
journal = {arXiv preprint arXiv:2609.37104},
year = {2026}
}
@misc{qwen3-4b-mathrl-multilingual-additive-2026,
title = {{Qwen3-4B-MathRL-Multilingual-Additive}: PPO from multilingual-SFT {Qwen3-4B} with a additive language reward},
author = {Li, Hongyang and Li, Xiao and Wu, Caesar and Danoy, Gr{\'e}goire and Bouvry, Pascal and {Sea-Fill Community}},
year = {2026},
publisher = {Hugging Face},
howpublished = {\url{https://huggingface.co/96kevinli29/Qwen3-4B-MathRL-Multilingual-Additive}},
note = {Maintained by Sea-Fill Community.}
}
```
Built and released by the authors together with the [Sea-Fill Community](https://huggingface.co/SeaFill2025), which maintains this release.
