File size: 10,148 Bytes
6875d5f
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
63d7807
6875d5f
63d7807
6875d5f
63d7807
6875d5f
63d7807
6875d5f
63d7807
6875d5f
63d7807
6875d5f
63d7807
6875d5f
63d7807
 
 
6875d5f
63d7807
6875d5f
63d7807
6875d5f
63d7807
 
 
 
 
 
 
 
6875d5f
63d7807
 
6875d5f
63d7807
6875d5f
 
 
 
 
 
 
 
63d7807
 
 
 
6875d5f
63d7807
6875d5f
63d7807
6875d5f
 
 
 
 
 
 
 
 
63d7807
6875d5f
 
 
 
 
 
 
63d7807
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
6875d5f
63d7807
 
 
 
 
 
 
 
 
 
 
6875d5f
 
 
63d7807
6875d5f
63d7807
 
 
 
 
 
 
 
 
6875d5f
 
 
63d7807
 
 
6875d5f
 
 
 
 
 
 
 
 
 
 
 
63d7807
c80273b
6875d5f
 
81d4403
 
6875d5f
 
81d4403
63d7807
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
---
language:
  - en
  - zh
  - es
  - fr
  - ar
  - ru
license: apache-2.0
library_name: transformers
pipeline_tag: text-generation
tags:
  - qwen3
  - causal-lm
  - reinforcement-learning
  - ppo
  - rlvr
  - math
  - reasoning
  - multilingual
  - multilingual-reasoning
  - chain-of-thought
  - verl
base_model: 96kevinli29/Qwen3-4B-SFT-Math-Multilingual
datasets:
  - 96kevinli29/DAPO-Math-Multilingual-6Lang
model-index:
  - name: Qwen3-4B-MathRL-Multilingual-Additive
    results:
      - task:
          type: text-generation
        dataset:
          name: AIME-2026 + AMC-2023, five non-English languages (zh, es, fr, ar, ru)
          type: 96kevinli29/Multilingual-Math-Reasoning-Eval
        metrics:
          - name: Delivered J@1 (correct, in the requested language, terminated)
            type: accuracy
            value: 59.8
          - name: Answer accuracy
            type: accuracy
            value: 60.1
          - name: Reasoning in the requested language
            type: accuracy
            value: 99.6
---

<div align="center">

# Qwen3-4B-MathRL-Multilingual-Additive

**Qwen3-4B that reasons in the language of the problem, and finishes**

<a href="https://arxiv.org/abs/2609.37104"><img alt="arXiv: 2609.37104" src="https://img.shields.io/badge/arXiv-2609.37104-b31b1b"></a> <a href="https://huggingface.co/Qwen/Qwen3-4B"><img alt="base: Qwen3-4B" src="https://img.shields.io/badge/base-Qwen3--4B-475467"></a> <img alt="stage: RL additive" src="https://img.shields.io/badge/stage-RL_additive-1baf7a"> <img alt="languages: 6" src="https://img.shields.io/badge/languages-6-2a78d6"> <img alt="license: Apache 2.0" src="https://img.shields.io/badge/license-Apache_2.0-1baf7a">

</div>

<img src="https://huggingface.co/96kevinli29/Qwen3-4B-MathRL-Multilingual-Additive/resolve/main/assets/lineage_4b.png" alt="Bar chart, non-English mean. Reasoning in the requested language: released 40%, SFT 99%, RL additive 100%, RL gated 100%. Did not finish: 5%, 21%, 1%, 2%. Delivered: 29%, 60%, 60%, 61%." width="100%">

<p align="center"><sub>Mean over Chinese, Spanish, French, Arabic and Russian · 70 problems × 16 samples per language</sub></p>

- **What it is.** [Qwen3-4B-SFT-Math-Multilingual](https://huggingface.co/96kevinli29/Qwen3-4B-SFT-Math-Multilingual) after 200 PPO steps on localized math prompts with the **additive reward `R = 0.7 · C + 0.3 · ℓ`**: correctness `C` plus a language score `ℓ` that is paid whether or not the answer is right.
- **What it fixes.** The SFT model reasons in the right language but 21.1% of its non-English responses never finish. Here that is **1.2%**, the reasoning stays in the requested language (99.6%), and the median response shrinks from 7,673 to 4,072 tokens.
- **What to know.** It is one of two matched runs that differ only in reward; [Qwen3-4B-MathRL-Multilingual-Gated](https://huggingface.co/96kevinli29/Qwen3-4B-MathRL-Multilingual-Gated) (gated reward) performs comparably. A third run rewarded for correctness alone also stops the looping but drifts back to English.

## Results

| | Released | SFT | **RL additive** | RL gated |
| :--- | :---: | :---: | :---: | :---: |
| Answer correct | 76.2% | 61.4% | **60.1%** | 61.7% |
| Reasoning in the requested language | 39.8% | 99.4% | **99.6%** | 99.7% |
| Did not finish | 4.7% | 21.1% | **1.2%** | 1.8% |
| **Delivered** (J@1) | 28.6% | 60.2% | **59.8%** | 61.4% |
| **Delivered**, best of 16 (J@16) | 35.7% | 84.6% | **83.1%** | 83.7% |
| Median response, tokens | 5,824 | 7,673 | **4,072** | 4,463 |

70 competition problems (30 AIME-2026 + 40 AMC-2023) per language from [Multilingual-Math-Reasoning-Eval](https://huggingface.co/datasets/96kevinli29/Multilingual-Math-Reasoning-Eval), 16 samples each at temperature 0.7, 24,576-token budget. A response is **delivered** when the answer is correct, the visible reasoning is in the requested language, and it terminates (`J = C ∧ L ∧ T`, scored per response). Tables show the mean over Chinese, Spanish, French, Arabic and Russian.

<details>
<summary><b>Per language · other benchmarks</b></summary>

| | Correct | In language | Did not finish | J@1 | J@16 | Median tokens |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: |
| English | 65.7% | 100.0% | 0.2% | 65.7% | 85.7% | 3,510 |
| Chinese | 58.7% | 99.7% | 0.6% | 58.5% | 81.4% | 3,596 |
| Spanish | 61.5% | 99.8% | 0.8% | 61.4% | 81.4% | 3,856 |
| French | 61.3% | 99.8% | 0.3% | 61.3% | 85.7% | 3,818 |
| Arabic | 57.3% | 98.6% | 2.1% | 56.1% | 82.9% | 4,566 |
| Russian | 61.7% | 99.9% | 2.0% | 61.7% | 84.3% | 4,524 |

| Delivered (J), non-English mean | Released | SFT | **RL additive** | RL gated |
| :--- | :---: | :---: | :---: | :---: |
| PolyMath, 444 problems | 16.8% | 34.1% | **35.4%** | 36.8% |
| MMLU-ProX-Lite, 588 questions | 24.0% | 49.0% | **56.2%** | 55.5% |

</details>

## Quick start

```python
from datasets import load_dataset
from transformers import AutoModelForCausalLM, AutoTokenizer

name = "96kevinli29/Qwen3-4B-MathRL-Multilingual-Additive"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForCausalLM.from_pretrained(name, torch_dtype="auto", device_map="auto")

# a French problem with a French instruction: "…en raisonnant entièrement en français."
prompt = load_dataset("96kevinli29/Multilingual-Math-Reasoning-Eval", "fr")["amc23"][0]["prompt"]
text = tok.apply_chat_template([{"role": "user", "content": prompt}], tokenize=False, add_generation_prompt=True)
inputs = tok(text, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=24576, do_sample=True, temperature=0.7)
print(tok.decode(out[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))
```

Qwen3 chat template with thinking on; the reasoning appears inside `<think> … </think>`. Give hard problems a large `max_new_tokens`. Loads with transformers 4.x and 5.x.

<details>
<summary><b>Prompt format in six languages</b></summary>

| | Instruction that opens the user turn |
| :--- | :--- |
| English | Solve the following math problem step by step, reasoning entirely in English. |
| Chinese | 请全程用中文逐步推理并解决以下数学问题。 |
| Spanish | Resuelve el siguiente problema matemático paso a paso, razonando enteramente en español. |
| French | Résous le problème de mathématiques suivant étape par étape, en raisonnant entièrement en français. |
| Arabic | حل المسألة الرياضية التالية خطوة بخطوة مع التفكير بالكامل باللغة العربية. |
| Russian | Решите следующую математическую задачу шаг за шагом, рассуждая полностью на русском языке. |

The full prompts (instruction, problem and answer format, all in the target language) are in the evaluation set.

</details>

## Model family

| Stage | Checkpoint | |
| :--- | :--- | :--- |
| I · Release | [Qwen/Qwen3-4B](https://huggingface.co/Qwen/Qwen3-4B) · [Qwen/Qwen3-8B](https://huggingface.co/Qwen/Qwen3-8B) | solves the problem, reasons in English |
| II · SFT | [Qwen3-4B-SFT-Math-Multilingual](https://huggingface.co/96kevinli29/Qwen3-4B-SFT-Math-Multilingual) | reasons in the requested language, sometimes loops |
| II · SFT | [Qwen3-8B-SFT-Math-Multilingual](https://huggingface.co/96kevinli29/Qwen3-8B-SFT-Math-Multilingual) | the same result at 8B |
| III · RL | [Qwen3-4B-MathRL-Multilingual-Gated](https://huggingface.co/96kevinli29/Qwen3-4B-MathRL-Multilingual-Gated) | keeps the language, stops the looping |
| III · RL | [Qwen3-4B-MathRL-Multilingual-Additive](https://huggingface.co/96kevinli29/Qwen3-4B-MathRL-Multilingual-Additive) ← this model | keeps the language, stops the looping |

Data: [SFT-Math-Multilingual-43k](https://huggingface.co/datasets/96kevinli29/SFT-Math-Multilingual-43k) · [SFT-Math-Multilingual-56k](https://huggingface.co/datasets/96kevinli29/SFT-Math-Multilingual-56k) · [DAPO-Math-Multilingual-6Lang](https://huggingface.co/datasets/96kevinli29/DAPO-Math-Multilingual-6Lang) · [Multilingual-Math-Reasoning-Eval](https://huggingface.co/datasets/96kevinli29/Multilingual-Math-Reasoning-Eval)

## Training

| | |
| :--- | :--- |
| Algorithm | PPO, GAE with γ = λ = 1, clip ratio 0.2, no KL term |
| Reward | `0.7 · C + 0.3 · ℓ`, plus a soft penalty for overlong responses |
| Prompts | [DAPO-Math-Multilingual-6Lang](https://huggingface.co/datasets/96kevinli29/DAPO-Math-Multilingual-6Lang), 16,754 localized DAPO-Math-17K prompts |
| Batch | 32 prompts × 8 rollouts per step, 16,384-token response cap |
| Learning rate | 1e-6 actor, 1e-5 critic |
| Steps | 200 |
| Hardware · framework | 8 GPUs on one node · [verl](https://github.com/volcengine/verl), FSDP |

`C` is rule-based correctness of the final answer; `ℓ ∈ [0, 1]` is a language-identification score of the response against the language of the prompt.

## Limitations

- **Accuracy** stays below the released Qwen3-4B on the same problems (76.2% → 60.1%). The cost enters with SFT; RL does not recover it, and it is now what limits delivery.
- 200 PPO steps, one seed, one model size.
- Mathematics and six languages only.

## Citation

```bibtex
@article{li2026posttraining,
  title   = {What Does Post-Training Change in Multilingual Reasoning?},
  author  = {Li, Hongyang and Li, Xiao and Wu, Caesar and Danoy, Gr{\'e}goire and Bouvry, Pascal},
  journal = {arXiv preprint arXiv:2609.37104},
  year    = {2026}
}

@misc{qwen3-4b-mathrl-multilingual-additive-2026,
  title        = {{Qwen3-4B-MathRL-Multilingual-Additive}: PPO from multilingual-SFT {Qwen3-4B} with a additive language reward},
  author       = {Li, Hongyang and Li, Xiao and Wu, Caesar and Danoy, Gr{\'e}goire and Bouvry, Pascal and {Sea-Fill Community}},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/96kevinli29/Qwen3-4B-MathRL-Multilingual-Additive}},
  note         = {Maintained by Sea-Fill Community.}
}
```

Built and released by the authors together with the [Sea-Fill Community](https://huggingface.co/SeaFill2025), which maintains this release.