Text Generation
Transformers
Safetensors
qwen3
causal-lm
reinforcement-learning
ppo
rlvr
math
reasoning
multilingual
multilingual-reasoning
chain-of-thought
verl
conversational
Eval Results (legacy)
text-generation-inference
Instructions to use 96kevinli29/Qwen3-4B-MathRL-Multilingual-Additive with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use 96kevinli29/Qwen3-4B-MathRL-Multilingual-Additive with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="96kevinli29/Qwen3-4B-MathRL-Multilingual-Additive") messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("96kevinli29/Qwen3-4B-MathRL-Multilingual-Additive") model = AutoModelForCausalLM.from_pretrained("96kevinli29/Qwen3-4B-MathRL-Multilingual-Additive", device_map="auto") messages = [ {"role": "user", "content": "Who are you?"}, ] inputs = tokenizer.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use 96kevinli29/Qwen3-4B-MathRL-Multilingual-Additive with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "96kevinli29/Qwen3-4B-MathRL-Multilingual-Additive" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "96kevinli29/Qwen3-4B-MathRL-Multilingual-Additive", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/96kevinli29/Qwen3-4B-MathRL-Multilingual-Additive
- SGLang
How to use 96kevinli29/Qwen3-4B-MathRL-Multilingual-Additive with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "96kevinli29/Qwen3-4B-MathRL-Multilingual-Additive" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "96kevinli29/Qwen3-4B-MathRL-Multilingual-Additive", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "96kevinli29/Qwen3-4B-MathRL-Multilingual-Additive" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "96kevinli29/Qwen3-4B-MathRL-Multilingual-Additive", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Docker Model Runner
How to use 96kevinli29/Qwen3-4B-MathRL-Multilingual-Additive with Docker Model Runner:
docker model run hf.co/96kevinli29/Qwen3-4B-MathRL-Multilingual-Additive
File size: 10,148 Bytes
6875d5f 63d7807 6875d5f 63d7807 6875d5f 63d7807 6875d5f 63d7807 6875d5f 63d7807 6875d5f 63d7807 6875d5f 63d7807 6875d5f 63d7807 6875d5f 63d7807 6875d5f 63d7807 6875d5f 63d7807 6875d5f 63d7807 6875d5f 63d7807 6875d5f 63d7807 6875d5f 63d7807 6875d5f 63d7807 6875d5f 63d7807 6875d5f 63d7807 6875d5f 63d7807 6875d5f 63d7807 6875d5f 63d7807 6875d5f 63d7807 6875d5f 63d7807 c80273b 6875d5f 81d4403 6875d5f 81d4403 63d7807 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 | ---
language:
- en
- zh
- es
- fr
- ar
- ru
license: apache-2.0
library_name: transformers
pipeline_tag: text-generation
tags:
- qwen3
- causal-lm
- reinforcement-learning
- ppo
- rlvr
- math
- reasoning
- multilingual
- multilingual-reasoning
- chain-of-thought
- verl
base_model: 96kevinli29/Qwen3-4B-SFT-Math-Multilingual
datasets:
- 96kevinli29/DAPO-Math-Multilingual-6Lang
model-index:
- name: Qwen3-4B-MathRL-Multilingual-Additive
results:
- task:
type: text-generation
dataset:
name: AIME-2026 + AMC-2023, five non-English languages (zh, es, fr, ar, ru)
type: 96kevinli29/Multilingual-Math-Reasoning-Eval
metrics:
- name: Delivered J@1 (correct, in the requested language, terminated)
type: accuracy
value: 59.8
- name: Answer accuracy
type: accuracy
value: 60.1
- name: Reasoning in the requested language
type: accuracy
value: 99.6
---
<div align="center">
# Qwen3-4B-MathRL-Multilingual-Additive
**Qwen3-4B that reasons in the language of the problem, and finishes**
<a href="https://arxiv.org/abs/2609.37104"><img alt="arXiv: 2609.37104" src="https://img.shields.io/badge/arXiv-2609.37104-b31b1b"></a> <a href="https://huggingface.co/Qwen/Qwen3-4B"><img alt="base: Qwen3-4B" src="https://img.shields.io/badge/base-Qwen3--4B-475467"></a> <img alt="stage: RL additive" src="https://img.shields.io/badge/stage-RL_additive-1baf7a"> <img alt="languages: 6" src="https://img.shields.io/badge/languages-6-2a78d6"> <img alt="license: Apache 2.0" src="https://img.shields.io/badge/license-Apache_2.0-1baf7a">
</div>
<img src="https://huggingface.co/96kevinli29/Qwen3-4B-MathRL-Multilingual-Additive/resolve/main/assets/lineage_4b.png" alt="Bar chart, non-English mean. Reasoning in the requested language: released 40%, SFT 99%, RL additive 100%, RL gated 100%. Did not finish: 5%, 21%, 1%, 2%. Delivered: 29%, 60%, 60%, 61%." width="100%">
<p align="center"><sub>Mean over Chinese, Spanish, French, Arabic and Russian · 70 problems × 16 samples per language</sub></p>
- **What it is.** [Qwen3-4B-SFT-Math-Multilingual](https://huggingface.co/96kevinli29/Qwen3-4B-SFT-Math-Multilingual) after 200 PPO steps on localized math prompts with the **additive reward `R = 0.7 · C + 0.3 · ℓ`**: correctness `C` plus a language score `ℓ` that is paid whether or not the answer is right.
- **What it fixes.** The SFT model reasons in the right language but 21.1% of its non-English responses never finish. Here that is **1.2%**, the reasoning stays in the requested language (99.6%), and the median response shrinks from 7,673 to 4,072 tokens.
- **What to know.** It is one of two matched runs that differ only in reward; [Qwen3-4B-MathRL-Multilingual-Gated](https://huggingface.co/96kevinli29/Qwen3-4B-MathRL-Multilingual-Gated) (gated reward) performs comparably. A third run rewarded for correctness alone also stops the looping but drifts back to English.
## Results
| | Released | SFT | **RL additive** | RL gated |
| :--- | :---: | :---: | :---: | :---: |
| Answer correct | 76.2% | 61.4% | **60.1%** | 61.7% |
| Reasoning in the requested language | 39.8% | 99.4% | **99.6%** | 99.7% |
| Did not finish | 4.7% | 21.1% | **1.2%** | 1.8% |
| **Delivered** (J@1) | 28.6% | 60.2% | **59.8%** | 61.4% |
| **Delivered**, best of 16 (J@16) | 35.7% | 84.6% | **83.1%** | 83.7% |
| Median response, tokens | 5,824 | 7,673 | **4,072** | 4,463 |
70 competition problems (30 AIME-2026 + 40 AMC-2023) per language from [Multilingual-Math-Reasoning-Eval](https://huggingface.co/datasets/96kevinli29/Multilingual-Math-Reasoning-Eval), 16 samples each at temperature 0.7, 24,576-token budget. A response is **delivered** when the answer is correct, the visible reasoning is in the requested language, and it terminates (`J = C ∧ L ∧ T`, scored per response). Tables show the mean over Chinese, Spanish, French, Arabic and Russian.
<details>
<summary><b>Per language · other benchmarks</b></summary>
| | Correct | In language | Did not finish | J@1 | J@16 | Median tokens |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: |
| English | 65.7% | 100.0% | 0.2% | 65.7% | 85.7% | 3,510 |
| Chinese | 58.7% | 99.7% | 0.6% | 58.5% | 81.4% | 3,596 |
| Spanish | 61.5% | 99.8% | 0.8% | 61.4% | 81.4% | 3,856 |
| French | 61.3% | 99.8% | 0.3% | 61.3% | 85.7% | 3,818 |
| Arabic | 57.3% | 98.6% | 2.1% | 56.1% | 82.9% | 4,566 |
| Russian | 61.7% | 99.9% | 2.0% | 61.7% | 84.3% | 4,524 |
| Delivered (J), non-English mean | Released | SFT | **RL additive** | RL gated |
| :--- | :---: | :---: | :---: | :---: |
| PolyMath, 444 problems | 16.8% | 34.1% | **35.4%** | 36.8% |
| MMLU-ProX-Lite, 588 questions | 24.0% | 49.0% | **56.2%** | 55.5% |
</details>
## Quick start
```python
from datasets import load_dataset
from transformers import AutoModelForCausalLM, AutoTokenizer
name = "96kevinli29/Qwen3-4B-MathRL-Multilingual-Additive"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForCausalLM.from_pretrained(name, torch_dtype="auto", device_map="auto")
# a French problem with a French instruction: "…en raisonnant entièrement en français."
prompt = load_dataset("96kevinli29/Multilingual-Math-Reasoning-Eval", "fr")["amc23"][0]["prompt"]
text = tok.apply_chat_template([{"role": "user", "content": prompt}], tokenize=False, add_generation_prompt=True)
inputs = tok(text, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=24576, do_sample=True, temperature=0.7)
print(tok.decode(out[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))
```
Qwen3 chat template with thinking on; the reasoning appears inside `<think> … </think>`. Give hard problems a large `max_new_tokens`. Loads with transformers 4.x and 5.x.
<details>
<summary><b>Prompt format in six languages</b></summary>
| | Instruction that opens the user turn |
| :--- | :--- |
| English | Solve the following math problem step by step, reasoning entirely in English. |
| Chinese | 请全程用中文逐步推理并解决以下数学问题。 |
| Spanish | Resuelve el siguiente problema matemático paso a paso, razonando enteramente en español. |
| French | Résous le problème de mathématiques suivant étape par étape, en raisonnant entièrement en français. |
| Arabic | حل المسألة الرياضية التالية خطوة بخطوة مع التفكير بالكامل باللغة العربية. |
| Russian | Решите следующую математическую задачу шаг за шагом, рассуждая полностью на русском языке. |
The full prompts (instruction, problem and answer format, all in the target language) are in the evaluation set.
</details>
## Model family
| Stage | Checkpoint | |
| :--- | :--- | :--- |
| I · Release | [Qwen/Qwen3-4B](https://huggingface.co/Qwen/Qwen3-4B) · [Qwen/Qwen3-8B](https://huggingface.co/Qwen/Qwen3-8B) | solves the problem, reasons in English |
| II · SFT | [Qwen3-4B-SFT-Math-Multilingual](https://huggingface.co/96kevinli29/Qwen3-4B-SFT-Math-Multilingual) | reasons in the requested language, sometimes loops |
| II · SFT | [Qwen3-8B-SFT-Math-Multilingual](https://huggingface.co/96kevinli29/Qwen3-8B-SFT-Math-Multilingual) | the same result at 8B |
| III · RL | [Qwen3-4B-MathRL-Multilingual-Gated](https://huggingface.co/96kevinli29/Qwen3-4B-MathRL-Multilingual-Gated) | keeps the language, stops the looping |
| III · RL | [Qwen3-4B-MathRL-Multilingual-Additive](https://huggingface.co/96kevinli29/Qwen3-4B-MathRL-Multilingual-Additive) ← this model | keeps the language, stops the looping |
Data: [SFT-Math-Multilingual-43k](https://huggingface.co/datasets/96kevinli29/SFT-Math-Multilingual-43k) · [SFT-Math-Multilingual-56k](https://huggingface.co/datasets/96kevinli29/SFT-Math-Multilingual-56k) · [DAPO-Math-Multilingual-6Lang](https://huggingface.co/datasets/96kevinli29/DAPO-Math-Multilingual-6Lang) · [Multilingual-Math-Reasoning-Eval](https://huggingface.co/datasets/96kevinli29/Multilingual-Math-Reasoning-Eval)
## Training
| | |
| :--- | :--- |
| Algorithm | PPO, GAE with γ = λ = 1, clip ratio 0.2, no KL term |
| Reward | `0.7 · C + 0.3 · ℓ`, plus a soft penalty for overlong responses |
| Prompts | [DAPO-Math-Multilingual-6Lang](https://huggingface.co/datasets/96kevinli29/DAPO-Math-Multilingual-6Lang), 16,754 localized DAPO-Math-17K prompts |
| Batch | 32 prompts × 8 rollouts per step, 16,384-token response cap |
| Learning rate | 1e-6 actor, 1e-5 critic |
| Steps | 200 |
| Hardware · framework | 8 GPUs on one node · [verl](https://github.com/volcengine/verl), FSDP |
`C` is rule-based correctness of the final answer; `ℓ ∈ [0, 1]` is a language-identification score of the response against the language of the prompt.
## Limitations
- **Accuracy** stays below the released Qwen3-4B on the same problems (76.2% → 60.1%). The cost enters with SFT; RL does not recover it, and it is now what limits delivery.
- 200 PPO steps, one seed, one model size.
- Mathematics and six languages only.
## Citation
```bibtex
@article{li2026posttraining,
title = {What Does Post-Training Change in Multilingual Reasoning?},
author = {Li, Hongyang and Li, Xiao and Wu, Caesar and Danoy, Gr{\'e}goire and Bouvry, Pascal},
journal = {arXiv preprint arXiv:2609.37104},
year = {2026}
}
@misc{qwen3-4b-mathrl-multilingual-additive-2026,
title = {{Qwen3-4B-MathRL-Multilingual-Additive}: PPO from multilingual-SFT {Qwen3-4B} with a additive language reward},
author = {Li, Hongyang and Li, Xiao and Wu, Caesar and Danoy, Gr{\'e}goire and Bouvry, Pascal and {Sea-Fill Community}},
year = {2026},
publisher = {Hugging Face},
howpublished = {\url{https://huggingface.co/96kevinli29/Qwen3-4B-MathRL-Multilingual-Additive}},
note = {Maintained by Sea-Fill Community.}
}
```
Built and released by the authors together with the [Sea-Fill Community](https://huggingface.co/SeaFill2025), which maintains this release.
|