| --- |
| license: apache-2.0 |
| base_model: SUSTech/Qwen3-8B-CalibSFT |
| base_model_relation: finetune |
| library_name: transformers |
| pipeline_tag: text-generation |
| language: |
| - en |
| datasets: |
| - SUSTech/CalibSFT-DeepScaleR |
| tags: |
| - reasoning |
| - math |
| - confidence-calibration |
| - verbalized-confidence |
| --- |
| |
| # Qwen3-8B-CalibSFT-RLCR |
|
|
| <p align="center"> |
| <a href="https://arxiv.org/abs/2609.32470"><img src="https://img.shields.io/badge/Paper-arXiv-b31b1b.svg?logo=arxiv" alt="Paper"></a> |
| <a href="https://github.com/ml-stat-Sustech/verbalized-confidence-training"><img src="https://img.shields.io/badge/Code-GitHub-181717.svg?logo=github" alt="Code"></a> |
| <a href="https://huggingface.co/datasets/SUSTech/CalibSFT-DeepScaleR"><img src="https://img.shields.io/badge/Dataset-Hugging%20Face-ffd21e.svg?logo=huggingface" alt="Dataset"></a> |
| </p> |
|
|
| ## Introduction |
|
|
| Qwen3-8B-CalibSFT-RLCR is [Qwen3-8B-CalibSFT](https://huggingface.co/SUSTech/Qwen3-8B-CalibSFT) further trained with RLCR, a confidence-aware reinforcement learning method. It is released with our paper [On the Pitfalls of Verbalized Confidence Priors for Calibrating Large Reasoning Models](https://arxiv.org/abs/2609.32470). Compared with RLCR trained from Qwen3-8B, it improves accuracy, discrimination, and calibration on both in-distribution and out-of-distribution benchmarks. For training details, please refer to our [GitHub repository](https://github.com/ml-stat-Sustech/verbalized-confidence-training). |
|
|
| ## Usage |
|
|
| Use the system prompt and user format from training: |
|
|
| ```python |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| |
| model_id = "SUSTech/Qwen3-8B-CalibSFT-RLCR" |
| tokenizer = AutoTokenizer.from_pretrained(model_id) |
| model = AutoModelForCausalLM.from_pretrained(model_id, dtype="auto", device_map="auto") |
| |
| SYSTEM_PROMPT = ( |
| "A conversation between User and Assistant. The user asks a question, and the Assistant solves it. " |
| "The assistant first thinks about the reasoning process in the mind and analyzes its confidence about " |
| "the solution and then provides the user with the final answer as well as its confidence level. " |
| "The confidence level indicates how certain the Assistant is about its answer, expressed as a decimal " |
| "between 0 and 1 with exactly two decimal places, enclosed within <confidence> </confidence> tags. " |
| "The response must strictly follow this format: <think> reasoning process here </think> " |
| "<answer> final short answer only </answer> <confidence> 0.xx </confidence>. " |
| "The <answer> tag must contain only the final answer string needed for exact-match evaluation, " |
| "not a full sentence, explanation, or reasoning." |
| ) |
| question = "What is the smallest positive integer n such that n^2 + n is divisible by 12?" |
| messages = [ |
| {"role": "system", "content": SYSTEM_PROMPT}, |
| {"role": "user", "content": f"\n\nPROBLEM: {question}\n\n"}, |
| ] |
| |
| text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) |
| inputs = tokenizer(text, return_tensors="pt").to(model.device) |
| outputs = model.generate(**inputs, max_new_tokens=16384, do_sample=True, temperature=0.6, top_p=0.95, top_k=20) |
| print(tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)) |
| # <think> ... </think> <answer> ... </answer> <confidence> 0.xx </confidence> |
| ``` |
|
|
| ## Evaluation |
|
|
| Results are averaged over 8 in-distribution math benchmarks (DeepScaleR-Eval, MATH-500, MinervaMath, OlympiadBench, GSM8K, AIME 2024–2026) and 8 out-of-distribution benchmarks (HotpotQA, TriviaQA, DROP, MuSiQue, LiveBenchReasoning, NQOpen, PopQA, WebQuestions), sampling at temperature 0.6 with 4 responses per question (32 on AIME). DeepScaleR-Eval consists of 2,000 DeepScaleR questions held out from training. |
|
|
| <table> |
| <thead> |
| <tr> |
| <th rowspan="2">Model</th> |
| <th colspan="4">In-distribution</th> |
| <th colspan="4">Out-of-distribution</th> |
| </tr> |
| <tr> |
| <th>Pass@1 ↑</th><th>AUROC ↑</th><th>Brier ↓</th><th>ECE ↓</th> |
| <th>Pass@1 ↑</th><th>AUROC ↑</th><th>Brier ↓</th><th>ECE ↓</th> |
| </tr> |
| </thead> |
| <tbody> |
| <tr><td>Qwen3-8B</td><td>59.32</td><td>71.90</td><td>33.20</td><td>34.36</td><td>45.36</td><td>65.71</td><td>45.15</td><td>46.94</td></tr> |
| <tr><td>Qwen3-8B-RLCR</td><td>65.33</td><td>82.36</td><td>14.43</td><td>14.46</td><td>43.96</td><td>69.38</td><td>31.14</td><td>31.17</td></tr> |
| <tr><td><b>Qwen3-8B-CalibSFT-RLCR</b></td><td>67.20</td><td>86.32</td><td>12.77</td><td>8.59</td><td>45.71</td><td>71.31</td><td>23.24</td><td>18.20</td></tr> |
| </tbody> |
| </table> |
| |
| ## Citation |
|
|
| ```bibtex |
| @article{wang2026pitfalls, |
| title={On the Pitfalls of Verbalized Confidence Priors for Calibrating Large Reasoning Models}, |
| author={Wang, Shuoyuan and Luo, Beier and Zeng, Hao and Yu, Chengyao and Zhang, Songxin and Xie, Zejian and Jing, Bingyi and Wei, Hongxin}, |
| journal={arXiv preprint arXiv:2609.32470}, |
| year={2026} |
| } |
| ``` |
|
|
