README.md

---
license: apache-2.0
base_model: SUSTech/Qwen3-8B-CalibSFT
base_model_relation: finetune
library_name: transformers
pipeline_tag: text-generation
language:
  - en
datasets:
  - SUSTech/CalibSFT-DeepScaleR
tags:
  - reasoning
  - math
  - confidence-calibration
  - verbalized-confidence
---

# Qwen3-8B-CalibSFT-RLCR

<p align="center">
  <a href="https://arxiv.org/abs/2609.32470"><img src="https://img.shields.io/badge/Paper-arXiv-b31b1b.svg?logo=arxiv" alt="Paper"></a>
  <a href="https://github.com/ml-stat-Sustech/verbalized-confidence-training"><img src="https://img.shields.io/badge/Code-GitHub-181717.svg?logo=github" alt="Code"></a>
  <a href="https://huggingface.co/datasets/SUSTech/CalibSFT-DeepScaleR"><img src="https://img.shields.io/badge/Dataset-Hugging%20Face-ffd21e.svg?logo=huggingface" alt="Dataset"></a>
</p>

## Introduction

Qwen3-8B-CalibSFT-RLCR is [Qwen3-8B-CalibSFT](https://huggingface.co/SUSTech/Qwen3-8B-CalibSFT) further trained with RLCR, a confidence-aware reinforcement learning method. It is released with our paper [On the Pitfalls of Verbalized Confidence Priors for Calibrating Large Reasoning Models](https://arxiv.org/abs/2609.32470). Compared with RLCR trained from Qwen3-8B, it improves accuracy, discrimination, and calibration on both in-distribution and out-of-distribution benchmarks. For training details, please refer to our [GitHub repository](https://github.com/ml-stat-Sustech/verbalized-confidence-training).

## Usage

Use the system prompt and user format from training:

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "SUSTech/Qwen3-8B-CalibSFT-RLCR"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype="auto", device_map="auto")

SYSTEM_PROMPT = (
    "A conversation between User and Assistant. The user asks a question, and the Assistant solves it. "
    "The assistant first thinks about the reasoning process in the mind and analyzes its confidence about "
    "the solution and then provides the user with the final answer as well as its confidence level. "
    "The confidence level indicates how certain the Assistant is about its answer, expressed as a decimal "
    "between 0 and 1 with exactly two decimal places, enclosed within <confidence> </confidence> tags. "
    "The response must strictly follow this format: <think> reasoning process here </think> "
    "<answer> final short answer only </answer> <confidence> 0.xx </confidence>. "
    "The <answer> tag must contain only the final answer string needed for exact-match evaluation, "
    "not a full sentence, explanation, or reasoning."
)
question = "What is the smallest positive integer n such that n^2 + n is divisible by 12?"
messages = [
    {"role": "system", "content": SYSTEM_PROMPT},
    {"role": "user", "content": f"\n\nPROBLEM: {question}\n\n"},
]

text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=16384, do_sample=True, temperature=0.6, top_p=0.95, top_k=20)
print(tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))
# <think> ... </think> <answer> ... </answer> <confidence> 0.xx </confidence>
```

## Evaluation

Results are averaged over 8 in-distribution math benchmarks (DeepScaleR-Eval, MATH-500, MinervaMath, OlympiadBench, GSM8K, AIME 2024–2026) and 8 out-of-distribution benchmarks (HotpotQA, TriviaQA, DROP, MuSiQue, LiveBenchReasoning, NQOpen, PopQA, WebQuestions), sampling at temperature 0.6 with 4 responses per question (32 on AIME). DeepScaleR-Eval consists of 2,000 DeepScaleR questions held out from training.

<table>
  <thead>
    <tr>
      <th rowspan="2">Model</th>
      <th colspan="4">In-distribution</th>
      <th colspan="4">Out-of-distribution</th>
    </tr>
    <tr>
      <th>Pass@1 ↑</th><th>AUROC ↑</th><th>Brier ↓</th><th>ECE ↓</th>
      <th>Pass@1 ↑</th><th>AUROC ↑</th><th>Brier ↓</th><th>ECE ↓</th>
    </tr>
  </thead>
  <tbody>
    <tr><td>Qwen3-8B</td><td>59.32</td><td>71.90</td><td>33.20</td><td>34.36</td><td>45.36</td><td>65.71</td><td>45.15</td><td>46.94</td></tr>
    <tr><td>Qwen3-8B-RLCR</td><td>65.33</td><td>82.36</td><td>14.43</td><td>14.46</td><td>43.96</td><td>69.38</td><td>31.14</td><td>31.17</td></tr>
    <tr><td><b>Qwen3-8B-CalibSFT-RLCR</b></td><td>67.20</td><td>86.32</td><td>12.77</td><td>8.59</td><td>45.71</td><td>71.31</td><td>23.24</td><td>18.20</td></tr>
  </tbody>
</table>

## Citation

```bibtex
@article{wang2026pitfalls,
  title={On the Pitfalls of Verbalized Confidence Priors for Calibrating Large Reasoning Models},
  author={Wang, Shuoyuan and Luo, Beier and Zeng, Hao and Yu, Chengyao and Zhang, Songxin and Xie, Zejian and Jing, Bingyi and Wei, Hongxin},
  journal={arXiv preprint arXiv:2609.32470},
  year={2026}
}
```