---
base_model: microsoft/Phi-4-mini-instruct
datasets:
- mehuldamani/big-math-digits
library_name: transformers
tags:
- rl-verify
- grpo
- reasoning
- calibration
license: other
---

# RLCR-phi4mini-bigmathdigits5000

This model is **Phi-4-mini-instruct** trained with **RLCR** (Reinforcement Learning with Calibration Rewards, Damani et al. 2026) on Big-Math-digits: adds a Brier-score-based reward on verbalized confidence to the correctness reward, jointly optimizing reasoning and calibration.

## Results

| Metric | Value |
|---|---|
| Accuracy (held-out) | 0.414 |
| ECE | 0.439 |
| PCE | 0.439 |
| Brier score | 0.427 |
| AUROC | 0.652 |

_Table 4 of the paper. Confidence source for the Phi-4-mini RLCC curricula._

## LoRA

LoRA target modules: `qkv_proj`, `o_proj`, `gate_up_proj`, `down_proj` (explicitly specified).

Rank 16, alpha 32, dropout 0.05.


## Training details

- **Algorithm**: GRPO, 1 epoch, per-device batch size 1, KL penalty β = 0, AdamW (8-bit), max grad norm 1.0.
- **Dataset**: [mehuldamani/big-math-digits](https://huggingface.co/datasets/mehuldamani/big-math-digits), 5,000 training samples, 1,000 held-out for evaluation.
- **Rollouts**: vLLM, temperature 0.7, 32 generations per prompt, effective rollout batch size 1,024.
- **Learning rate**: 5e-6, linear schedule, warmup ratio 0.20.
- **Max response length**: 1,024 tokens.
- **Answer judging**: rule-based (exact match on extracted final answer).
- **Compute**: single RTX 3090 (16-36 GPU-hours) except Qwen3-4B (single A100, 13-19 GPU-hours).


## Citation

This checkpoint accompanies the paper *"Confidence as Curriculum: Reinforcement Learning for Joint Reasoning and Calibration"* (currently under anonymous peer review). A formal citation will be added once the paper is public. It builds directly on:

```bibtex
@inproceedings{damani2026rlcr,
  title     = {Beyond Binary Rewards: Training {LM}s to Reason about their Uncertainty},
  author    = {Damani, Mehul and Puri, Isha and Slocum, Stewart and Shenfeld, Idan and Choshen, Leshem and Kim, Yoon and Andreas, Jacob},
  booktitle = {The Fourteenth International Conference on Learning Representations},
  year      = {2026}
}
```