How to use from
Docker Model Runner
docker model run hf.co/hector-gr/CARE-Qwen2.5-7B-Math
Quick Links

CARE Qwen2.5-7B — Math

Qwen2.5-7B trained on Big-Math-Digits (mehuldamani/big-math-digits) to verbalize the confidence in its own answer with CARE.

Paper: Cross-Question Reliable Reinforcement Learning — Hector G. Rodriguez and Marcus Rohrbach.

Abstract

AI systems operating near the limits of their capabilities can maximize utility while minimizing the risk of error by abstaining or deferring questions they are not confident about to a human or a stronger model. Recently, large language models (LLMs) have been post-trained to estimate confidence in their own answers. To achieve this, methods reinforce confidence scores based on the correctness signal to each question-answer pair individually, with rewards that encourage assigning confidence 1 to correct answers and 0 to incorrect answers. However, estimating confidence scores is useful in that it enables selecting only the most confident answers, to maximize the share of solved questions while accommodating the user's risk tolerance. Assigning confidence 1 to correct answers and 0 to incorrect ones is theoretically optimal, but unattainable in practice and detrimental for selective prediction with a non-zero risk tolerance: confidences pushed to 0 and 1 carry no information about how answers compare to one another. We therefore introduce CARE (Cross-question Abstention Reward Estimation), which computes rewards over a set of question-answer pairs: At each of several risk tolerances, the confidences across the set induce an abstention threshold, and each confidence is rewarded for lying on the correct side of it. Since the threshold itself comes from the set, every verbalized confidence is reinforced relative to other questions' confidence, not against its own binary correctness label. On math, CARE increases the coverage at 5% risk (C@5) across MATH-500, GSM8K, and Big-Math-Digits from 0.8% to 42.4%, compared to existing pointwise correctness-matching rewards. When trained on multi-hop Wikipedia QA from HotpotQA, CARE improves C@5 in out-of-distribution general-knowledge, reasoning, and math benchmarks (including CommonsenseQA, GPQA, SimpleQA, and TriviaQA) from 12.3% to 23.6%.

Results

Selective prediction (all values in %; C@r is coverage at risk r; AURC lower is better). ID: Big-Math-Digits, MATH-500, GSM8K. OOD: CommonsenseQA, GPQA, SimpleQA, TriviaQA, HotpotQA.

Model ID C@0 ID C@1 ID C@5 ID AURC ↓ ID Acc OOD C@0 OOD C@1 OOD C@5 OOD AURC ↓ OOD Acc
RLCR (Cold) 0.0 0.0 0.8 19.3 73.1 0.0 0.0 4.9 38.7 56.6
CARE (this model) 5.4 18.2 42.4 12.5 73.1 2.9 5.4 15.9 37.7 53.6

Use

Load with AutoTokenizer.from_pretrained("hector-gr/CARE-Qwen2.5-7B-Math") and AutoModelForCausalLM.from_pretrained("hector-gr/CARE-Qwen2.5-7B-Math"). Use the exact system prompt in system_prompt.txt and a user message of the form PROBLEM: <question>. The paper evaluates with temperature 0, one completion and a 4096-token generation limit.

Outputs contain <answer>...</answer> and <confidence>...</confidence>. Rank answers by confidence and choose a threshold using suitable held-out data. A risk measured on one dataset need not transfer to a different distribution.

The training configuration is included in training_config.yaml (seed 43).

Limitations

The model can still give incorrect answers with high confidence. It has not been validated for safety-critical decisions. Model weights are released under Apache 2.0, with the base-model license included.

Attribution

Built on RLCR, Beyond Binary Rewards: Training LMs to Reason About Their Uncertainty (Damani et al., 2025), and TRL. Base model: Qwen2.5-7B, Qwen Team.

Downloads last month
19
Safetensors
Model size
8B params
Tensor type
BF16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for hector-gr/CARE-Qwen2.5-7B-Math

Base model

Qwen/Qwen2.5-7B
Finetuned
(904)
this model

Dataset used to train hector-gr/CARE-Qwen2.5-7B-Math

Collection including hector-gr/CARE-Qwen2.5-7B-Math