hector-gr commited on
Commit
a80da59
·
verified ·
1 Parent(s): dc88f9c

Update model card

Browse files
Files changed (1) hide show
  1. README.md +17 -8
README.md CHANGED
@@ -15,15 +15,22 @@ tags:
15
 
16
  # CARE Qwen2.5-7B — Math
17
 
18
- CARE learns confidence for selective prediction: answering more questions while respecting a chosen error tolerance. Its coverage rewards compare answers across questions rather than scoring each confidence value in isolation.
19
 
20
- This checkpoint was trained from Qwen2.5-7B on `mehuldamani/big-math-digits`. It reports an answer and a verbalized confidence score, using the TABC output format. Confidence is a selection score, not a guarantee of correctness.
21
 
22
- ## Private release staging
23
 
24
- This repository is a private checkpoint candidate. The local weight structure has been validated, but final paper-row provenance, end-to-end inference checks and release documentation are still being audited. Do not treat unverified historical evaluation numbers as measurements of this uploaded revision. No benchmark scores are asserted here yet.
25
 
26
- The original saved run reached step 208, approximately 0.4992 epochs. The intended seed is 43; the public training recipe is included for inspection. This is not the newly launched minimal-code reproduction run.
 
 
 
 
 
 
 
27
 
28
  ## Use
29
 
@@ -31,10 +38,12 @@ Load with `AutoTokenizer.from_pretrained("hector-gr/CARE-Qwen2.5-7B-Math")` and
31
 
32
  Outputs contain `<answer>...</answer>` and `<confidence>...</confidence>`. Rank answers by confidence and choose a threshold using suitable held-out data. A risk measured on one dataset need not transfer to a different distribution.
33
 
34
- ## Method and limitations
 
 
35
 
36
- Training retains format, answer-correctness and Brier rewards and adds cross-question coverage, confidence uniqueness and histogram-occupancy rewards. The model can still give incorrect answers with high confidence. It has not been validated for safety-critical decisions. Model weights are released under Apache 2.0, with the base-model license included.
37
 
38
  ## Attribution
39
 
40
- Cross-Question Reliable Reinforcement Learning — Hector G. Rodriguez and Marcus Rohrbach. Built on RLCR, Beyond Binary Rewards: Training LMs to Reason About Their Uncertainty (Damani et al., 2025), and TRL. Base model: Qwen2.5-7B, Qwen Team.
 
15
 
16
  # CARE Qwen2.5-7B — Math
17
 
18
+ Qwen2.5-7B trained on Big-Math-Digits (`mehuldamani/big-math-digits`) to verbalize the confidence in its own answer with CARE.
19
 
20
+ Paper: *Cross-Question Reliable Reinforcement Learning* — Hector G. Rodriguez and Marcus Rohrbach.
21
 
22
+ ## Abstract
23
 
24
+ AI systems operating near the limits of their capabilities can maximize utility while minimizing the risk of error by abstaining or deferring questions they are not confident about to a human or a stronger model. Recently, large language models (LLMs) have been post-trained to estimate confidence in their own answers. To achieve this, methods reinforce confidence scores based on the correctness signal to each question-answer pair *individually*, with rewards that encourage assigning confidence 1 to correct answers and 0 to incorrect answers. However, estimating confidence scores is useful in that it enables selecting only the most confident answers, to maximize the share of solved questions while accommodating the user's risk tolerance. Assigning confidence 1 to correct answers and 0 to incorrect ones is theoretically optimal, but unattainable in practice and detrimental for selective prediction with a non-zero risk tolerance: confidences pushed to 0 and 1 carry no information about how answers compare to one another. We therefore introduce CARE (Cross-question Abstention Reward Estimation), which computes rewards over a set of question-answer pairs: At each of several risk tolerances, the confidences across the set induce an abstention threshold, and each confidence is rewarded for lying on the correct side of it. Since the threshold itself comes from the set, every verbalized confidence is reinforced relative to other questions' confidence, not against its own binary correctness label. On math, CARE increases the coverage at 5% risk (C@5) across MATH-500, GSM8K, and Big-Math-Digits from 0.8% to 42.4%, compared to existing pointwise correctness-matching rewards. When trained on multi-hop Wikipedia QA from HotpotQA, CARE improves C@5 in out-of-distribution general-knowledge, reasoning, and math benchmarks (including CommonsenseQA, GPQA, SimpleQA, and TriviaQA) from 12.3% to 23.6%.
25
 
26
+ ## Results
27
+
28
+ Selective prediction (all values in %; C@r is coverage at risk r; AURC lower is better). ID: Big-Math-Digits, MATH-500, GSM8K. OOD: CommonsenseQA, GPQA, SimpleQA, TriviaQA, HotpotQA.
29
+
30
+ | Model | ID C@0 | ID C@1 | ID C@5 | ID AURC ↓ | ID Acc | OOD C@0 | OOD C@1 | OOD C@5 | OOD AURC ↓ | OOD Acc |
31
+ |---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
32
+ | RLCR (Cold) | 0.0 | 0.0 | 0.8 | 19.3 | 73.1 | 0.0 | 0.0 | 4.9 | 38.7 | 56.6 |
33
+ | **CARE (this model)** | **5.4** | **18.2** | **42.4** | **12.5** | 73.1 | **2.9** | **5.4** | **15.9** | 37.7 | 53.6 |
34
 
35
  ## Use
36
 
 
38
 
39
  Outputs contain `<answer>...</answer>` and `<confidence>...</confidence>`. Rank answers by confidence and choose a threshold using suitable held-out data. A risk measured on one dataset need not transfer to a different distribution.
40
 
41
+ The training configuration is included in `training_config.yaml` (seed 43).
42
+
43
+ ## Limitations
44
 
45
+ The model can still give incorrect answers with high confidence. It has not been validated for safety-critical decisions. Model weights are released under Apache 2.0, with the base-model license included.
46
 
47
  ## Attribution
48
 
49
+ Built on RLCR, Beyond Binary Rewards: Training LMs to Reason About Their Uncertainty (Damani et al., 2025), and TRL. Base model: Qwen2.5-7B, Qwen Team.