sriram1983007 commited on
Commit
0a2927b
·
verified ·
1 Parent(s): 0e07be5

Eval results: v2 (2026-11-12)

Browse files
.eval_results/sra-stablecoin-risk-bench.yaml CHANGED
@@ -1,18 +1,18 @@
1
  - dataset:
2
  id: sriram1983007/sra-stablecoin-risk-bench
3
  task_id: risk_gate
4
- value: 0.9949
5
- date: '2026-10-06'
6
  source:
7
- url: https://huggingface.co/datasets/sriram1983007/sra-bench-results
8
  name: Published predictions (self-reported)
9
- notes: Decision accuracy on 1,175 held-out payment cases; greedy decoding; exact training prompt. Unsafe approval rate 0.0047; over-blocking 0; injected payments approved 4/69.
10
  - dataset:
11
  id: sriram1983007/sra-stablecoin-risk-bench
12
  task_id: dispute
13
- value: 0.7782
14
- date: '2026-10-06'
15
  source:
16
- url: https://huggingface.co/datasets/sriram1983007/sra-bench-results
17
  name: Published predictions (self-reported)
18
- notes: Outcome accuracy on 825 held-out dispute cases; greedy decoding; exact training prompt. Impossible remedy rate 0.0038; wrongful refund rate 0.022.
 
1
  - dataset:
2
  id: sriram1983007/sra-stablecoin-risk-bench
3
  task_id: risk_gate
4
+ value: 0.9957
5
+ date: '2026-11-12'
6
  source:
7
+ url: https://huggingface.co/datasets/sriram1983007/sra-bench-results/tree/main/sra-riskgate-4b-v2
8
  name: Published predictions (self-reported)
9
+ notes: v2. Decision accuracy on 1,175 held-out payment cases; greedy decoding; exact training prompt. Unsafe approval rate 0.0035; over-blocking 0; injected payments approved 2/69. v1 (2026-10-06) scored 0.9949.
10
  - dataset:
11
  id: sriram1983007/sra-stablecoin-risk-bench
12
  task_id: dispute
13
+ value: 0.9018
14
+ date: '2026-11-12'
15
  source:
16
+ url: https://huggingface.co/datasets/sriram1983007/sra-bench-results/tree/main/sra-riskgate-4b-v2
17
  name: Published predictions (self-reported)
18
+ notes: v2, with the computed dispute window in the prompt (dispute_messages in sra-riskgate 1.3.0+). Outcome accuracy on 825 held-out dispute cases; greedy decoding. Impossible remedy rate 0.0057; wrongful refund rate 0.022; without the window in the prompt 0.8497. v1 (2026-10-06) scored 0.7782.