ybelkada HF Staff commited on
Commit
2f0e3e3
·
verified ·
1 Parent(s): 051bcc4

Add safeagents judge bench results

Browse files

Hi! This adds `LiquidAI/d1-3B`'s results on the [safeagents judge bench](https://huggingface.co/datasets/harness-hub/safeagents-judge-bench), a benchmark of real agent tool calls that [safeagents](https://github.com/huggingface/safeagents)' scanner flags as suspicious, each labelled block or allow and with a kind of misuse (reward hacking, harness introspection, credential misuse, ...).

On the 562 test calls: balanced accuracy 0.823, AUROC 0.919, false-block rate 0.026, category macro F1 0.238. Run zero-shot, one System One request per call, on Apple M5 Pro (MPS). Per-call verdicts are in the dataset's `results/` folder.

The file is named after the benchmark, so it doesn't touch any existing `.eval_results` file.

.eval_results/safeagents-judge-bench.yaml ADDED
@@ -0,0 +1,32 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ - dataset:
2
+ id: harness-hub/safeagents-judge-bench
3
+ task_id: balanced_accuracy
4
+ value: 0.823
5
+ date: "2026-10-30"
6
+ source:
7
+ url: https://huggingface.co/datasets/harness-hub/safeagents-judge-bench
8
+ name: "safeagents judge bench v0, test split, run on Apple M5 Pro (MPS)"
9
+ - dataset:
10
+ id: harness-hub/safeagents-judge-bench
11
+ task_id: auroc
12
+ value: 0.919
13
+ date: "2026-10-30"
14
+ source:
15
+ url: https://huggingface.co/datasets/harness-hub/safeagents-judge-bench
16
+ name: "safeagents judge bench v0, test split, run on Apple M5 Pro (MPS)"
17
+ - dataset:
18
+ id: harness-hub/safeagents-judge-bench
19
+ task_id: false_block_rate
20
+ value: 0.026
21
+ date: "2026-10-30"
22
+ source:
23
+ url: https://huggingface.co/datasets/harness-hub/safeagents-judge-bench
24
+ name: "safeagents judge bench v0, test split, run on Apple M5 Pro (MPS)"
25
+ - dataset:
26
+ id: harness-hub/safeagents-judge-bench
27
+ task_id: category_macro_f1
28
+ value: 0.238
29
+ date: "2026-10-30"
30
+ source:
31
+ url: https://huggingface.co/datasets/harness-hub/safeagents-judge-bench
32
+ name: "safeagents judge bench v0, test split, run on Apple M5 Pro (MPS)"