Add safeagents judge bench results
#6
by ybelkada HF Staff - opened
Hi! This adds vllm-sr/d3-mini's results on the safeagents judge bench, a benchmark of real agent tool calls that safeagents' scanner flags as suspicious, each labelled block or allow and with a kind of misuse (reward hacking, harness introspection, credential misuse, ...).
On the 562 test calls: balanced accuracy 0.692, AUROC 0.975, false-block rate 0.615, category macro F1 0.367. Run zero-shot, one System One request per call, on Apple M5 Pro (MPS). Per-call verdicts are in the dataset's results/ folder.
The file is named after the benchmark, so it doesn't touch any existing .eval_results file.
Xunzhuo changed pull request status to merged