Add safeagents judge bench results

#15
by ybelkada HF Staff - opened

Hi! This adds fastino/GLiNER2.5-Decide's results on the safeagents judge bench, a benchmark of real agent tool calls that safeagents' scanner flags as suspicious, each labelled block or allow and with a kind of misuse (reward hacking, harness introspection, credential misuse, ...).

On the 562 test calls: balanced accuracy 0.577, AUROC 0.523, false-block rate 0.0, category macro F1 0.344. Run zero-shot, one System One request per call, on Apple M5 Pro (CPU). Per-call verdicts are in the dataset's results/ folder.

The file is named after the benchmark, so it doesn't touch any existing .eval_results file.

Updated to the v2 release.

Hi! This adds fastino/GLiNER2.5-Decide's results on the safeagents judge bench v2, a benchmark of real agent tool calls that safeagents' scanner flags as suspicious, each labelled block or allow and with a kind of misuse (reward hacking, harness introspection, credential misuse, ...).

On the 661 test calls: balanced accuracy 0.576, AUROC 0.517, false-block rate 0.0, category macro F1 0.355. Run zero-shot, one System One request per call, on Apple M5 Pro (CPU). Per-call verdicts are in the dataset's results/ folder.

The file is named after the benchmark, so it doesn't touch any existing .eval_results file. v2 keeps every earlier call and adds new ones (see the dataset card); these numbers replace the earlier ones in this file.

Ready to merge
This branch is ready to get merged automatically.

Sign up or log in to comment