| scored 12689/12689 cases, 215,713 decisions; off-menu mass 0.0000; missing mass 0.0000 |
|
|
| | Macro F1 at 0.5 | Level macro F1 | Accuracy | Log loss | Brier | ECE | p50 latency | |
| |---:|---:|---:|---:|---:|---:|---:| |
| | 0.929 (0.707 to 0.984) | 0.669 (0.464 to 0.789) | 0.963 (0.948 to 0.981) | 0.136 (0.072 to 0.200) | 0.042 (0.022 to 0.060) | 0.030 | 44.2 ms | |
|
|
| Accuracy by type: score 0.756, noul 0.976; safe cases judged safe 0.988 |
|
|
| | Question | Type | Positives | Accuracy | F1 at 0.5 | Log loss | Brier | ECE | |
| |---|---|---:|---:|---:|---:|---:|---:| |
| | `risk_level` | score | | 0.756 | | 1.091 | 0.411 | 0.186 | |
| | `risky` | noul | 8393 | 0.996 | 0.997 | 0.022 | 0.003 | 0.004 | |
| | `data_destruction` | noul | 3467 | 0.923 | 0.876 | 0.170 | 0.053 | 0.065 | |
| | `device_overwrite` | noul | 2400 | 0.998 | 0.994 | 0.006 | 0.002 | 0.002 | |
| | `overbroad_permissions` | noul | 1291 | 1.000 | 0.998 | 0.001 | 0.000 | 0.000 | |
| | `privilege_escalation` | noul | 20 | 1.000 | 1.000 | 0.000 | 0.000 | 0.000 | |
| | `remote_code_execution` | noul | 250 | 1.000 | 1.000 | 0.000 | 0.000 | 0.000 | |
| | `unsafe_download_exec` | noul | 620 | 1.000 | 1.000 | 0.000 | 0.000 | 0.000 | |
| | `resource_exhaustion` | noul | 335 | 1.000 | 1.000 | 0.002 | 0.000 | 0.002 | |
| | `credential_exposure` | noul | 510 | 1.000 | 1.000 | 0.001 | 0.000 | 0.001 | |
| | `irreversible_config` | noul | 649 | 0.956 | 0.693 | 0.054 | 0.020 | 0.032 | |
| | `history_log_tampering` | noul | 14 | 1.000 | 0.812 | 0.002 | 0.000 | 0.000 | |
| | `network_disruption` | noul | 460 | 1.000 | 1.000 | 0.000 | 0.000 | 0.000 | |
| | `signal_kill` | noul | 33 | 1.000 | 1.000 | 0.000 | 0.000 | 0.000 | |
| | `shell_footgun` | noul | 1776 | 0.934 | 0.704 | 0.316 | 0.061 | 0.063 | |
| | `reversible` | noul | 5576 | 0.884 | 0.859 | 0.380 | 0.097 | 0.098 | |
| | `requires_root` | noul | 4495 | 0.933 | 0.911 | 0.265 | 0.058 | 0.053 | |
|
|
