# robopulsepp-v1: 7720 items

| model | answered | acc (95% CI) | balanced acc | macro-F1 | F1 pos / neg | AUROC | NLL | ECE | pred pos rate |
|---|---|---|---|---|---|---|---|---|---|
| JevAny-4B-RoboJudge | 7720/7720 | 60.4 [57.9, 62.9] | 65.9 | 55.8 | 70.0 / 41.6 | 0.728 | 0.649 | 0.095 | 51.0 |
| JevAny-4B | 7720/7720 | 69.9 [67.7, 72.0] | 59.1 | 57.5 | 80.5 / 34.4 | 0.641 | 0.580 | 0.050 | 73.0 |
| Qwen3.5-4B-zeroshot | 7720/7720 | 81.1 [78.2, 84.0] | 50.0 | 44.8 | 89.6 / 0.0 | 0.592 | 0.591 | 0.131 | 100.0 |
| JevAny-27B | 7720/7720 | 82.0 [79.2, 84.8] | 55.7 | 55.9 | 89.8 / 22.0 | 0.692 | 0.448 | 0.047 | 95.8 |
| Qwen3.8-27B-zeroshot | 7720/7720 | 78.7 [75.9, 81.5] | 55.2 | 55.6 | 87.6 / 23.5 | 0.569 | 0.536 | 0.119 | 91.0 |
| OneJev-27B | 7720/7720 | 48.0 [45.7, 50.3] | 62.7 | 46.7 | 54.9 / 38.5 | 0.700 | 0.754 | 0.215 | 34.2 |
| OneJev-9B | 7720/7720 | 29.2 [26.6, 31.8] | 54.0 | 28.9 | 24.4 / 33.4 | 0.564 | 1.142 | 0.457 | 12.6 |
| OneJev-4B | 7720/7720 | 47.0 [44.3, 49.6] | 58.3 | 45.2 | 55.1 / 35.3 | 0.617 | 0.773 | 0.176 | 37.0 |
| Intern-Decision-4B | 7720/7720 | 65.8 [63.6, 68.0] | 60.5 | 56.5 | 76.6 / 36.4 | 0.653 | 0.594 | 0.049 | 65.1 |
| OmniJev-4B | 7720/7720 | 63.9 [61.7, 66.0] | 54.6 | 52.6 | 75.8 / 29.4 | 0.556 | 0.612 | 0.039 | 67.8 |
| Jev-Omni-12B | 7720/7720 | 58.0 [54.9, 61.0] | 52.3 | 49.1 | 70.3 / 27.9 | 0.537 | 0.663 | 0.042 | 60.6 |

## Timing

| model | GPUs | load s | wall s (loop) | mean / median / p95 ms per item | items/s | median input tokens | GPU-hours |
|---|---|---|---|---|---|---|---|
| JevAny-4B-RoboJudge | 1x NVIDIA A100-SXM4-40GB | 4 | 2022 | 262 / 207 / 520 | 3.82 | 790 | 0.56 |
| JevAny-4B | 1x NVIDIA A100-SXM4-40GB | 7 | 2034 | 263 / 210 / 515 | 3.80 | 790 | 0.57 |
| Qwen3.5-4B-zeroshot | 1x NVIDIA A100-SXM4-40GB | 24 | 1792 | 232 / 186 / 470 | 4.31 | 899 | 0.50 |
| JevAny-27B | 2x NVIDIA A100-SXM4-40GB | 60 | 5425 | 703 / 572 / 1422 | 1.42 | 790 | 3.05 |
| Qwen3.8-27B-zeroshot | 2x NVIDIA A100-SXM4-40GB | 72 | 4739 | 614 / 514 / 1214 | 1.63 | 899 | 2.67 |
| OneJev-27B | 2x NVIDIA A100-SXM4-40GB | 279 | 4607 | 597 / 483 / 1212 | 1.68 | 883 | 2.71 |
| OneJev-9B | 1x NVIDIA A100-SXM4-40GB | 33 | 2054 | 266 / 217 / 538 | 3.76 | 883 | 0.58 |
| OneJev-4B | 1x NVIDIA A100-SXM4-40GB | 25 | 1740 | 225 / 181 / 466 | 4.44 | 883 | 0.49 |
| Intern-Decision-4B | 1x NVIDIA A100-SXM4-40GB | 26 | 1477 | 233 / 188 / 461 | 4.29 | 964 | 0.42 |
| OmniJev-4B | 1x NVIDIA A100-SXM4-40GB | 50 | 2286 | 296 / 285 / 379 | 3.38 | 570 | 0.65 |
| Jev-Omni-12B | 1x NVIDIA A100-SXM4-40GB | 39 | 1771 | 229 / 174 / 432 | 4.36 | 769 | 0.50 |

## Interval level (majority of kept pairs)

| model | intervals | acc (95% CI) | macro-F1 |
|---|---|---|---|
| JevAny-4B-RoboJudge | 1062 | 67.6 [64.6, 70.4] | 65.6 |
| JevAny-4B | 1062 | 67.3 [64.8, 70.0] | 58.7 |
| Qwen3.5-4B-zeroshot | 1062 | 71.1 [69.0, 73.2] | 41.6 |
| JevAny-27B | 1062 | 75.4 [73.3, 77.7] | 56.4 |
| Qwen3.8-27B-zeroshot | 1062 | 74.2 [72.0, 76.6] | 58.1 |
| OneJev-27B | 1062 | 51.9 [49.1, 54.7] | 51.9 |
| OneJev-9B | 1062 | 32.3 [29.9, 34.6] | 28.6 |
| OneJev-4B | 1062 | 50.1 [47.3, 53.0] | 49.9 |
| Intern-Decision-4B | 1062 | 68.6 [66.1, 71.3] | 62.1 |
| OmniJev-4B | 1062 | 64.4 [61.7, 67.1] | 52.6 |
| Jev-Omni-12B | 1062 | 56.0 [53.4, 58.6] | 50.5 |

## Slices (accuracy / macro-F1, %)

| slice | n | JevAny-4B-RoboJudge | JevAny-4B | Qwen3.5-4B-zeroshot | JevAny-27B | Qwen3.8-27B-zeroshot | OneJev-27B | OneJev-9B | OneJev-4B | Intern-Decision-4B | OmniJev-4B | Jev-Omni-12B |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| n_views=1 | 5097 | 65.9 / 60.2 | 70.0 / 58.6 | 80.8 / 44.7 | 82.5 / 59.5 | 77.1 / 58.2 | 49.0 / 47.6 | 28.0 / 27.5 | 50.6 / 47.7 | 63.6 / 55.0 | 60.2 / 49.6 | 55.2 / 47.6 |
| n_views=2 | 1693 | 45.4 / 44.8 | 66.6 / 52.3 | 79.0 / 44.1 | 77.9 / 45.1 | 78.7 / 44.6 | 48.6 / 47.9 | 33.7 / 33.6 | 43.0 / 42.7 | 68.3 / 60.4 | 70.7 / 60.4 | 58.5 / 51.8 |
| n_views=3 | 930 | 57.2 / 49.3 | 75.5 / 59.9 | 87.0 / 46.5 | 86.6 / 53.3 | 87.1 / 47.4 | 41.3 / 39.5 | 27.2 / 27.2 | 34.6 / 33.9 | 72.9 / 56.4 | 71.9 / 54.2 | 72.2 / 45.7 |

## Paired: JevAny-4B-RoboJudge − JevAny-4B

- accuracy: -9.6 pts, 95% CI [-12.6, -6.2]
- macro-F1: -1.7 pts, 95% CI [-4.8, 1.8]
- McNemar: JevAny-4B-RoboJudge only correct 995, JevAny-4B only correct 1733, exact p = 9.384e-46