eval/comparison/robopulse.md

# robopulse-v1: 1800 items

| model | answered | acc (95% CI) | balanced acc | macro-F1 | F1 pos / neg | AUROC | NLL | ECE | pred pos rate |
|---|---|---|---|---|---|---|---|---|---|
| JevAny-4B-RoboJudge | 1800/1800 | 71.1 [68.8, 73.2] | 71.1 | 71.0 | 70.6 / 71.5 | 0.806 | 0.518 | 0.040 | 48.4 |
| JevAny-4B | 1800/1800 | 52.7 [50.5, 55.0] | 52.7 | 44.2 | 66.0 / 22.3 | 0.588 | 0.746 | 0.151 | 89.2 |
| Qwen3.5-4B-zeroshot | 1800/1800 | 49.9 [47.6, 52.2] | 49.9 | 33.4 | 66.5 / 0.2 | 0.519 | 1.277 | 0.393 | 99.8 |
| JevAny-27B | 1800/1800 | 56.6 [54.3, 59.0] | 56.6 | 53.0 | 66.1 / 39.9 | 0.627 | 0.750 | 0.141 | 77.8 |
| Qwen3.8-27B-zeroshot | 1800/1800 | 56.7 [54.5, 59.1] | 56.7 | 54.5 | 64.6 / 44.3 | 0.604 | 0.686 | 0.041 | 72.3 |
| OneJev-27B | 1800/1800 | 60.4 [58.1, 62.6] | 60.4 | 60.3 | 59.0 / 61.6 | 0.651 | 0.659 | 0.056 | 46.7 |
| OneJev-9B | 1800/1800 | 57.6 [55.2, 59.8] | 57.6 | 57.6 | 57.4 / 57.7 | 0.595 | 0.710 | 0.099 | 49.7 |
| OneJev-4B | 1800/1800 | 53.9 [51.6, 56.2] | 53.9 | 51.9 | 61.8 / 42.0 | 0.586 | 0.711 | 0.100 | 70.6 |
| Intern-Decision-4B | 1800/1800 | 51.6 [49.3, 53.9] | 51.6 | 48.1 | 61.6 / 34.6 | 0.547 | 0.722 | 0.110 | 76.1 |
| OmniJev-4B | 1800/1800 | 49.7 [47.4, 52.1] | 49.7 | 42.2 | 63.1 / 21.2 | 0.536 | 0.831 | 0.218 | 86.2 |
| Jev-Omni-12B | 1800/1800 | 49.4 [47.1, 51.8] | 49.4 | 46.9 | 58.4 / 35.3 | 0.492 | 0.744 | 0.112 | 71.7 |

## Timing

| model | GPUs | load s | wall s (loop) | mean / median / p95 ms per item | items/s | median input tokens | GPU-hours |
|---|---|---|---|---|---|---|---|
| JevAny-4B-RoboJudge | 1x NVIDIA A100-SXM4-40GB | 11 | 586 | 325 / 294 / 540 | 3.07 | 1269 | 0.17 |
| JevAny-4B | 1x NVIDIA A100-SXM4-40GB | 13 | 573 | 318 / 287 / 541 | 3.14 | 1269 | 0.16 |
| Qwen3.5-4B-zeroshot | 1x NVIDIA A100-SXM4-40GB | 29 | 523 | 291 / 265 / 465 | 3.44 | 1427 | 0.15 |
| JevAny-27B | 2x NVIDIA A100-SXM4-40GB | 56 | 1666 | 926 / 838 / 1542 | 1.08 | 1269 | 0.96 |
| Qwen3.8-27B-zeroshot | 2x NVIDIA A100-SXM4-40GB | 74 | 1491 | 828 / 772 / 1307 | 1.21 | 1427 | 0.87 |
| OneJev-27B | 2x NVIDIA A100-SXM4-40GB | 27 | 1438 | 799 / 732 / 1306 | 1.25 | 1384 | 0.81 |
| OneJev-9B | 1x NVIDIA A100-SXM4-40GB | 14 | 609 | 338 / 307 / 558 | 2.96 | 1384 | 0.17 |
| OneJev-4B | 1x NVIDIA A100-SXM4-40GB | 12 | 517 | 287 / 266 / 467 | 3.48 | 1384 | 0.15 |
| Intern-Decision-4B | 1x NVIDIA A100-SXM4-40GB | 11 | 545 | 302 / 277 / 487 | 3.31 | 1497 | 0.15 |
| OmniJev-4B | 1x NVIDIA A100-SXM4-40GB | 26 | 701 | 389 / 381 / 413 | 2.57 | 1254 | 0.20 |
| Jev-Omni-12B | 1x NVIDIA A100-SXM4-40GB | 17 | 1099 | 610 / 608 / 632 | 1.64 | 2792 | 0.31 |

## Slices (accuracy / macro-F1, %)

| slice | n | JevAny-4B-RoboJudge | JevAny-4B | Qwen3.5-4B-zeroshot | JevAny-27B | Qwen3.8-27B-zeroshot | OneJev-27B | OneJev-9B | OneJev-4B | Intern-Decision-4B | OmniJev-4B | Jev-Omni-12B |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| hop_bin=large | 552 | 81.9 / 81.8 | 57.1 / 52.5 | 48.9 / 32.8 | 64.3 / 62.8 | 64.7 / 64.4 | 75.9 / 75.9 | 71.6 / 71.6 | 62.3 / 61.0 | 54.9 / 52.8 | 50.7 / 42.8 | 49.8 / 47.8 |
| hop_bin=medium | 500 | 76.0 / 75.9 | 52.4 / 39.9 | 52.4 / 34.4 | 59.6 / 53.7 | 59.4 / 56.0 | 64.2 / 64.1 | 59.4 / 58.9 | 56.2 / 52.3 | 55.2 / 49.2 | 51.4 / 40.4 | 52.8 / 48.9 |
| hop_bin=small | 748 | 59.8 / 59.7 | 49.7 / 39.6 | 48.9 / 33.1 | 48.9 / 44.5 | 49.1 / 44.6 | 46.4 / 45.9 | 46.0 / 45.7 | 46.1 / 44.5 | 46.8 / 43.5 | 47.9 / 42.4 | 46.8 / 44.7 |
| image_dataset=agibotworld | 200 | 62.5 / 62.2 | 51.0 / 36.3 | 50.0 / 33.3 | 50.5 / 39.6 | 54.0 / 46.7 | 55.0 / 54.6 | 53.0 / 52.3 | 52.5 / 46.9 | 49.5 / 42.7 | 47.5 / 35.9 | 52.5 / 50.2 |
| image_dataset=agilex_newdragon | 200 | 69.5 / 69.4 | 50.5 / 39.6 | 50.0 / 33.3 | 54.0 / 52.1 | 57.0 / 53.6 | 60.5 / 60.1 | 58.5 / 58.5 | 55.5 / 54.1 | 51.5 / 50.3 | 49.0 / 37.4 | 47.5 / 45.6 |
| image_dataset=droid_oxe | 200 | 73.0 / 72.9 | 52.0 / 39.1 | 50.0 / 33.3 | 56.5 / 50.6 | 60.0 / 59.3 | 63.0 / 63.0 | 61.0 / 61.0 | 54.0 / 49.1 | 51.5 / 40.8 | 49.5 / 40.7 | 52.0 / 49.6 |
| image_dataset=galaxea_r1lite | 200 | 64.5 / 64.2 | 51.5 / 47.2 | 50.0 / 33.3 | 60.0 / 59.2 | 60.5 / 60.5 | 67.5 / 66.7 | 60.5 / 60.5 | 51.0 / 49.4 | 49.0 / 40.9 | 51.0 / 43.7 | 45.0 / 39.6 |
| image_dataset=human_egodex | 200 | 68.0 / 68.0 | 55.5 / 52.7 | 49.5 / 34.0 | 52.5 / 45.7 | 53.5 / 49.1 | 56.0 / 55.6 | 54.5 / 54.0 | 56.0 / 55.4 | 57.5 / 57.1 | 50.0 / 46.7 | 46.5 / 34.0 |
| image_dataset=human_pika | 200 | 67.5 / 67.4 | 50.0 / 41.6 | 50.0 / 33.3 | 51.5 / 48.1 | 51.0 / 46.2 | 57.0 / 57.0 | 52.5 / 52.5 | 52.0 / 49.8 | 49.5 / 45.0 | 49.0 / 36.8 | 49.5 / 48.1 |
| image_dataset=libero_data | 200 | 84.0 / 84.0 | 59.0 / 51.2 | 50.0 / 33.3 | 69.5 / 67.0 | 69.5 / 68.0 | 68.5 / 68.4 | 65.0 / 65.0 | 61.0 / 58.6 | 53.5 / 48.4 | 51.0 / 45.8 | 53.0 / 49.9 |
| image_dataset=robocasa_data | 200 | 72.0 / 71.8 | 50.5 / 34.4 | 50.0 / 33.3 | 55.5 / 51.9 | 45.5 / 39.1 | 56.0 / 55.4 | 54.0 / 54.0 | 51.5 / 48.9 | 49.0 / 47.8 | 50.0 / 43.5 | 49.5 / 49.2 |
| image_dataset=robotwin2_agilex_part1 | 200 | 78.5 / 78.5 | 54.5 / 47.9 | 49.5 / 33.1 | 59.5 / 57.1 | 59.5 / 59.3 | 60.0 / 59.4 | 59.0 / 58.7 | 51.5 / 51.3 | 53.5 / 51.5 | 50.5 / 45.0 | 49.0 / 45.3 |

## Paired: JevAny-4B-RoboJudge − JevAny-4B

- accuracy: 18.3 pts, 95% CI [15.3, 21.5]
- macro-F1: 26.9 pts, 95% CI [24.0, 29.8]
- McNemar: JevAny-4B-RoboJudge only correct 584, JevAny-4B only correct 254, exact p = 1.062e-30