eval/comparison/robopulsepp.md

robopulsepp-v1: 7720 items

model	answered	acc (95% CI)	balanced acc	macro-F1	F1 pos / neg	AUROC	NLL	ECE	pred pos rate
JevAny-4B-RoboJudge	7720/7720	60.4 [57.9, 62.9]	65.9	55.8	70.0 / 41.6	0.728	0.649	0.095	51.0
JevAny-4B	7720/7720	69.9 [67.7, 72.0]	59.1	57.5	80.5 / 34.4	0.641	0.580	0.050	73.0
Qwen3.5-4B-zeroshot	7720/7720	81.1 [78.2, 84.0]	50.0	44.8	89.6 / 0.0	0.592	0.591	0.131	100.0
JevAny-27B	7720/7720	82.0 [79.2, 84.8]	55.7	55.9	89.8 / 22.0	0.692	0.448	0.047	95.8
Qwen3.8-27B-zeroshot	7720/7720	78.7 [75.9, 81.5]	55.2	55.6	87.6 / 23.5	0.569	0.536	0.119	91.0
OneJev-27B	7720/7720	48.0 [45.7, 50.3]	62.7	46.7	54.9 / 38.5	0.700	0.754	0.215	34.2
OneJev-9B	7720/7720	29.2 [26.6, 31.8]	54.0	28.9	24.4 / 33.4	0.564	1.142	0.457	12.6
OneJev-4B	7720/7720	47.0 [44.3, 49.6]	58.3	45.2	55.1 / 35.3	0.617	0.773	0.176	37.0
Intern-Decision-4B	7720/7720	65.8 [63.6, 68.0]	60.5	56.5	76.6 / 36.4	0.653	0.594	0.049	65.1
OmniJev-4B	7720/7720	63.9 [61.7, 66.0]	54.6	52.6	75.8 / 29.4	0.556	0.612	0.039	67.8
Jev-Omni-12B	7720/7720	58.0 [54.9, 61.0]	52.3	49.1	70.3 / 27.9	0.537	0.663	0.042	60.6

Timing

model	GPUs	load s	wall s (loop)	mean / median / p95 ms per item	items/s	median input tokens	GPU-hours
JevAny-4B-RoboJudge	1x NVIDIA A100-SXM4-40GB	4	2022	262 / 207 / 520	3.82	790	0.56
JevAny-4B	1x NVIDIA A100-SXM4-40GB	7	2034	263 / 210 / 515	3.80	790	0.57
Qwen3.5-4B-zeroshot	1x NVIDIA A100-SXM4-40GB	24	1792	232 / 186 / 470	4.31	899	0.50
JevAny-27B	2x NVIDIA A100-SXM4-40GB	60	5425	703 / 572 / 1422	1.42	790	3.05
Qwen3.8-27B-zeroshot	2x NVIDIA A100-SXM4-40GB	72	4739	614 / 514 / 1214	1.63	899	2.67
OneJev-27B	2x NVIDIA A100-SXM4-40GB	279	4607	597 / 483 / 1212	1.68	883	2.71
OneJev-9B	1x NVIDIA A100-SXM4-40GB	33	2054	266 / 217 / 538	3.76	883	0.58
OneJev-4B	1x NVIDIA A100-SXM4-40GB	25	1740	225 / 181 / 466	4.44	883	0.49
Intern-Decision-4B	1x NVIDIA A100-SXM4-40GB	26	1477	233 / 188 / 461	4.29	964	0.42
OmniJev-4B	1x NVIDIA A100-SXM4-40GB	50	2286	296 / 285 / 379	3.38	570	0.65
Jev-Omni-12B	1x NVIDIA A100-SXM4-40GB	39	1771	229 / 174 / 432	4.36	769	0.50

Interval level (majority of kept pairs)

model	intervals	acc (95% CI)	macro-F1
JevAny-4B-RoboJudge	1062	67.6 [64.6, 70.4]	65.6
JevAny-4B	1062	67.3 [64.8, 70.0]	58.7
Qwen3.5-4B-zeroshot	1062	71.1 [69.0, 73.2]	41.6
JevAny-27B	1062	75.4 [73.3, 77.7]	56.4
Qwen3.8-27B-zeroshot	1062	74.2 [72.0, 76.6]	58.1
OneJev-27B	1062	51.9 [49.1, 54.7]	51.9
OneJev-9B	1062	32.3 [29.9, 34.6]	28.6
OneJev-4B	1062	50.1 [47.3, 53.0]	49.9
Intern-Decision-4B	1062	68.6 [66.1, 71.3]	62.1
OmniJev-4B	1062	64.4 [61.7, 67.1]	52.6
Jev-Omni-12B	1062	56.0 [53.4, 58.6]	50.5

Slices (accuracy / macro-F1, %)

slice	n	JevAny-4B-RoboJudge	JevAny-4B	Qwen3.5-4B-zeroshot	JevAny-27B	Qwen3.8-27B-zeroshot	OneJev-27B	OneJev-9B	OneJev-4B	Intern-Decision-4B	OmniJev-4B	Jev-Omni-12B
n_views=1	5097	65.9 / 60.2	70.0 / 58.6	80.8 / 44.7	82.5 / 59.5	77.1 / 58.2	49.0 / 47.6	28.0 / 27.5	50.6 / 47.7	63.6 / 55.0	60.2 / 49.6	55.2 / 47.6
n_views=2	1693	45.4 / 44.8	66.6 / 52.3	79.0 / 44.1	77.9 / 45.1	78.7 / 44.6	48.6 / 47.9	33.7 / 33.6	43.0 / 42.7	68.3 / 60.4	70.7 / 60.4	58.5 / 51.8
n_views=3	930	57.2 / 49.3	75.5 / 59.9	87.0 / 46.5	86.6 / 53.3	87.1 / 47.4	41.3 / 39.5	27.2 / 27.2	34.6 / 33.9	72.9 / 56.4	71.9 / 54.2	72.2 / 45.7

Paired: JevAny-4B-RoboJudge − JevAny-4B

• accuracy: -9.6 pts, 95% CI [-12.6, -6.2]
• macro-F1: -1.7 pts, 95% CI [-4.8, 1.8]
• McNemar: JevAny-4B-RoboJudge only correct 995, JevAny-4B only correct 1733, exact p = 9.384e-46