Hugging Face's logo

SimpleJev
/
JevAny-Robotics-4B-LoRA

Robotics
PEFT
Safetensors
jevany
lora
decision-model
pointer
reward-model
judge
Model card Files
xet
 Community
JevAny-Robotics-4B-LoRA / eval /comparison /roboreward.md
HuanxinSheng's picture
HuanxinSheng
model card: full baseline comparison (same harness) + paired CIs; add eval/comparison reports
a52d6ed verified
preview code
|
Raw History Blame Contribute Delete
12.6 kB

RoboRewardBench (roboreward-v1): 2831/2831 items scored by every run

model Overall (group-wise MAE) RoboArena OXE subsets micro MAE median-label MAE expected-score MAE exact acc within-1 Spearman QWK mean pred
JevAny-4B-RoboJudge 0.443 0.687 0.432 0.523 0.419 0.458 62.1% 89.0% 0.827 0.788 2.82
JevAny-4B 1.057 1.110 1.055 1.090 0.769 0.835 40.8% 69.0% 0.684 0.528 2.24
Qwen3.5-4B-zeroshot 1.261 1.325 1.258 1.297 0.869 0.941 34.9% 63.9% 0.653 0.369 1.78
JevAny-27B 0.692 0.867 0.684 0.758 0.614 0.751 51.4% 82.7% 0.754 0.667 2.55
Qwen3.8-27B-zeroshot 0.992 1.106 0.987 1.046 0.792 0.771 41.4% 70.7% 0.741 0.557 2.12
OneJev-27B 0.805 0.919 0.800 0.856 0.705 0.764 47.3% 77.5% 0.726 0.657 2.53
OneJev-9B 0.927 1.042 0.922 0.980 0.788 0.822 43.4% 73.3% 0.682 0.589 2.75
OneJev-4B 1.005 1.103 1.001 1.053 0.832 0.884 41.7% 69.8% 0.663 0.562 2.84
Intern-Decision-4B 0.820 0.918 0.815 0.864 0.703 0.821 44.2% 78.7% 0.716 0.637 2.65
OmniJev-4B 0.811 0.941 0.805 0.853 0.836 0.935 41.7% 79.0% 0.627 0.590 2.54
Jev-Omni-12B 1.085 1.219 1.079 1.149 0.943 0.970 36.1% 69.7% 0.529 0.411 1.94

Leaderboard (paper Table 1/7; Overall / RoboArena / OXE): RoboReward-8B 0.665 / 0.768 / 0.660; GPT-5 mini 0.691 / 0.862 / 0.683; GPT-5 0.811 / 1.028 / 0.801; RoboReward-4B 0.845 / 0.806 / 0.847; Gemini 3 Pro (preview) 0.851 / 1.234 / 0.833; Qwen3-VL-8B Instruct 0.892 / 0.847 / 0.894; Gemini 2.5 Pro 0.902 / 0.936 / 0.900; Qwen3-VL-30B Instruct 0.903; Gemini Robotics-ER 1.5 0.906 / 1.002 / 0.902; Qwen3-VL-4B Instruct 1.032 / 0.929 / 1.036; Qwen2.5-VL-72B Instruct 0.991

JevAny-4B-RoboJudge: confusion (rows = true 1-5, cols = predicted 1-5)

true pred 1 pred 2 pred 3 pred 4 pred 5 n mean pred
1 573 84 23 5 4 689 1.23
2 94 322 153 42 30 641 2.36
3 40 76 304 89 47 556 3.05
4 29 27 110 162 90 418 3.61
5 7 12 44 67 397 527 4.58

JevAny-4B: confusion (rows = true 1-5, cols = predicted 1-5)

true pred 1 pred 2 pred 3 pred 4 pred 5 n mean pred
1 672 0 3 0 14 689 1.09
2 518 6 38 0 79 641 1.62
3 321 7 117 5 106 556 2.22
4 181 2 49 3 183 418 3.01
5 120 3 48 0 356 527 3.89

Qwen3.5-4B-zeroshot: confusion (rows = true 1-5, cols = predicted 1-5)

true pred 1 pred 2 pred 3 pred 4 pred 5 n mean pred
1 683 0 0 3 3 689 1.03
2 594 1 4 11 31 641 1.26
3 395 0 28 76 57 556 1.92
4 267 0 19 74 58 418 2.18
5 250 0 5 71 201 527 2.95

JevAny-27B: confusion (rows = true 1-5, cols = predicted 1-5)

true pred 1 pred 2 pred 3 pred 4 pred 5 n mean pred
1 597 56 0 2 34 689 1.29
2 292 241 51 8 49 641 1.88
3 110 179 173 31 63 556 2.56
4 50 68 100 58 142 418 3.42
5 39 34 33 34 387 527 4.32

Qwen3.8-27B-zeroshot: confusion (rows = true 1-5, cols = predicted 1-5)

true pred 1 pred 2 pred 3 pred 4 pred 5 n mean pred
1 674 2 0 0 13 689 1.08
2 570 21 12 0 38 641 1.31
3 355 21 93 0 87 556 2.00
4 176 14 71 2 155 418 2.87
5 113 7 26 0 381 527 4.00

OneJev-27B: confusion (rows = true 1-5, cols = predicted 1-5)

true pred 1 pred 2 pred 3 pred 4 pred 5 n mean pred
1 631 4 25 1 28 689 1.25
2 440 73 72 5 51 641 1.68
3 221 55 157 20 103 556 2.51
4 89 42 61 38 188 418 3.46
5 40 17 15 15 440 527 4.51

OneJev-9B: confusion (rows = true 1-5, cols = predicted 1-5)

true pred 1 pred 2 pred 3 pred 4 pred 5 n mean pred
1 592 10 15 2 70 689 1.47
2 383 104 36 4 114 641 2.00
3 166 92 86 37 175 556 2.93
4 79 47 60 20 212 418 3.57
5 31 22 30 16 428 527 4.50

OneJev-4B: confusion (rows = true 1-5, cols = predicted 1-5)

true pred 1 pred 2 pred 3 pred 4 pred 5 n mean pred
1 572 1 35 1 80 689 1.57
2 411 32 55 10 133 641 2.10
3 212 13 102 22 207 556 3.00
4 99 8 60 21 230 418 3.66
5 37 9 24 3 454 527 4.57

Intern-Decision-4B: confusion (rows = true 1-5, cols = predicted 1-5)

true pred 1 pred 2 pred 3 pred 4 pred 5 n mean pred
1 614 11 19 29 16 689 1.29
2 369 45 102 80 45 641 2.04
3 164 27 124 172 69 556 2.92
4 62 20 65 180 91 418 3.52
5 43 12 43 140 289 527 4.18

OmniJev-4B: confusion (rows = true 1-5, cols = predicted 1-5)

true pred 1 pred 2 pred 3 pred 4 pred 5 n mean pred
1 492 64 115 10 8 689 1.52
2 252 115 223 27 24 641 2.15
3 90 70 315 53 28 556 2.75
4 41 30 218 108 21 418 3.09
5 27 25 170 154 151 527 3.72

Jev-Omni-12B: confusion (rows = true 1-5, cols = predicted 1-5)

true pred 1 pred 2 pred 3 pred 4 pred 5 n mean pred
1 620 46 0 12 11 689 1.18
2 432 150 3 44 12 641 1.52
3 260 187 18 71 20 556 1.93
4 132 118 7 119 42 418 2.57
5 160 88 1 162 116 527 2.97

Micro MAE / exact acc by item type

model robo_arena (n=1000) oxe_success (n=371) oxe_relabelled (n=669) oxe_clipped (n=791)
JevAny-4B-RoboJudge 0.687 / 51.7% 0.358 / 76.5% 0.297 / 77.3% 0.584 / 55.6%
JevAny-4B 1.110 / 38.1% 1.124 / 67.1% 1.021 / 42.9% 1.106 / 30.0%
Qwen3.5-4B-zeroshot 1.325 / 33.4% 1.995 / 36.7% 0.943 / 43.6% 1.235 / 28.4%
JevAny-27B 0.867 / 46.0% 0.698 / 72.8% 0.713 / 51.3% 0.686 / 48.4%
Qwen3.8-27B-zeroshot 1.106 / 38.7% 1.046 / 71.4% 0.933 / 42.2% 1.067 / 30.0%
OneJev-27B 0.919 / 44.0% 0.528 / 81.9% 0.842 / 46.8% 0.942 / 35.7%
OneJev-9B 1.042 / 40.7% 0.601 / 78.2% 1.109 / 39.2% 0.971 / 34.3%
OneJev-4B 1.103 / 39.2% 0.485 / 84.1% 1.254 / 34.8% 1.086 / 30.8%
Intern-Decision-4B 0.918 / 41.6% 0.803 / 53.4% 0.843 / 46.9% 0.843 / 41.0%
OmniJev-4B 0.941 / 36.9% 1.199 / 29.4% 0.794 / 46.3% 0.630 / 49.7%
Jev-Omni-12B 1.219 / 34.7% 1.916 / 22.9% 0.780 / 48.1% 1.011 / 34.0%

Per-subset MAE (argmax)

subset n JevAny-4B-RoboJudge JevAny-4B Qwen3.5-4B-zeroshot JevAny-27B Qwen3.8-27B-zeroshot OneJev-27B OneJev-9B OneJev-4B Intern-Decision-4B OmniJev-4B Jev-Omni-12B
robo_arena 1000 0.687 1.110 1.325 0.867 1.106 0.919 1.042 1.103 0.918 0.941 1.219
austin_sirius_dataset_converted_externally_to_rlds 87 0.414 0.448 0.839 0.437 0.678 0.770 0.770 1.138 0.563 0.563 0.920
berkeley_autolab_ur5 100 0.360 1.390 1.720 0.930 1.480 1.160 1.120 1.200 0.600 0.780 0.890
berkeley_fanuc_manipulation 91 0.220 1.352 1.626 0.967 1.495 1.143 0.945 1.022 1.253 1.077 1.319
berkeley_mvp_converted_externally_to_rlds 71 0.099 0.577 0.634 0.183 0.746 0.380 0.380 0.437 0.606 0.704 0.859
berkeley_rpt_converted_externally_to_rlds 86 0.826 0.605 1.430 0.500 0.930 0.733 1.267 1.302 1.140 1.314 1.140
bridge 100 0.240 1.010 0.900 0.510 0.880 0.820 0.910 1.020 0.830 0.640 1.570
cmu_play_fusion 92 0.413 1.500 1.522 1.163 1.489 1.087 1.326 1.337 1.098 0.913 1.457
dlr_edan_shared_control_converted_externally_to_rlds 19 0.000 0.579 0.684 0.474 0.421 0.474 0.421 0.368 0.263 0.842 0.421
droid 100 0.710 1.280 1.710 0.880 1.160 0.810 1.040 1.210 0.920 0.980 1.420
fractal20220817_data 100 0.240 1.100 1.090 0.600 0.770 0.830 0.870 1.160 0.720 0.750 1.250
iamlab_cmu_pickup_insert_converted_externally_to_rlds 83 0.133 1.024 1.386 0.313 0.831 0.675 0.687 0.807 0.855 0.867 1.470
jaco_play 100 0.270 1.280 1.720 0.660 0.980 0.870 1.060 1.130 0.910 0.740 1.530
kaist_nonprehensile_converted_externally_to_rlds 53 0.415 1.377 1.585 0.736 1.094 0.792 0.736 0.698 0.887 0.925 1.321
roboturk 97 0.505 0.701 1.031 0.546 0.670 0.577 0.845 0.711 0.753 0.722 0.588
stanford_hydra_dataset_converted_externally_to_rlds 91 0.923 1.297 0.989 0.736 0.879 0.560 0.989 0.648 0.780 0.637 1.396
taco_play 91 0.198 1.626 1.824 1.429 1.813 1.484 1.286 1.396 1.253 1.374 1.121
tokyo_u_lsmo_converted_externally_to_rlds 71 0.549 1.761 1.930 0.563 1.366 0.704 0.859 0.718 0.803 0.577 1.268
ucsd_kitchen_dataset_converted_externally_to_rlds 95 0.337 1.042 1.232 0.495 0.863 0.832 0.916 0.947 0.726 0.684 0.863
ucsd_pick_and_place_dataset_converted_externally_to_rlds 100 0.150 0.740 0.660 0.500 0.730 0.560 0.600 0.700 0.460 0.420 0.570
utokyo_pr2_tabletop_manipulation_converted_externally_to_rlds 73 0.288 0.671 0.877 0.562 0.521 0.397 0.589 0.781 0.808 0.671 0.658
utokyo_xarm_bimanual_converted_externally_to_rlds 71 1.451 1.408 1.479 1.606 1.254 1.324 1.662 1.521 1.056 1.085 0.746
viola 60 0.767 0.433 0.800 0.267 0.667 0.617 1.000 1.767 0.650 0.450 0.967

Paired: Overall MAE JevAny-4B-RoboJudge - JevAny-4B = -0.614, 95% cluster-bootstrap CI [-0.725, -0.554]; per-item |err| smaller for JevAny-4B-RoboJudge on 1135, larger on 253, tie 1443 Paired: Overall MAE JevAny-4B-RoboJudge - Qwen3.5-4B-zeroshot = -0.817, 95% cluster-bootstrap CI [-0.875, -0.732]; per-item |err| smaller for JevAny-4B-RoboJudge on 1338, larger on 226, tie 1267 Paired: Overall MAE JevAny-4B-RoboJudge - JevAny-27B = -0.249, 95% cluster-bootstrap CI [-0.374, -0.220]; per-item |err| smaller for JevAny-4B-RoboJudge on 842, larger on 447, tie 1542 Paired: Overall MAE JevAny-4B-RoboJudge - Qwen3.8-27B-zeroshot = -0.549, 95% cluster-bootstrap CI [-0.656, -0.512]; per-item |err| smaller for JevAny-4B-RoboJudge on 1126, larger on 289, tie 1416 Paired: Overall MAE JevAny-4B-RoboJudge - OneJev-27B = -0.362, 95% cluster-bootstrap CI [-0.488, -0.336]; per-item |err| smaller for JevAny-4B-RoboJudge on 950, larger on 393, tie 1488 Paired: Overall MAE JevAny-4B-RoboJudge - OneJev-9B = -0.484, 95% cluster-bootstrap CI [-0.714, -0.456]; per-item |err| smaller for JevAny-4B-RoboJudge on 1059, larger on 323, tie 1449 Paired: Overall MAE JevAny-4B-RoboJudge - OneJev-4B = -0.562, 95% cluster-bootstrap CI [-0.811, -0.541]; per-item |err| smaller for JevAny-4B-RoboJudge on 1115, larger on 297, tie 1419 Paired: Overall MAE JevAny-4B-RoboJudge - Intern-Decision-4B = -0.376, 95% cluster-bootstrap CI [-0.518, -0.341]; per-item |err| smaller for JevAny-4B-RoboJudge on 993, larger on 385, tie 1453 Paired: Overall MAE JevAny-4B-RoboJudge - OmniJev-4B = -0.368, 95% cluster-bootstrap CI [-0.535, -0.320]; per-item |err| smaller for JevAny-4B-RoboJudge on 1034, larger on 440, tie 1357 Paired: Overall MAE JevAny-4B-RoboJudge - Jev-Omni-12B = -0.642, 95% cluster-bootstrap CI [-0.699, -0.549]; per-item |err| smaller for JevAny-4B-RoboJudge on 1324, larger on 348, tie 1159