Hugging Face's logo

SimpleJev
/
JevAny-Robotics-4B-LoRA

Robotics
PEFT
Safetensors
jevany
lora
decision-model
pointer
reward-model
judge
Model card Files
xet
 Community
JevAny-Robotics-4B-LoRA / eval /comparison /roboreward.md
HuanxinSheng's picture
HuanxinSheng
model card: full baseline comparison (same harness) + paired CIs; add eval/comparison reports
a52d6ed verified
preview code
|
Raw History Blame Contribute Delete
12.6 kB
# RoboRewardBench (roboreward-v1): 2831/2831 items scored by every run
| model | Overall (group-wise MAE) | RoboArena | OXE subsets | micro MAE | median-label MAE | expected-score MAE | exact acc | within-1 | Spearman | QWK | mean pred |
|---|---|---|---|---|---|---|---|---|---|---|---|
| JevAny-4B-RoboJudge | **0.443** | 0.687 | 0.432 | 0.523 | 0.419 | 0.458 | 62.1% | 89.0% | 0.827 | 0.788 | 2.82 |
| JevAny-4B | **1.057** | 1.110 | 1.055 | 1.090 | 0.769 | 0.835 | 40.8% | 69.0% | 0.684 | 0.528 | 2.24 |
| Qwen3.5-4B-zeroshot | **1.261** | 1.325 | 1.258 | 1.297 | 0.869 | 0.941 | 34.9% | 63.9% | 0.653 | 0.369 | 1.78 |
| JevAny-27B | **0.692** | 0.867 | 0.684 | 0.758 | 0.614 | 0.751 | 51.4% | 82.7% | 0.754 | 0.667 | 2.55 |
| Qwen3.8-27B-zeroshot | **0.992** | 1.106 | 0.987 | 1.046 | 0.792 | 0.771 | 41.4% | 70.7% | 0.741 | 0.557 | 2.12 |
| OneJev-27B | **0.805** | 0.919 | 0.800 | 0.856 | 0.705 | 0.764 | 47.3% | 77.5% | 0.726 | 0.657 | 2.53 |
| OneJev-9B | **0.927** | 1.042 | 0.922 | 0.980 | 0.788 | 0.822 | 43.4% | 73.3% | 0.682 | 0.589 | 2.75 |
| OneJev-4B | **1.005** | 1.103 | 1.001 | 1.053 | 0.832 | 0.884 | 41.7% | 69.8% | 0.663 | 0.562 | 2.84 |
| Intern-Decision-4B | **0.820** | 0.918 | 0.815 | 0.864 | 0.703 | 0.821 | 44.2% | 78.7% | 0.716 | 0.637 | 2.65 |
| OmniJev-4B | **0.811** | 0.941 | 0.805 | 0.853 | 0.836 | 0.935 | 41.7% | 79.0% | 0.627 | 0.590 | 2.54 |
| Jev-Omni-12B | **1.085** | 1.219 | 1.079 | 1.149 | 0.943 | 0.970 | 36.1% | 69.7% | 0.529 | 0.411 | 1.94 |
Leaderboard (paper Table 1/7; Overall / RoboArena / OXE): RoboReward-8B 0.665 / 0.768 / 0.660; GPT-5 mini 0.691 / 0.862 / 0.683; GPT-5 0.811 / 1.028 / 0.801; RoboReward-4B 0.845 / 0.806 / 0.847; Gemini 3 Pro (preview) 0.851 / 1.234 / 0.833; Qwen3-VL-8B Instruct 0.892 / 0.847 / 0.894; Gemini 2.5 Pro 0.902 / 0.936 / 0.900; Qwen3-VL-30B Instruct 0.903; Gemini Robotics-ER 1.5 0.906 / 1.002 / 0.902; Qwen3-VL-4B Instruct 1.032 / 0.929 / 1.036; Qwen2.5-VL-72B Instruct 0.991
## JevAny-4B-RoboJudge: confusion (rows = true 1-5, cols = predicted 1-5)
| true | pred 1 | pred 2 | pred 3 | pred 4 | pred 5 | n | mean pred |
|---|---|---|---|---|---|---|---|
| 1 | 573 | 84 | 23 | 5 | 4 | 689 | 1.23 |
| 2 | 94 | 322 | 153 | 42 | 30 | 641 | 2.36 |
| 3 | 40 | 76 | 304 | 89 | 47 | 556 | 3.05 |
| 4 | 29 | 27 | 110 | 162 | 90 | 418 | 3.61 |
| 5 | 7 | 12 | 44 | 67 | 397 | 527 | 4.58 |
## JevAny-4B: confusion (rows = true 1-5, cols = predicted 1-5)
| true | pred 1 | pred 2 | pred 3 | pred 4 | pred 5 | n | mean pred |
|---|---|---|---|---|---|---|---|
| 1 | 672 | 0 | 3 | 0 | 14 | 689 | 1.09 |
| 2 | 518 | 6 | 38 | 0 | 79 | 641 | 1.62 |
| 3 | 321 | 7 | 117 | 5 | 106 | 556 | 2.22 |
| 4 | 181 | 2 | 49 | 3 | 183 | 418 | 3.01 |
| 5 | 120 | 3 | 48 | 0 | 356 | 527 | 3.89 |
## Qwen3.5-4B-zeroshot: confusion (rows = true 1-5, cols = predicted 1-5)
| true | pred 1 | pred 2 | pred 3 | pred 4 | pred 5 | n | mean pred |
|---|---|---|---|---|---|---|---|
| 1 | 683 | 0 | 0 | 3 | 3 | 689 | 1.03 |
| 2 | 594 | 1 | 4 | 11 | 31 | 641 | 1.26 |
| 3 | 395 | 0 | 28 | 76 | 57 | 556 | 1.92 |
| 4 | 267 | 0 | 19 | 74 | 58 | 418 | 2.18 |
| 5 | 250 | 0 | 5 | 71 | 201 | 527 | 2.95 |
## JevAny-27B: confusion (rows = true 1-5, cols = predicted 1-5)
| true | pred 1 | pred 2 | pred 3 | pred 4 | pred 5 | n | mean pred |
|---|---|---|---|---|---|---|---|
| 1 | 597 | 56 | 0 | 2 | 34 | 689 | 1.29 |
| 2 | 292 | 241 | 51 | 8 | 49 | 641 | 1.88 |
| 3 | 110 | 179 | 173 | 31 | 63 | 556 | 2.56 |
| 4 | 50 | 68 | 100 | 58 | 142 | 418 | 3.42 |
| 5 | 39 | 34 | 33 | 34 | 387 | 527 | 4.32 |
## Qwen3.8-27B-zeroshot: confusion (rows = true 1-5, cols = predicted 1-5)
| true | pred 1 | pred 2 | pred 3 | pred 4 | pred 5 | n | mean pred |
|---|---|---|---|---|---|---|---|
| 1 | 674 | 2 | 0 | 0 | 13 | 689 | 1.08 |
| 2 | 570 | 21 | 12 | 0 | 38 | 641 | 1.31 |
| 3 | 355 | 21 | 93 | 0 | 87 | 556 | 2.00 |
| 4 | 176 | 14 | 71 | 2 | 155 | 418 | 2.87 |
| 5 | 113 | 7 | 26 | 0 | 381 | 527 | 4.00 |
## OneJev-27B: confusion (rows = true 1-5, cols = predicted 1-5)
| true | pred 1 | pred 2 | pred 3 | pred 4 | pred 5 | n | mean pred |
|---|---|---|---|---|---|---|---|
| 1 | 631 | 4 | 25 | 1 | 28 | 689 | 1.25 |
| 2 | 440 | 73 | 72 | 5 | 51 | 641 | 1.68 |
| 3 | 221 | 55 | 157 | 20 | 103 | 556 | 2.51 |
| 4 | 89 | 42 | 61 | 38 | 188 | 418 | 3.46 |
| 5 | 40 | 17 | 15 | 15 | 440 | 527 | 4.51 |
## OneJev-9B: confusion (rows = true 1-5, cols = predicted 1-5)
| true | pred 1 | pred 2 | pred 3 | pred 4 | pred 5 | n | mean pred |
|---|---|---|---|---|---|---|---|
| 1 | 592 | 10 | 15 | 2 | 70 | 689 | 1.47 |
| 2 | 383 | 104 | 36 | 4 | 114 | 641 | 2.00 |
| 3 | 166 | 92 | 86 | 37 | 175 | 556 | 2.93 |
| 4 | 79 | 47 | 60 | 20 | 212 | 418 | 3.57 |
| 5 | 31 | 22 | 30 | 16 | 428 | 527 | 4.50 |
## OneJev-4B: confusion (rows = true 1-5, cols = predicted 1-5)
| true | pred 1 | pred 2 | pred 3 | pred 4 | pred 5 | n | mean pred |
|---|---|---|---|---|---|---|---|
| 1 | 572 | 1 | 35 | 1 | 80 | 689 | 1.57 |
| 2 | 411 | 32 | 55 | 10 | 133 | 641 | 2.10 |
| 3 | 212 | 13 | 102 | 22 | 207 | 556 | 3.00 |
| 4 | 99 | 8 | 60 | 21 | 230 | 418 | 3.66 |
| 5 | 37 | 9 | 24 | 3 | 454 | 527 | 4.57 |
## Intern-Decision-4B: confusion (rows = true 1-5, cols = predicted 1-5)
| true | pred 1 | pred 2 | pred 3 | pred 4 | pred 5 | n | mean pred |
|---|---|---|---|---|---|---|---|
| 1 | 614 | 11 | 19 | 29 | 16 | 689 | 1.29 |
| 2 | 369 | 45 | 102 | 80 | 45 | 641 | 2.04 |
| 3 | 164 | 27 | 124 | 172 | 69 | 556 | 2.92 |
| 4 | 62 | 20 | 65 | 180 | 91 | 418 | 3.52 |
| 5 | 43 | 12 | 43 | 140 | 289 | 527 | 4.18 |
## OmniJev-4B: confusion (rows = true 1-5, cols = predicted 1-5)
| true | pred 1 | pred 2 | pred 3 | pred 4 | pred 5 | n | mean pred |
|---|---|---|---|---|---|---|---|
| 1 | 492 | 64 | 115 | 10 | 8 | 689 | 1.52 |
| 2 | 252 | 115 | 223 | 27 | 24 | 641 | 2.15 |
| 3 | 90 | 70 | 315 | 53 | 28 | 556 | 2.75 |
| 4 | 41 | 30 | 218 | 108 | 21 | 418 | 3.09 |
| 5 | 27 | 25 | 170 | 154 | 151 | 527 | 3.72 |
## Jev-Omni-12B: confusion (rows = true 1-5, cols = predicted 1-5)
| true | pred 1 | pred 2 | pred 3 | pred 4 | pred 5 | n | mean pred |
|---|---|---|---|---|---|---|---|
| 1 | 620 | 46 | 0 | 12 | 11 | 689 | 1.18 |
| 2 | 432 | 150 | 3 | 44 | 12 | 641 | 1.52 |
| 3 | 260 | 187 | 18 | 71 | 20 | 556 | 1.93 |
| 4 | 132 | 118 | 7 | 119 | 42 | 418 | 2.57 |
| 5 | 160 | 88 | 1 | 162 | 116 | 527 | 2.97 |
## Micro MAE / exact acc by item type
| model | robo_arena (n=1000) | oxe_success (n=371) | oxe_relabelled (n=669) | oxe_clipped (n=791) |
|---|---|---|---|---|
| JevAny-4B-RoboJudge | 0.687 / 51.7% | 0.358 / 76.5% | 0.297 / 77.3% | 0.584 / 55.6% |
| JevAny-4B | 1.110 / 38.1% | 1.124 / 67.1% | 1.021 / 42.9% | 1.106 / 30.0% |
| Qwen3.5-4B-zeroshot | 1.325 / 33.4% | 1.995 / 36.7% | 0.943 / 43.6% | 1.235 / 28.4% |
| JevAny-27B | 0.867 / 46.0% | 0.698 / 72.8% | 0.713 / 51.3% | 0.686 / 48.4% |
| Qwen3.8-27B-zeroshot | 1.106 / 38.7% | 1.046 / 71.4% | 0.933 / 42.2% | 1.067 / 30.0% |
| OneJev-27B | 0.919 / 44.0% | 0.528 / 81.9% | 0.842 / 46.8% | 0.942 / 35.7% |
| OneJev-9B | 1.042 / 40.7% | 0.601 / 78.2% | 1.109 / 39.2% | 0.971 / 34.3% |
| OneJev-4B | 1.103 / 39.2% | 0.485 / 84.1% | 1.254 / 34.8% | 1.086 / 30.8% |
| Intern-Decision-4B | 0.918 / 41.6% | 0.803 / 53.4% | 0.843 / 46.9% | 0.843 / 41.0% |
| OmniJev-4B | 0.941 / 36.9% | 1.199 / 29.4% | 0.794 / 46.3% | 0.630 / 49.7% |
| Jev-Omni-12B | 1.219 / 34.7% | 1.916 / 22.9% | 0.780 / 48.1% | 1.011 / 34.0% |
## Per-subset MAE (argmax)
| subset | n | JevAny-4B-RoboJudge | JevAny-4B | Qwen3.5-4B-zeroshot | JevAny-27B | Qwen3.8-27B-zeroshot | OneJev-27B | OneJev-9B | OneJev-4B | Intern-Decision-4B | OmniJev-4B | Jev-Omni-12B |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| robo_arena | 1000 | 0.687 | 1.110 | 1.325 | 0.867 | 1.106 | 0.919 | 1.042 | 1.103 | 0.918 | 0.941 | 1.219 |
| austin_sirius_dataset_converted_externally_to_rlds | 87 | 0.414 | 0.448 | 0.839 | 0.437 | 0.678 | 0.770 | 0.770 | 1.138 | 0.563 | 0.563 | 0.920 |
| berkeley_autolab_ur5 | 100 | 0.360 | 1.390 | 1.720 | 0.930 | 1.480 | 1.160 | 1.120 | 1.200 | 0.600 | 0.780 | 0.890 |
| berkeley_fanuc_manipulation | 91 | 0.220 | 1.352 | 1.626 | 0.967 | 1.495 | 1.143 | 0.945 | 1.022 | 1.253 | 1.077 | 1.319 |
| berkeley_mvp_converted_externally_to_rlds | 71 | 0.099 | 0.577 | 0.634 | 0.183 | 0.746 | 0.380 | 0.380 | 0.437 | 0.606 | 0.704 | 0.859 |
| berkeley_rpt_converted_externally_to_rlds | 86 | 0.826 | 0.605 | 1.430 | 0.500 | 0.930 | 0.733 | 1.267 | 1.302 | 1.140 | 1.314 | 1.140 |
| bridge | 100 | 0.240 | 1.010 | 0.900 | 0.510 | 0.880 | 0.820 | 0.910 | 1.020 | 0.830 | 0.640 | 1.570 |
| cmu_play_fusion | 92 | 0.413 | 1.500 | 1.522 | 1.163 | 1.489 | 1.087 | 1.326 | 1.337 | 1.098 | 0.913 | 1.457 |
| dlr_edan_shared_control_converted_externally_to_rlds | 19 | 0.000 | 0.579 | 0.684 | 0.474 | 0.421 | 0.474 | 0.421 | 0.368 | 0.263 | 0.842 | 0.421 |
| droid | 100 | 0.710 | 1.280 | 1.710 | 0.880 | 1.160 | 0.810 | 1.040 | 1.210 | 0.920 | 0.980 | 1.420 |
| fractal20220817_data | 100 | 0.240 | 1.100 | 1.090 | 0.600 | 0.770 | 0.830 | 0.870 | 1.160 | 0.720 | 0.750 | 1.250 |
| iamlab_cmu_pickup_insert_converted_externally_to_rlds | 83 | 0.133 | 1.024 | 1.386 | 0.313 | 0.831 | 0.675 | 0.687 | 0.807 | 0.855 | 0.867 | 1.470 |
| jaco_play | 100 | 0.270 | 1.280 | 1.720 | 0.660 | 0.980 | 0.870 | 1.060 | 1.130 | 0.910 | 0.740 | 1.530 |
| kaist_nonprehensile_converted_externally_to_rlds | 53 | 0.415 | 1.377 | 1.585 | 0.736 | 1.094 | 0.792 | 0.736 | 0.698 | 0.887 | 0.925 | 1.321 |
| roboturk | 97 | 0.505 | 0.701 | 1.031 | 0.546 | 0.670 | 0.577 | 0.845 | 0.711 | 0.753 | 0.722 | 0.588 |
| stanford_hydra_dataset_converted_externally_to_rlds | 91 | 0.923 | 1.297 | 0.989 | 0.736 | 0.879 | 0.560 | 0.989 | 0.648 | 0.780 | 0.637 | 1.396 |
| taco_play | 91 | 0.198 | 1.626 | 1.824 | 1.429 | 1.813 | 1.484 | 1.286 | 1.396 | 1.253 | 1.374 | 1.121 |
| tokyo_u_lsmo_converted_externally_to_rlds | 71 | 0.549 | 1.761 | 1.930 | 0.563 | 1.366 | 0.704 | 0.859 | 0.718 | 0.803 | 0.577 | 1.268 |
| ucsd_kitchen_dataset_converted_externally_to_rlds | 95 | 0.337 | 1.042 | 1.232 | 0.495 | 0.863 | 0.832 | 0.916 | 0.947 | 0.726 | 0.684 | 0.863 |
| ucsd_pick_and_place_dataset_converted_externally_to_rlds | 100 | 0.150 | 0.740 | 0.660 | 0.500 | 0.730 | 0.560 | 0.600 | 0.700 | 0.460 | 0.420 | 0.570 |
| utokyo_pr2_tabletop_manipulation_converted_externally_to_rlds | 73 | 0.288 | 0.671 | 0.877 | 0.562 | 0.521 | 0.397 | 0.589 | 0.781 | 0.808 | 0.671 | 0.658 |
| utokyo_xarm_bimanual_converted_externally_to_rlds | 71 | 1.451 | 1.408 | 1.479 | 1.606 | 1.254 | 1.324 | 1.662 | 1.521 | 1.056 | 1.085 | 0.746 |
| viola | 60 | 0.767 | 0.433 | 0.800 | 0.267 | 0.667 | 0.617 | 1.000 | 1.767 | 0.650 | 0.450 | 0.967 |
Paired: Overall MAE JevAny-4B-RoboJudge - JevAny-4B = -0.614, 95% cluster-bootstrap CI [-0.725, -0.554]; per-item |err| smaller for JevAny-4B-RoboJudge on 1135, larger on 253, tie 1443
Paired: Overall MAE JevAny-4B-RoboJudge - Qwen3.5-4B-zeroshot = -0.817, 95% cluster-bootstrap CI [-0.875, -0.732]; per-item |err| smaller for JevAny-4B-RoboJudge on 1338, larger on 226, tie 1267
Paired: Overall MAE JevAny-4B-RoboJudge - JevAny-27B = -0.249, 95% cluster-bootstrap CI [-0.374, -0.220]; per-item |err| smaller for JevAny-4B-RoboJudge on 842, larger on 447, tie 1542
Paired: Overall MAE JevAny-4B-RoboJudge - Qwen3.8-27B-zeroshot = -0.549, 95% cluster-bootstrap CI [-0.656, -0.512]; per-item |err| smaller for JevAny-4B-RoboJudge on 1126, larger on 289, tie 1416
Paired: Overall MAE JevAny-4B-RoboJudge - OneJev-27B = -0.362, 95% cluster-bootstrap CI [-0.488, -0.336]; per-item |err| smaller for JevAny-4B-RoboJudge on 950, larger on 393, tie 1488
Paired: Overall MAE JevAny-4B-RoboJudge - OneJev-9B = -0.484, 95% cluster-bootstrap CI [-0.714, -0.456]; per-item |err| smaller for JevAny-4B-RoboJudge on 1059, larger on 323, tie 1449
Paired: Overall MAE JevAny-4B-RoboJudge - OneJev-4B = -0.562, 95% cluster-bootstrap CI [-0.811, -0.541]; per-item |err| smaller for JevAny-4B-RoboJudge on 1115, larger on 297, tie 1419
Paired: Overall MAE JevAny-4B-RoboJudge - Intern-Decision-4B = -0.376, 95% cluster-bootstrap CI [-0.518, -0.341]; per-item |err| smaller for JevAny-4B-RoboJudge on 993, larger on 385, tie 1453
Paired: Overall MAE JevAny-4B-RoboJudge - OmniJev-4B = -0.368, 95% cluster-bootstrap CI [-0.535, -0.320]; per-item |err| smaller for JevAny-4B-RoboJudge on 1034, larger on 440, tie 1357
Paired: Overall MAE JevAny-4B-RoboJudge - Jev-Omni-12B = -0.642, 95% cluster-bootstrap CI [-0.699, -0.549]; per-item |err| smaller for JevAny-4B-RoboJudge on 1324, larger on 348, tie 1159