Model card: same-harness comparison with dedicated robot reward models; --media full usage

Adds RoboReward-8B/4B, Robometer-4B and Robo-Dopamine-GRM-2.0-8B (all run in our harness), calibration, policy ranking, speed and media-encoding results; removes leaderboard comparisons; usage now requires JevAny --media full.

README.md

@@ -10,27 +10,39 @@ tags:
 - robotics
 - reward-model
 - judge
-datasets:
-- SimpleJev/JevAny-Robotics-Train-v0.2
 ---
 
 # JevAny-Robotics-4B-LoRA (v0.2, SFT)
 
-A 4B robot judge / reward model: a robotics continuation of [`SimpleJev/JevAny-Qwen3.5-4B-LoRA`](https://huggingface.co/SimpleJev/JevAny-Qwen3.5-4B-LoRA)
-(pointer readout on [`Qwen/Qwen3.5-4B`](https://huggingface.co/Qwen/Qwen3.5-4B)), fine-tuned on [`SimpleJev/JevAny-Robotics-Train-v0.2`](https://huggingface.co/datasets/SimpleJev/JevAny-Robotics-Train-v0.2) — 184,942 judgement questions over
+A 4B Jev-style (System One) judge for robot manipulation: one forward pass returns a probability for every option. It is a robotics continuation of [`SimpleJev/JevAny-Qwen3.5-4B-LoRA`](https://huggingface.co/SimpleJev/JevAny-Qwen3.5-4B-LoRA)
+(pointer readout on [`Qwen/Qwen3.5-4B`](https://huggingface.co/Qwen/Qwen3.5-4B)), fine-tuned on JevAny-Robotics-Train-v0.2 (to be released) — 184,942 judgement questions over
 real and simulated robot images and videos (task progress, 1–5 rollout scores, execution success, failure mode, plan checking,
 failure localisation, trajectory preference), built only from train-side splits and decontaminated against every test set below.
 
-**Highlights** (all models run by us in the same harness, on the same items with the same media protocol):
-
-- **RoboPulse: 71.1% accuracy / 0.806 AUROC** — best of every judge we ran, including 27B models (OneJev-27B 60.4% / 0.651, JevAny-27B 56.6% / 0.627); +18.3 points over its parent.
-- **RoboRewardBench: MAE 0.443** — lower than every model we ran (OneJev-27B 0.805, JevAny-27B 0.692, Intern-Decision-4B 0.820) and than the best entry of the paper leaderboard (RoboReward-8B 0.665, their protocol); paired MAE difference vs the best 27B model −0.249 [−0.374, −0.220].
-- **RoboPulse++: best AUROC (0.728)** and balanced accuracy among 4B models (65.9 vs Intern-Decision-4B 60.5); at the level of the best 27B model (OneJev-27B 0.700 AUROC, paired +0.028 [−0.004, +0.063]).
-- **JevAny-Robotics-Public: skill_role 0.567, micro accuracy 80.4% (highest of all models)** — 4× its parent, +0.14 over OneJev-4B, statistically tied with the strongest 27B judge (OneJev-27B 0.605, paired −0.038 [−0.081, +0.006]).
-- Same 4B backbone and speed as the parent (median 161 ms per JevAny-Robotics-Public item on an A100).
+**Highlights** (every model below was run by us in the same harness, on the same items; dedicated robot reward models
+included, see [Dedicated robot reward models](#dedicated-robot-reward-models-same-harness)):
+
+- **No single model wins every benchmark; this one stays near the top on all of them.** Specialists win their own format:
+  Robo-Dopamine-GRM-2.0-8B on RoboPulse (90.1% vs 71.1%) and RoboReward-8B on RoboRewardBench (MAE 0.358 vs 0.443).
+- **Real robots, human ratings:** on RoboRewardBench's RoboArena split (1,000 real-robot rollouts) it has the lowest error of every model
+  we ran, 0.687 (best reward model: Robometer-4B 0.834, paired −0.147 [−0.245, −0.048]; RoboReward-8B 0.968; 27B judges 0.867–0.919).
+  When two policies attempted the same task, it orders them like the human raters 88.4% of the time (805 pairs; OneJev-27B 88.8%,
+  RoboReward-8B 85.3%, Robo-Dopamine 78.3%, Robometer-4B 75.5%).
+- **RoboPulse++: AUROC 0.728**, statistically tied with Robo-Dopamine (0.698–0.730 across its readouts) and the 27B judges (0.692–0.700);
+  above Robometer-4B (+0.098 [+0.062, +0.131]).
+- **Breadth:** it answers all 13 question families of JevAny-Robotics-Public (skill_role 0.567, micro accuracy 80.4%, tied with the
+  strongest 27B judge). The reward models can answer 6 of them; on success detection (5 sets) its mean AUROC is 0.872 vs 0.756 for the best
+  reward model (Robometer-4B, paired +0.116 [+0.088, +0.144]); OneJev-27B is higher (0.903).
+- **Calibrated probabilities:** on RoboRewardBench its probabilities have the lowest NLL of all models we ran, 1.02 vs 1.39 for
+  RoboReward-8B (paired −0.37 [−0.52, −0.22]) at similar exact-score accuracy (62.1% vs 63.5%). Robo-Dopamine and Robometer output no
+  per-option probabilities.
+- **Speed:** batch 1 on one A100, a median 185 ms per RoboPulse pair with the flash-linear-attention / causal-conv1d kernels (294 ms with
+  the reference kernels used for the tables) vs 316 ms for Robo-Dopamine-8B (vLLM) and 732–838 ms for 27B judges on two GPUs.
 
 ## Robot benchmarks
 
+### Jev-style judges and base VLMs
+
 | Model | Size | RoboPulse acc ↑ | RoboPulse AUROC ↑ | RoboPulse++ AUROC ↑ | RoboPulse++ bal. acc ↑ | RoboRewardBench MAE ↓ | JevAny-Robotics-Public skill_role ↑ |
 |---|---:|---:|---:|---:|---:|---:|---:|
 | **JevAny-Robotics-4B-LoRA (this model)** | 4B | **71.1** | **0.806** | **0.728** | **65.9** | **0.443** | 0.567 |
@@ -63,9 +75,51 @@ chance-corrected skill. *OmniJev-4B's training data overlaps RoboRewardBench. Ev
 Every difference in favour of this model is significant except RoboPulse++ AUROC and JevAny-Robotics-Public against the two 27B judges (ties).
 Per-benchmark reports with all slices are in `eval/comparison/`.
 
-For reference, published numbers under the benchmarks' own protocols: RoboRewardBench leaderboard — RoboReward-8B 0.665, GPT-5 mini 0.691,
-GPT-5 0.811, RoboReward-4B 0.845, Gemini 3 Pro (preview) 0.851, Qwen3-VL-8B 0.892; RoboPulse paper — Gemini 3 Pro Preview 0.66, Qwen3-VL-8B 0.59,
-Qwen3-VL-4B 0.56.
+### Dedicated robot reward models (same harness)
+
+| Model | Size | RoboPulse acc ↑ | RoboPulse AUROC ↑ | RoboPulse++ AUROC ↑ | RoboPulse++ bal. acc ↑ | RoboRewardBench MAE ↓ | RoboArena MAE ↓ |
+|---|---:|---:|---:|---:|---:|---:|---:|
+| **JevAny-Robotics-4B-LoRA (this model)** | 4B | 71.1 | 0.806 | **0.728** | 65.9 | 0.443 | **0.687** |
+| RoboReward-8B | 8B | 49.9ᵃ | 0.499ᵃ | 0.489ᵃ | 50.0ᵃ | **0.358** | 0.968 |
+| RoboReward-4B | 4B | 49.6ᵃ | 0.501ᵃ | 0.507ᵃ | 51.6ᵃ | 0.407 | 0.999 |
+| Robometer-4B | 4B | 63.6ᵇ | 0.700ᵇ | 0.630ᵇ | 58.6ᵇ | 0.777ᶜ | 0.834ᶜ |
+| Robo-Dopamine-GRM-2.0-8B | 8B | **90.1** | **0.953** | 0.717ᵈ | **68.6**ᵈ | 1.023ᵉ | 0.928ᵉ |
+
+How each model was run (all decisions fixed before scoring the test sets):
+- **RoboReward-8B / 4B** (`teetone/RoboReward-*`): model-card prompt verbatim, assistant turn prefilled with `ANSWER: `, softmax over the
+  digit tokens 1–5; every frame of the same derived clips. Free generation gives the same RoboRewardBench MAE (0.357 vs 0.358 for 8B,
+  0 unparsable). ᵃ RoboReward scores whole rollouts; for the pairwise benchmarks we score the 2-frame video (before, after) minus
+  (after, before). It was not trained for pairs, so treat these as an out-of-scope use.
+- **Robometer-4B** (`robometer/Robometer-4B`, official code): ᵇ progress difference between the two observations, with the two frames also
+  scored in reversed order to remove a position bias. ᶜ 1–5 levels from the last-frame progress, with cut points fitted on a
+  label-stratified subset of the RoboReward **validation** split (never the test set).
+- **Robo-Dopamine-GRM-2.0-8B** (`tanhuajie2001/Robo-Dopamine-GRM-2.0-8B-Preview`): the inference of PRM-as-a-Judge's `GRMInference`
+  (vLLM, same prompt, sampling and model length), its native 8-image layout (reference start / end + three views before and after).
+  ᵈ RoboPulse++: hop from the earlier to the later observation, reference start = the episode's first observation, blank goal image
+  (the model card's default when no goal image exists), missing views repeated as in the model card; 146 of 23,559 requests exceeded the
+  8,192-token limit and were rerun with images downscaled to ≤76,800 px. Its other readouts: forward 0.698, position-debiased 0.730.
+  ᵉ RoboRewardBench: hop from the first to the last frame, mapped to 1–5 with cut points fitted on the RoboReward validation subset.
+  Robo-Dopamine's RoboPulse lead is not explained by test overlap: on the 939 items whose episodes do not appear in its public
+  training data (GRM), it scores 87.5% (this model 68.8%).
+
+**Paired comparisons with the reward models** (this model minus the reward model, 95% cluster-bootstrap CI):
+
+| Reward model | RoboPulse acc | RoboPulse++ AUROC | RoboRewardBench MAE | RoboArena MAE | RoboRewardBench NLL |
+|---|---:|---:|---:|---:|---:|
+| RoboReward-8B | +21.2 [+17.9, +24.2] | +0.239 [+0.204, +0.272] | +0.085 [+0.039, +0.116] | −0.281 [−0.380, −0.190] | −0.367 [−0.517, −0.220] |
+| RoboReward-4B | +21.4 [+18.2, +24.6] | +0.221 [+0.185, +0.256] | +0.036 [+0.005, +0.093] | −0.312 [−0.410, −0.216] | −0.191 [−0.332, −0.060] |
+| Robometer-4B | +7.5 [+4.6, +10.3] | +0.098 [+0.062, +0.131] | −0.334 [−0.565, −0.316] | −0.147 [−0.245, −0.048] | – |
+| Robo-Dopamine-8B | −19.1 [−21.3, −16.8] | +0.011 [−0.026, +0.045] | −0.580 [−0.773, −0.521] | −0.241 [−0.345, −0.132] | – |
+
+Full reports: `eval/comparison/reward_models_{robopulse,robopulsepp,roboreward,jevany_robotics_public,calibration_ranking_speed}.md`.
+Positive RoboRewardBench MAE differences mean the reward model is better. Our harness measures the RoboReward models with *lower*
+error than published on the official leaderboard (8B: 0.358 vs 0.665; 4B: 0.407 vs 0.845) on the same 2,831 test videos (HF test split,
+revision 469b9af, all labels identical). Free generation reproduces our number and a zero-shot Qwen3-VL-4B-Instruct control lands near its
+published value (0.939 vs 1.032), so we report what we measured and do not compare with leaderboard entries.
+
+Ranking policies: on RoboArena sessions where two policies attempted the same task and received different human scores (805 policy
+pairs, 134 sessions), the model orders the two policies like the human raters 88.4% of the time (RoboReward-8B 85.3%, RoboReward-4B 83.7%,
+Robometer-4B 75.5%, OneJev-27B 88.8%; paired vs RoboReward-8B +3.1 [−0.2, +6.5], vs RoboReward-4B +4.7 [+1.4, +8.0]).
 
 ### Baselines
 
@@ -111,15 +165,37 @@ manifest hash) and is not directly comparable to the official 80.09%.
   under-predicts "rising" (accuracy 60.4% vs 69.9% for the parent; a constant "rising" answer scores 81.1%). Ranking quality (AUROC) and balanced
   accuracy are the meaningful metrics here and both improve; set the threshold on your own data. Multi-view (2–3 camera) items and long, slow-progress
   intervals remain the weakest cases.
+- **RoboRewardBench's synthetic OXE items.** On relabelled-instruction and clipped-video items the RoboReward models, trained on exactly
+  that construction, are clearly better (MAE 0.223 vs 0.297 and 0.418 vs 0.584 for RoboReward-8B vs this model); clipped videos are
+  often scored as near-complete.
+- **Speed vs RoboReward.** Our full-media protocol gives the model ~2.5× more visual tokens per video than RoboReward's processor budget,
+  and Qwen3.5's linear-attention layers are slower than Qwen3-VL attention: per RoboRewardBench video this model takes a median 279 ms
+  (fast kernels) vs 142–160 ms for a single RoboReward forward pass and 361 ms for RoboReward-8B's normal generation.
 - **Calibration outside robotics** (see above).
 
 ## Usage
 
 ```bash
+# --media is in a pending JevAny PR; until it is merged, use the full-media branch of the fork:
+git clone -b full-media https://github.com/BruceSheng1202/JevAny && cd JevAny
 pip install -e '.[serve,multimodal]'
-jevany serve --checkpoint SimpleJev/JevAny-Robotics-4B-LoRA --device cuda --dtype bf16
+jevany serve --checkpoint SimpleJev/JevAny-Robotics-4B-LoRA --device cuda --dtype bf16 \
+    --media full --max-state-tokens 16384 --max-packed-tokens 32768
 ```
 
-For video inputs use the same media handling as training/evaluation (native resolution, all frames of short derived clips).
+**Use `--media full`** (or `JEVANY_MEDIA=full`; also on `jevany decide --checkpoint`): images at native resolution and every frame of a
+video, the inputs this model was trained and evaluated with. With it, our evaluation harness returns exactly the probabilities
+of the reported runs (checked on 300 RoboRewardBench videos and 300 RoboPulse pairs). The JevAny default (`capped`: the images of a request share 512×512 pixels, a video is 8 frames
+within 512×512) costs accuracy, most of all on video:
+
+| | full | capped |
+|---|---:|---:|
+| RoboRewardBench MAE ↓ (RoboArena MAE) | 0.443 (0.687) | 0.607 (1.289) |
+| RoboPulse accuracy / AUROC | 71.1 / 0.806 | 69.4 / 0.786 |
+
+Send short clips like the evaluation's (about 1 frame per second, 8–32 frames); every frame becomes visual tokens. For speed, install the
+optional `flash-linear-attention` and `causal-conv1d` kernels (see JevAny's DEPLOYMENT.md): median 185 ms per RoboPulse pair on an A100.
+
 The repository contains LoRA adapter weights and JevAny readout metadata, not the base weights; the base model's license and
-access terms also apply, as do the licenses of the training data sources listed in the dataset card.
+access terms also apply, as do the licenses of the training data sources (RoboReward, Robo-Dopamine GRM, Guardian BDV2 / RLBench, DenseReward,
+RoboFAC, ViFailback, RoboVQA).

eval/comparison/reward_models_calibration_ranking_speed.md

@@ -0,0 +1,74 @@
+# Calibration, policy ranking and speed vs dedicated robot reward models (2026-10-23)
+
+This model is the v0.2 checkpoint (`JevAny-Robotics-4B-LoRA`). All models were run in the same harness on the same items. CIs are paired
+95% cluster-bootstrap intervals over task groups / sessions.
+
+## Calibration on RoboRewardBench (2,831 videos; probabilities over the five scores)
+
+| Model | NLL ↓ | ECE ↓ | Brier ↓ | RoboArena NLL ↓ | RoboArena ECE ↓ |
+|---|---:|---:|---:|---:|---:|
+| **JevAny-Robotics-4B-LoRA** | **1.021** | 0.103 | **0.498** | **1.143** | 0.065 |
+| RoboReward-8B | 1.388 | 0.227 | 0.563 | 2.082 | 0.347 |
+| RoboReward-4B | 1.212 | 0.175 | 0.532 | 1.649 | 0.240 |
+| JevAny-27B | 1.158 | **0.061** | 0.598 | 1.220 | **0.050** |
+| OneJev-27B | 1.252 | 0.090 | 0.649 | 1.287 | 0.131 |
+
+Other models that output probabilities, NLL: Intern-Decision-4B 1.262, OneJev-9B 1.302, parent JevAny-4B 1.337, OneJev-4B 1.349,
+OmniJev-4B 1.351, Jev-Omni-12B 1.477, Qwen3.5-4B zero-shot 1.483, Qwen3.8-27B zero-shot 1.540. Robometer-4B and Robo-Dopamine-8B
+do not output per-score probabilities. ECE: 10 equal-width confidence bins over the top score.
+
+Paired differences (this model − reward model):
+
+| | NLL | ECE | Brier |
+|---|---:|---:|---:|
+| vs RoboReward-8B, all | −0.367 [−0.517, −0.220] | −0.124 [−0.159, −0.087] | −0.064 [−0.114, −0.013] |
+| vs RoboReward-4B, all | −0.191 [−0.332, −0.060] | −0.072 [−0.106, −0.037] | −0.034 [−0.079, +0.011] |
+| vs RoboReward-8B, RoboArena | −0.939 [−1.108, −0.776] | −0.282 [−0.320, −0.233] | −0.275 [−0.328, −0.222] |
+| vs RoboReward-4B, RoboArena | −0.506 [−0.631, −0.383] | −0.174 [−0.209, −0.130] | −0.155 [−0.197, −0.112] |
+
+Exact-score accuracy is similar (this model 62.1%, RoboReward-8B 63.5%, RoboReward-4B 62.6%). On the 50% of items each model is most
+confident about, RoboReward is slightly more accurate (0.859 / 0.847 vs 0.820), driven by the synthetic OXE items.
+
+## Ranking policies (RoboRewardBench RoboArena split)
+
+RoboArena sessions run several policies on the same task. Pairs of policies in one session with different human scores: 805 pairs
+in 134 sessions. A pair counts as correct when the model's mean expected score (over the policy's camera views) orders the two
+policies like the human scores; ties count one half.
+
+| Model | Agreement with human ranking [95% CI] |
+|---|---:|
+| **JevAny-Robotics-4B-LoRA** | 88.4% [85.3, 91.4] |
+| OneJev-27B | 88.8% [85.7, 91.8] |
+| JevAny-27B | 88.3% [85.3, 91.2] |
+| RoboReward-8B | 85.3% [82.0, 88.4] |
+| RoboReward-4B | 83.7% [80.3, 87.2] |
+| Robo-Dopamine-8B (progress hop, first → last frame) | 78.3% |
+| Robometer-4B | 75.5% [71.6, 79.5] |
+| OmniJev-4B (training data includes RoboArena) | 81.2% |
+
+Paired (this model − other): RoboReward-8B +3.1 [−0.2, +6.5], RoboReward-4B +4.7 [+1.4, +8.0], Robometer-4B +12.9 [+8.4, +17.0],
+JevAny-27B +0.1 [−2.6, +2.6], OneJev-27B −0.4 [−3.8, +3.1].
+
+## Speed (one A100-40GB, batch 1, median end-to-end per item)
+
+| Model | RoboPulse pair | RoboRewardBench video |
+|---|---:|---:|
+| JevAny-Robotics-4B-LoRA, flash-linear-attention + causal-conv1d | 185 ms | 279 ms |
+| JevAny-Robotics-4B-LoRA, reference PyTorch kernels (used for all accuracy tables) | 294 ms | 392 ms |
+| Robo-Dopamine-8B (vLLM, first 200 RoboPulse items) | 316 ms | – |
+| RoboReward-4B / 8B, one forward pass (digit readout) | – | 142 / 160 ms |
+| RoboReward-8B, normal generation | – | 361 ms |
+| 27B Jev-style judges, reference kernels, 2 GPUs | 732–838 ms | – |
+
+The full-media protocol gives this model about 2.5× more visual tokens per RoboRewardBench video than RoboReward's processor budget
+(median 1,433 vs 564). Accuracy with the fast kernels is unchanged within numerical noise (RoboPulse 71.4%, RoboRewardBench MAE 0.446).
+
+## Media encoding
+
+| JevAny `--media` | RoboRewardBench MAE (RoboArena) | RoboPulse acc / AUROC | Median visual+text tokens (RRB / RoboPulse) |
+|---|---:|---:|---:|
+| `full` (evaluation protocol) | 0.443 (0.687) | 71.1 / 0.806 | 1,433 / 1,269 |
+| `capped` (JevAny default) | 0.607 (1.289) | 69.4 / 0.786 | 342 / 843 |
+
+With `--media full`, the evaluation harness returns exactly the reported probabilities (checked on 300 RoboRewardBench videos and 300
+RoboPulse pairs).

eval/comparison/reward_models_jevany_robotics_public.md

@@ -0,0 +1,22 @@
+# JevAny-Robotics-Public, reward-model-answerable families (2450 items); success: AUROC, preference: accuracy
+
+> JevAny-Robotics-Public v0.1, only the 6 of 13 families a reward model can answer. Reference = this model (v0.2). Success: RoboReward (expected score − 1)/4, Robometer success head on the last frame, Robo-Dopamine sigmoid(hop start→end); preference: score of A vs B. This model's training data includes the BDV2 and RoboVQA train splits.
+
+| family | n | JevAny-Robotics-4B-LoRA | OneJev-27B | RoboReward-8B | RoboReward-4B | Robometer-4B | Robo-Dopamine-8B | OmniJev-4B |
+|---|---|---|---|---|---|---|---|---|
+| bdv2fail_exec_success | 969 | 0.872 | 0.835 | 0.635 | 0.622 | 0.655 | 0.699 | 0.654 |
+| robofail_exec_success | 153 | 0.837 | 0.938 | 0.600 | 0.580 | 0.717 | 0.769 | 0.718 |
+| robovqa_exec_success | 357 | 0.948 | 0.946 | 0.753 | 0.737 | 0.826 | 0.739 | 0.838 |
+| ur5fail_exec_success | 140 | 0.857 | 0.907 | 0.654 | 0.625 | 0.688 | 0.665 | 0.749 |
+| robometer_traj_success | 399 | 0.846 | 0.888 | 0.800 | 0.792 | 0.893 | 0.745 | 0.812 |
+| robometer_traj_pref | 432 | 0.771 | 0.836 | 0.715 | 0.743 | 0.769 | 0.699 | 0.699 |
+| mean AUROC over the 5 success families | | 0.872 | 0.903 | 0.688 | 0.671 | 0.756 | 0.723 | 0.754 |
+
+Paired (JevAny-Robotics-4B-LoRA − model), 95% cluster-bootstrap CI over group_id:
+
+- vs OneJev-27B: mean success AUROC -0.031 [-0.054, -0.013]; preference acc -0.065 [-0.117, -0.024]
+- vs RoboReward-8B: mean success AUROC +0.184 [+0.147, +0.212]; preference acc +0.056 [+0.005, +0.120]
+- vs RoboReward-4B: mean success AUROC +0.201 [+0.168, +0.239]; preference acc +0.028 [-0.033, +0.094]
+- vs Robometer-4B: mean success AUROC +0.116 [+0.084, +0.151]; preference acc +0.002 [-0.047, +0.060]
+- vs Robo-Dopamine-8B: mean success AUROC +0.149 [+0.115, +0.185]; preference acc +0.072 [-0.007, +0.143]
+- vs OmniJev-4B: mean success AUROC +0.118 [+0.093, +0.147]; preference acc +0.072 [+0.035, +0.108]

eval/comparison/reward_models_robopulse.md

@@ -0,0 +1,14 @@
+# robopulse-v1: 1800 items; reference = JevAny-Robotics-4B-LoRA
+
+> Robot judges and dedicated reward models on RoboPulse, all run in our harness (2026-10-23). Reference = this model (v0.2). RoboReward scores whole rollouts; its pair score is the expected 1–5 score of the 2-frame video (before, after) minus (after, before), an out-of-scope use. Robometer: per-frame progress difference, position-debiased. Robo-Dopamine: PRM-as-a-Judge GRMInference (vLLM), native 8-image layout; `-debiased` subtracts the BEFORE/AFTER-swapped hop.
+
+| model | answered | acc | AUROC | balanced acc | pred pos rate | ref − model: acc [95% CI] | AUROC [95% CI] | balanced acc [95% CI] |
+|---|---|---|---|---|---|---|---|---|
+| JevAny-Robotics-4B-LoRA | 1800/1800 | 71.1 | 0.806 | 71.1 | 48.4 |  |  |  |
+| JevAny-27B | 1800/1800 | 56.6 | 0.627 | 56.6 | 77.8 | +14.4 [+11.5, +17.2] | +0.178 [+0.150, +0.206] | +14.4 [+11.8, +17.1] |
+| OneJev-27B | 1800/1800 | 60.4 | 0.651 | 60.4 | 46.7 | +10.7 [+7.8, +13.6] | +0.155 [+0.126, +0.183] | +10.7 [+7.9, +13.6] |
+| RoboReward-8B | 1800/1800 | 49.9 | 0.499 | 49.9 | 50.3 | +21.2 [+17.9, +24.2] | +0.307 [+0.271, +0.340] | +21.2 [+17.9, +24.2] |
+| RoboReward-4B | 1800/1800 | 49.6 | 0.501 | 49.6 | 48.8 | +21.4 [+18.2, +24.6] | +0.304 [+0.270, +0.337] | +21.4 [+18.2, +24.6] |
+| Robometer-4B | 1800/1800 | 63.6 | 0.700 | 63.6 | 49.4 | +7.5 [+4.6, +10.3] | +0.105 [+0.077, +0.132] | +7.5 [+4.6, +10.3] |
+| Robo-Dopamine-8B | 1800/1800 | 90.1 | 0.953 | 90.1 | 48.4 | -19.1 [-21.3, -16.8] | -0.147 [-0.168, -0.129] | -19.1 [-21.3, -16.8] |
+| Robo-Dopamine-8B-debiased | 1800/1800 | 90.0 | 0.963 | 90.0 | 47.4 | -18.9 [-21.4, -16.6] | -0.157 [-0.177, -0.138] | -18.9 [-21.4, -16.6] |

eval/comparison/reward_models_robopulsepp.md

@@ -0,0 +1,15 @@
+# robopulsepp-v1: 7720 items; reference = JevAny-Robotics-4B-LoRA
+
+> RoboPulse++ (7,720 consecutive-step pairs, 81% rising; compare AUROC). Reference = this model (v0.2). Robo-Dopamine readouts: incremental = hop(earlier→later), forward = hop(start→later) − hop(start→earlier), debiased = incremental minus the swapped hop; reference start = episode's first observation, blank goal image; 146 of 23,559 requests over 8,192 tokens rerun with images ≤76,800 px. RoboReward / Robometer as in the RoboPulse report.
+
+| model | answered | acc | AUROC | balanced acc | pred pos rate | ref − model: acc [95% CI] | AUROC [95% CI] | balanced acc [95% CI] |
+|---|---|---|---|---|---|---|---|---|
+| JevAny-Robotics-4B-LoRA | 7720/7720 | 60.4 | 0.728 | 65.9 | 51.0 |  |  |  |
+| JevAny-27B | 7720/7720 | 82.0 | 0.692 | 55.7 | 95.8 | -21.6 [-25.8, -17.4] | +0.036 [-0.004, +0.076] | +10.2 [+6.8, +13.3] |
+| OneJev-27B | 7720/7720 | 48.0 | 0.700 | 62.7 | 34.2 | +12.4 [+10.0, +14.8] | +0.028 [-0.006, +0.064] | +3.2 [+0.3, +5.9] |
+| RoboReward-8B | 7720/7720 | 49.4 | 0.489 | 50.0 | 49.0 | +11.0 [+8.3, +13.7] | +0.239 [+0.204, +0.272] | +15.9 [+13.0, +18.5] |
+| RoboReward-4B | 7720/7720 | 51.6 | 0.507 | 51.6 | 51.0 | +8.8 [+6.1, +11.4] | +0.221 [+0.185, +0.256] | +14.3 [+11.2, +17.3] |
+| Robometer-4B | 7720/7720 | 60.9 | 0.630 | 58.6 | 59.1 | -0.6 [-3.3, +2.0] | +0.098 [+0.062, +0.131] | +7.3 [+3.9, +10.2] |
+| Robo-Dopamine-8B-incremental | 7720/7720 | 61.8 | 0.717 | 68.6 | 50.6 | -1.4 [-4.5, +1.6] | +0.011 [-0.026, +0.045] | -2.8 [-6.0, +0.4] |
+| Robo-Dopamine-8B-forward | 7720/7720 | 59.8 | 0.698 | 65.2 | 50.8 | +0.5 [-2.0, +2.9] | +0.030 [-0.004, +0.062] | +0.6 [-2.4, +3.5] |
+| Robo-Dopamine-8B-debiased | 7720/7720 | 60.5 | 0.730 | 68.0 | 49.2 | -0.2 [-3.2, +2.7] | -0.002 [-0.036, +0.029] | -2.2 [-5.3, +1.0] |

eval/comparison/reward_models_roboreward.md

@@ -0,0 +1,131 @@
+# RoboRewardBench (roboreward-v1): 2831/2831 items scored by every run
+
+> RoboRewardBench (2,831 videos, HF test split revision 469b9af). Reference = this model (v0.2). RoboReward: model-card prompt, `ANSWER: ` prefill, digit-token softmax (free generation gives the same MAE). Robometer / Robo-Dopamine: last-frame progress / first→last-frame hop mapped to 1–5 with cut points fitted on the RoboReward validation split (adapted use). Our harness measures RoboReward with lower error than its leaderboard entry; we do not compare with leaderboard numbers.
+
+| model | Overall (group-wise MAE) | RoboArena | OXE subsets | micro MAE | median-label MAE | expected-score MAE | exact acc | within-1 | Spearman | QWK | mean pred |
+|---|---|---|---|---|---|---|---|---|---|---|---|
+| JevAny-Robotics-4B-LoRA | **0.443** | 0.687 | 0.432 | 0.523 | 0.419 | 0.458 | 62.1% | 89.0% | 0.827 | 0.788 | 2.82 |
+| JevAny-27B | **0.692** | 0.867 | 0.684 | 0.758 | 0.614 | 0.751 | 51.4% | 82.7% | 0.754 | 0.667 | 2.55 |
+| OneJev-27B | **0.805** | 0.919 | 0.800 | 0.856 | 0.705 | 0.764 | 47.3% | 77.5% | 0.726 | 0.657 | 2.53 |
+| RoboReward-8B | **0.358** | 0.968 | 0.330 | 0.552 | 0.359 | 0.364 | 63.5% | 86.6% | 0.797 | 0.754 | 2.91 |
+| RoboReward-4B | **0.407** | 0.999 | 0.380 | 0.594 | 0.398 | 0.417 | 62.6% | 85.0% | 0.777 | 0.720 | 2.90 |
+| Robometer-4B | **0.777** | 0.834 | 0.774 | 0.795 | 0.777 | 0.793 | 40.6% | 83.7% | 0.646 | 0.652 | 2.87 |
+| Robo-Dopamine-8B | **1.023** | 0.928 | 1.028 | 0.984 | 1.023 | 1.032 | 30.4% | 77.6% | 0.486 | 0.451 | 2.97 |
+
+
+## JevAny-Robotics-4B-LoRA: confusion (rows = true 1-5, cols = predicted 1-5)
+
+| true | pred 1 | pred 2 | pred 3 | pred 4 | pred 5 | n | mean pred |
+|---|---|---|---|---|---|---|---|
+| 1 | 573 | 84 | 23 | 5 | 4 | 689 | 1.23 |
+| 2 | 94 | 322 | 153 | 42 | 30 | 641 | 2.36 |
+| 3 | 40 | 76 | 304 | 89 | 47 | 556 | 3.05 |
+| 4 | 29 | 27 | 110 | 162 | 90 | 418 | 3.61 |
+| 5 | 7 | 12 | 44 | 67 | 397 | 527 | 4.58 |
+
+## JevAny-27B: confusion (rows = true 1-5, cols = predicted 1-5)
+
+| true | pred 1 | pred 2 | pred 3 | pred 4 | pred 5 | n | mean pred |
+|---|---|---|---|---|---|---|---|
+| 1 | 597 | 56 | 0 | 2 | 34 | 689 | 1.29 |
+| 2 | 292 | 241 | 51 | 8 | 49 | 641 | 1.88 |
+| 3 | 110 | 179 | 173 | 31 | 63 | 556 | 2.56 |
+| 4 | 50 | 68 | 100 | 58 | 142 | 418 | 3.42 |
+| 5 | 39 | 34 | 33 | 34 | 387 | 527 | 4.32 |
+
+## OneJev-27B: confusion (rows = true 1-5, cols = predicted 1-5)
+
+| true | pred 1 | pred 2 | pred 3 | pred 4 | pred 5 | n | mean pred |
+|---|---|---|---|---|---|---|---|
+| 1 | 631 | 4 | 25 | 1 | 28 | 689 | 1.25 |
+| 2 | 440 | 73 | 72 | 5 | 51 | 641 | 1.68 |
+| 3 | 221 | 55 | 157 | 20 | 103 | 556 | 2.51 |
+| 4 | 89 | 42 | 61 | 38 | 188 | 418 | 3.46 |
+| 5 | 40 | 17 | 15 | 15 | 440 | 527 | 4.51 |
+
+## RoboReward-8B: confusion (rows = true 1-5, cols = predicted 1-5)
+
+| true | pred 1 | pred 2 | pred 3 | pred 4 | pred 5 | n | mean pred |
+|---|---|---|---|---|---|---|---|
+| 1 | 568 | 62 | 32 | 5 | 22 | 689 | 1.33 |
+| 2 | 76 | 343 | 136 | 44 | 42 | 641 | 2.43 |
+| 3 | 27 | 85 | 288 | 57 | 99 | 556 | 3.21 |
+| 4 | 11 | 35 | 86 | 189 | 97 | 418 | 3.78 |
+| 5 | 16 | 16 | 31 | 54 | 410 | 527 | 4.57 |
+
+## RoboReward-4B: confusion (rows = true 1-5, cols = predicted 1-5)
+
+| true | pred 1 | pred 2 | pred 3 | pred 4 | pred 5 | n | mean pred |
+|---|---|---|---|---|---|---|---|
+| 1 | 561 | 74 | 16 | 7 | 31 | 689 | 1.36 |
+| 2 | 79 | 336 | 101 | 63 | 62 | 641 | 2.52 |
+| 3 | 35 | 92 | 279 | 76 | 74 | 556 | 3.11 |
+| 4 | 19 | 46 | 64 | 209 | 80 | 418 | 3.68 |
+| 5 | 14 | 19 | 40 | 66 | 388 | 527 | 4.51 |
+
+## Robometer-4B: confusion (rows = true 1-5, cols = predicted 1-5)
+
+| true | pred 1 | pred 2 | pred 3 | pred 4 | pred 5 | n | mean pred |
+|---|---|---|---|---|---|---|---|
+| 1 | 243 | 305 | 100 | 39 | 2 | 689 | 1.91 |
+| 2 | 107 | 301 | 165 | 59 | 9 | 641 | 2.32 |
+| 3 | 30 | 189 | 181 | 116 | 40 | 556 | 2.90 |
+| 4 | 12 | 60 | 105 | 159 | 82 | 418 | 3.57 |
+| 5 | 2 | 39 | 70 | 150 | 266 | 527 | 4.21 |
+
+## Robo-Dopamine-8B: confusion (rows = true 1-5, cols = predicted 1-5)
+
+| true | pred 1 | pred 2 | pred 3 | pred 4 | pred 5 | n | mean pred |
+|---|---|---|---|---|---|---|---|
+| 1 | 173 | 293 | 111 | 112 | 0 | 689 | 2.24 |
+| 2 | 42 | 237 | 203 | 159 | 0 | 641 | 2.75 |
+| 3 | 17 | 95 | 194 | 250 | 0 | 556 | 3.22 |
+| 4 | 4 | 58 | 100 | 256 | 0 | 418 | 3.45 |
+| 5 | 2 | 62 | 109 | 354 | 0 | 527 | 3.55 |
+
+## Micro MAE / exact acc by item type
+
+| model | robo_arena (n=1000) | oxe_success (n=371) | oxe_relabelled (n=669) | oxe_clipped (n=791) |
+|---|---|---|---|---|
+| JevAny-Robotics-4B-LoRA | 0.687 / 51.7% | 0.358 / 76.5% | 0.297 / 77.3% | 0.584 / 55.6% |
+| JevAny-27B | 0.867 / 46.0% | 0.698 / 72.8% | 0.713 / 51.3% | 0.686 / 48.4% |
+| OneJev-27B | 0.919 / 44.0% | 0.528 / 81.9% | 0.842 / 46.8% | 0.942 / 35.7% |
+| RoboReward-8B | 0.968 / 42.2% | 0.310 / 81.4% | 0.223 / 81.9% | 0.418 / 66.5% |
+| RoboReward-4B | 0.999 / 43.7% | 0.447 / 74.1% | 0.214 / 82.7% | 0.472 / 64.2% |
+| Robometer-4B | 0.834 / 37.9% | 0.590 / 60.4% | 1.049 / 24.2% | 0.626 / 48.7% |
+| Robo-Dopamine-8B | 0.928 / 33.3% | 1.472 / 0.0% | 1.148 / 26.9% | 0.689 / 43.9% |
+
+## Per-subset MAE (argmax)
+
+| subset | n | JevAny-Robotics-4B-LoRA | JevAny-27B | OneJev-27B | RoboReward-8B | RoboReward-4B | Robometer-4B | Robo-Dopamine-8B |
+|---|---|---|---|---|---|---|---|---|
+| robo_arena | 1000 | 0.687 | 0.867 | 0.919 | 0.968 | 0.999 | 0.834 | 0.928 |
+| austin_sirius_dataset_converted_externally_to_rlds | 87 | 0.414 | 0.437 | 0.770 | 0.310 | 0.609 | 0.862 | 1.184 |
+| berkeley_autolab_ur5 | 100 | 0.360 | 0.930 | 1.160 | 0.190 | 0.390 | 0.530 | 1.280 |
+| berkeley_fanuc_manipulation | 91 | 0.220 | 0.967 | 1.143 | 0.176 | 0.176 | 0.725 | 0.835 |
+| berkeley_mvp_converted_externally_to_rlds | 71 | 0.099 | 0.183 | 0.380 | 0.056 | 0.099 | 0.549 | 1.056 |
+| berkeley_rpt_converted_externally_to_rlds | 86 | 0.826 | 0.500 | 0.733 | 0.453 | 0.407 | 0.721 | 1.407 |
+| bridge | 100 | 0.240 | 0.510 | 0.820 | 0.170 | 0.160 | 0.730 | 0.860 |
+| cmu_play_fusion | 92 | 0.413 | 1.163 | 1.087 | 0.250 | 0.239 | 0.957 | 0.967 |
+| dlr_edan_shared_control_converted_externally_to_rlds | 19 | 0.000 | 0.474 | 0.474 | 0.105 | 0.105 | 0.632 | 1.105 |
+| droid | 100 | 0.710 | 0.880 | 0.810 | 0.460 | 0.700 | 0.940 | 1.010 |
+| fractal20220817_data | 100 | 0.240 | 0.600 | 0.830 | 0.170 | 0.240 | 0.830 | 1.030 |
+| iamlab_cmu_pickup_insert_converted_externally_to_rlds | 83 | 0.133 | 0.313 | 0.675 | 0.422 | 0.313 | 0.542 | 1.084 |
+| jaco_play | 100 | 0.270 | 0.660 | 0.870 | 0.190 | 0.180 | 0.760 | 0.950 |
+| kaist_nonprehensile_converted_externally_to_rlds | 53 | 0.415 | 0.736 | 0.792 | 0.302 | 0.509 | 1.113 | 1.038 |
+| roboturk | 97 | 0.505 | 0.546 | 0.577 | 0.443 | 0.598 | 0.722 | 1.000 |
+| stanford_hydra_dataset_converted_externally_to_rlds | 91 | 0.923 | 0.736 | 0.560 | 0.615 | 0.440 | 0.637 | 0.846 |
+| taco_play | 91 | 0.198 | 1.429 | 1.484 | 0.099 | 0.099 | 1.055 | 1.330 |
+| tokyo_u_lsmo_converted_externally_to_rlds | 71 | 0.549 | 0.563 | 0.704 | 0.296 | 0.169 | 0.324 | 1.324 |
+| ucsd_kitchen_dataset_converted_externally_to_rlds | 95 | 0.337 | 0.495 | 0.832 | 0.484 | 0.463 | 0.726 | 0.674 |
+| ucsd_pick_and_place_dataset_converted_externally_to_rlds | 100 | 0.150 | 0.500 | 0.560 | 0.060 | 0.090 | 0.690 | 0.550 |
+| utokyo_pr2_tabletop_manipulation_converted_externally_to_rlds | 73 | 0.288 | 0.562 | 0.397 | 0.164 | 0.164 | 1.247 | 1.014 |
+| utokyo_xarm_bimanual_converted_externally_to_rlds | 71 | 1.451 | 1.606 | 1.324 | 1.000 | 0.972 | 0.958 | 0.930 |
+| viola | 60 | 0.767 | 0.267 | 0.617 | 0.850 | 1.233 | 0.783 | 1.133 |
+
+Paired: Overall MAE JevAny-Robotics-4B-LoRA - JevAny-27B = -0.249, 95% cluster-bootstrap CI [-0.374, -0.220]; per-item |err| smaller for JevAny-Robotics-4B-LoRA on 842, larger on 447, tie 1542
+Paired: Overall MAE JevAny-Robotics-4B-LoRA - OneJev-27B = -0.362, 95% cluster-bootstrap CI [-0.488, -0.336]; per-item |err| smaller for JevAny-Robotics-4B-LoRA on 950, larger on 393, tie 1488
+Paired: Overall MAE JevAny-Robotics-4B-LoRA - RoboReward-8B = +0.085, 95% cluster-bootstrap CI [+0.039, +0.116]; per-item |err| smaller for JevAny-Robotics-4B-LoRA on 480, larger on 504, tie 1847
+Paired: Overall MAE JevAny-Robotics-4B-LoRA - RoboReward-4B = +0.036, 95% cluster-bootstrap CI [+0.005, +0.093]; per-item |err| smaller for JevAny-Robotics-4B-LoRA on 511, larger on 481, tie 1839
+Paired: Overall MAE JevAny-Robotics-4B-LoRA - Robometer-4B = -0.334, 95% cluster-bootstrap CI [-0.565, -0.316]; per-item |err| smaller for JevAny-Robotics-4B-LoRA on 1160, larger on 576, tie 1095
+Paired: Overall MAE JevAny-Robotics-4B-LoRA - Robo-Dopamine-8B = -0.580, 95% cluster-bootstrap CI [-0.773, -0.521]; per-item |err| smaller for JevAny-Robotics-4B-LoRA on 1434, larger on 536, tie 861