Card: use the JevAny-* repository names (renamed 2026-10-17); data files unchanged

README.md

@@ -11,13 +11,13 @@ tags:
 - reward-model
 - judge
 datasets:
-- SimpleJev/RoboJudge-Train-v0.2
+- SimpleJev/JevAny-Robotics-Train-v0.2
 ---
 
 # JevAny-Robotics-4B-LoRA (v0.2, SFT)
 
-A 4B robot judge / reward model: a robotics continuation of [`SimpleJev/JevAny-Robotics-4B-LoRA`](https://huggingface.co/SimpleJev/JevAny-Robotics-4B-LoRA)
-(pointer readout on [`Qwen/Qwen3.5-4B`](https://huggingface.co/Qwen/Qwen3.5-4B)), fine-tuned on SimpleJev/JevAny-Robotics-Train-v0.2 — 184,942 judgement questions over
+A 4B robot judge / reward model: a robotics continuation of [`SimpleJev/JevAny-Qwen3.5-4B-LoRA`](https://huggingface.co/SimpleJev/JevAny-Qwen3.5-4B-LoRA)
+(pointer readout on [`Qwen/Qwen3.5-4B`](https://huggingface.co/Qwen/Qwen3.5-4B)), fine-tuned on [`SimpleJev/JevAny-Robotics-Train-v0.2`](https://huggingface.co/datasets/SimpleJev/JevAny-Robotics-Train-v0.2) — 184,942 judgement questions over
 real and simulated robot images and videos (task progress, 1–5 rollout scores, execution success, failure mode, plan checking,
 failure localisation, trajectory preference), built only from train-side splits and decontaminated against every test set below.
 
@@ -26,12 +26,12 @@ failure localisation, trajectory preference), built only from train-side splits
 - **RoboPulse: 71.1% accuracy / 0.806 AUROC** — best of every judge we ran, including 27B models (OneJev-27B 60.4% / 0.651, JevAny-27B 56.6% / 0.627); +18.3 points over its parent.
 - **RoboRewardBench: MAE 0.443** — lower than every model we ran (OneJev-27B 0.805, JevAny-27B 0.692, Intern-Decision-4B 0.820) and than the best entry of the paper leaderboard (RoboReward-8B 0.665, their protocol); paired MAE difference vs the best 27B model −0.249 [−0.374, −0.220].
 - **RoboPulse++: best AUROC (0.728)** and balanced accuracy among 4B models (65.9 vs Intern-Decision-4B 60.5); at the level of the best 27B model (OneJev-27B 0.700 AUROC, paired +0.028 [−0.004, +0.063]).
-- **JevJudge-Robotics-Public: skill_role 0.567, micro accuracy 80.4% (highest of all models)** — 4× its parent, +0.14 over OneJev-4B, statistically tied with the strongest 27B judge (OneJev-27B 0.605, paired −0.038 [−0.081, +0.006]).
-- Same 4B backbone and speed as the parent (median 161 ms per JevJudge-Robotics item on an A100).
+- **JevAny-Robotics-Public: skill_role 0.567, micro accuracy 80.4% (highest of all models)** — 4× its parent, +0.14 over OneJev-4B, statistically tied with the strongest 27B judge (OneJev-27B 0.605, paired −0.038 [−0.081, +0.006]).
+- Same 4B backbone and speed as the parent (median 161 ms per JevAny-Robotics-Public item on an A100).
 
 ## Robot benchmarks
 
-| Model | Size | RoboPulse acc ↑ | RoboPulse AUROC ↑ | RoboPulse++ AUROC ↑ | RoboPulse++ bal. acc ↑ | RoboRewardBench MAE ↓ | JevJudge-Robotics skill_role ↑ |
+| Model | Size | RoboPulse acc ↑ | RoboPulse AUROC ↑ | RoboPulse++ AUROC ↑ | RoboPulse++ bal. acc ↑ | RoboRewardBench MAE ↓ | JevAny-Robotics-Public skill_role ↑ |
 |---|---:|---:|---:|---:|---:|---:|---:|
 | **JevAny-Robotics-4B-LoRA (this model)** | 4B | **71.1** | **0.806** | **0.728** | **65.9** | **0.443** | 0.567 |
 | JevAny-Qwen3.5-4B-LoRA (parent) | 4B | 52.7 | 0.588 | 0.641 | 59.1 | 1.057 | 0.149 |
@@ -46,12 +46,12 @@ failure localisation, trajectory preference), built only from train-side splits
 | OneJev-27B | 27B | 60.4 | 0.651 | 0.700 | 62.7 | 0.805 | **0.605** |
 
 RoboPulse: 1,800 pairs, native image resolution. RoboPulse++: 7,720 consecutive-step pairs, native resolution. RoboRewardBench: 2,831 videos, full media,
-official group-wise MAE (RoboArena 0.687 / OXE subsets 0.432 for this model). JevJudge-Robotics-Public v0.1: 3,990 items, skill_role is the role-balanced
+official group-wise MAE (RoboArena 0.687 / OXE subsets 0.432 for this model). JevAny-Robotics-Public v0.1: 3,990 items, skill_role is the role-balanced
 chance-corrected skill. *OmniJev-4B's training data overlaps RoboRewardBench. Every model answered every item.
 
 **Paired comparisons** (this model minus baseline, 95% cluster-bootstrap CI over episodes / task groups):
 
-| Baseline | RoboPulse acc | RoboPulse++ AUROC | RoboRewardBench MAE | JevJudge-Robotics skill_role |
+| Baseline | RoboPulse acc | RoboPulse++ AUROC | RoboRewardBench MAE | JevAny-Robotics-Public skill_role |
 |---|---:|---:|---:|---:|
 | Parent JevAny-4B | +18.3 [+15.1, +21.4] | +0.087 [+0.049, +0.131] | −0.614 [−0.725, −0.554] | +0.419 [+0.365, +0.463] |
 | Qwen3.5-4B zero-shot | +21.2 [+18.1, +24.4] | +0.136 [+0.091, +0.187] | −0.817 [−0.875, −0.732] | +0.364 [+0.318, +0.411] |
@@ -60,7 +60,7 @@ chance-corrected skill. *OmniJev-4B's training data overlaps RoboRewardBench. Ev
 | JevAny-27B | +14.4 [+11.4, +17.0] | +0.036 [−0.003, +0.076] | −0.249 [−0.374, −0.220] | +0.019 [−0.027, +0.052] |
 | OneJev-27B | +10.7 [+8.0, +13.3] | +0.028 [−0.004, +0.063] | −0.362 [−0.488, −0.336] | −0.038 [−0.081, +0.006] |
 
-Every difference in favour of this model is significant except RoboPulse++ AUROC and JevJudge-Robotics against the two 27B judges (ties).
+Every difference in favour of this model is significant except RoboPulse++ AUROC and JevAny-Robotics-Public against the two 27B judges (ties).
 Per-benchmark reports with all slices are in `eval/comparison/`.
 
 For reference, published numbers under the benchmarks' own protocols: RoboRewardBench leaderboard — RoboReward-8B 0.665, GPT-5 mini 0.691,
@@ -72,7 +72,7 @@ Qwen3-VL-4B 0.56.
 - **Zero-shot Qwen3.5-4B / Qwen3.8-27B (constrained choice)**: the untrained base VLM with the same judge prompt; options are lettered and the answer is
   restricted to the option letters (one forward pass, probabilities = softmax over the letter logits). This is the same scoring as vLLM
   structured-output choice decoding (`structured_outputs.choice` + letter logprobs); the robot suites were run with the transformers implementation,
-  JevJudge-Public with vLLM.
+  JevAny-Multimodal-agentic-Public with vLLM.
 - **OneJev / Intern-Decision / OmniJev / Jev-Omni**: released checkpoints run with their own inference code inside the same harness and items.
 - **JevAny-Qwen3.8-27B-LoRA** and the parent **JevAny-Qwen3.5-4B-LoRA**: released JevAny checkpoints, same media settings as this model.
 
@@ -82,10 +82,10 @@ Qwen3-VL-4B 0.56.
 |---|---:|---:|---:|---|
 | Transfer v9 (1,046) acc / NLL | — | 78.68% / 0.587 | 78.39% / 0.664 | acc −0.29 [−1.75, +1.05] |
 | JevBench public (local rebuild, 231) acc | — | 77.92% | 77.92% | 0.00 [−3.10, +2.64] |
-| JevJudge-Public v0.4 (3,208) skill_role / acc | 0.223 / 53.6% | 0.201 / 52.6% | 0.176 / 51.4% | acc −1.2 (n.s., McNemar p = 0.12) |
+| JevAny-Multimodal-agentic-Public v0.4 (3,208) skill_role / acc | 0.223 / 53.6% | 0.201 / 52.6% | 0.176 / 51.4% | acc −1.2 (n.s., McNemar p = 0.12) |
 
 Accuracy on general (non-robot) judgement is retained within noise; probabilities become over-confident outside robotics
-(NLL +0.077 on Transfer, +0.163 on JevJudge-Public), so recalibrate before using them there. For general-purpose judging use the parent
+(NLL +0.077 on Transfer, +0.163 on JevAny-Multimodal-agentic-Public), so recalibrate before using them there. For general-purpose judging use the parent
 checkpoint; this one is specialised for robotics. The JevBench number is on a local rebuild of the public suite (same 231 items, different
 manifest hash) and is not directly comparable to the official 80.09%.
 
@@ -117,7 +117,7 @@ manifest hash) and is not directly comparable to the official 80.09%.
 
 ```bash
 pip install -e '.[serve,multimodal]'
-jevany serve --checkpoint SimpleJev/JevAny-Qwen3.5-4B-RoboJudge-LoRA --device cuda --dtype bf16
+jevany serve --checkpoint SimpleJev/JevAny-Robotics-4B-LoRA --device cuda --dtype bf16
 ```
 
 For video inputs use the same media handling as training/evaluation (native resolution, all frames of short derived clips).