| --- |
| base_model: Qwen/Qwen3.5-4B |
| library_name: peft |
| license: apache-2.0 |
| tags: |
| - jevany |
| - lora |
| - decision-model |
| - pointer |
| - robotics |
| - reward-model |
| - judge |
| datasets: |
| - SimpleJev/JevAny-Robotics-Train-v0.2 |
| --- |
| |
| # JevAny-Robotics-4B-LoRA (v0.2, SFT) |
|
|
| A 4B robot judge / reward model: a robotics continuation of [`SimpleJev/JevAny-Qwen3.5-4B-LoRA`](https://huggingface.co/SimpleJev/JevAny-Qwen3.5-4B-LoRA) |
| (pointer readout on [`Qwen/Qwen3.5-4B`](https://huggingface.co/Qwen/Qwen3.5-4B)), fine-tuned on [`SimpleJev/JevAny-Robotics-Train-v0.2`](https://huggingface.co/datasets/SimpleJev/JevAny-Robotics-Train-v0.2) — 184,942 judgement questions over |
| real and simulated robot images and videos (task progress, 1–5 rollout scores, execution success, failure mode, plan checking, |
| failure localisation, trajectory preference), built only from train-side splits and decontaminated against every test set below. |
|
|
| **Highlights** (all models run by us in the same harness, on the same items with the same media protocol): |
|
|
| - **RoboPulse: 71.1% accuracy / 0.806 AUROC** — best of every judge we ran, including 27B models (OneJev-27B 60.4% / 0.651, JevAny-27B 56.6% / 0.627); +18.3 points over its parent. |
| - **RoboRewardBench: MAE 0.443** — lower than every model we ran (OneJev-27B 0.805, JevAny-27B 0.692, Intern-Decision-4B 0.820) and than the best entry of the paper leaderboard (RoboReward-8B 0.665, their protocol); paired MAE difference vs the best 27B model −0.249 [−0.374, −0.220]. |
| - **RoboPulse++: best AUROC (0.728)** and balanced accuracy among 4B models (65.9 vs Intern-Decision-4B 60.5); at the level of the best 27B model (OneJev-27B 0.700 AUROC, paired +0.028 [−0.004, +0.063]). |
| - **JevAny-Robotics-Public: skill_role 0.567, micro accuracy 80.4% (highest of all models)** — 4× its parent, +0.14 over OneJev-4B, statistically tied with the strongest 27B judge (OneJev-27B 0.605, paired −0.038 [−0.081, +0.006]). |
| - Same 4B backbone and speed as the parent (median 161 ms per JevAny-Robotics-Public item on an A100). |
| |
| ## Robot benchmarks |
| |
| | Model | Size | RoboPulse acc ↑ | RoboPulse AUROC ↑ | RoboPulse++ AUROC ↑ | RoboPulse++ bal. acc ↑ | RoboRewardBench MAE ↓ | JevAny-Robotics-Public skill_role ↑ | |
| |---|---:|---:|---:|---:|---:|---:|---:| |
| | **JevAny-Robotics-4B-LoRA (this model)** | 4B | **71.1** | **0.806** | **0.728** | **65.9** | **0.443** | 0.567 | |
| | JevAny-Qwen3.5-4B-LoRA (parent) | 4B | 52.7 | 0.588 | 0.641 | 59.1 | 1.057 | 0.149 | |
| | Qwen3.5-4B zero-shot (constrained choice) | 4B | 49.9 | 0.519 | 0.592 | 50.0 | 1.261 | 0.203 | |
| | OneJev-4B | 4B | 53.9 | 0.586 | 0.617 | 58.3 | 1.005 | 0.425 | |
| | Intern-Decision-4B | 4B | 51.6 | 0.547 | 0.653 | 60.5 | 0.820 | 0.303 | |
| | OmniJev-4B | 4B | 49.7 | 0.536 | 0.556 | 54.6 | 0.811* | 0.264 | |
| | OneJev-9B | 9B | 57.6 | 0.595 | 0.564 | 54.0 | 0.927 | 0.442 | |
| | Jev-Omni-12B | 12B | 49.4 | 0.492 | 0.537 | 52.3 | 1.085 | 0.194 | |
| | Qwen3.8-27B zero-shot (constrained choice) | 27B | 56.7 | 0.604 | 0.569 | 55.2 | 0.992 | 0.472 | |
| | JevAny-Qwen3.8-27B-LoRA | 27B | 56.6 | 0.627 | 0.692 | 55.7 | 0.692 | 0.549 | |
| | OneJev-27B | 27B | 60.4 | 0.651 | 0.700 | 62.7 | 0.805 | **0.605** | |
|
|
| RoboPulse: 1,800 pairs, native image resolution. RoboPulse++: 7,720 consecutive-step pairs, native resolution. RoboRewardBench: 2,831 videos, full media, |
| official group-wise MAE (RoboArena 0.687 / OXE subsets 0.432 for this model). JevAny-Robotics-Public v0.1: 3,990 items, skill_role is the role-balanced |
| chance-corrected skill. *OmniJev-4B's training data overlaps RoboRewardBench. Every model answered every item. |
| |
| **Paired comparisons** (this model minus baseline, 95% cluster-bootstrap CI over episodes / task groups): |
| |
| | Baseline | RoboPulse acc | RoboPulse++ AUROC | RoboRewardBench MAE | JevAny-Robotics-Public skill_role | |
| |---|---:|---:|---:|---:| |
| | Parent JevAny-4B | +18.3 [+15.1, +21.4] | +0.087 [+0.049, +0.131] | −0.614 [−0.725, −0.554] | +0.419 [+0.365, +0.463] | |
| | Qwen3.5-4B zero-shot | +21.2 [+18.1, +24.4] | +0.136 [+0.091, +0.187] | −0.817 [−0.875, −0.732] | +0.364 [+0.318, +0.411] | |
| | OneJev-4B | +17.2 [+14.1, +20.1] | +0.111 [+0.072, +0.151] | −0.562 [−0.811, −0.541] | +0.142 [+0.100, +0.189] | |
| | Intern-Decision-4B | +19.4 [+16.5, +22.5] | +0.075 [+0.047, +0.104] | −0.376 [−0.518, −0.341] | +0.264 [+0.218, +0.312] | |
| | JevAny-27B | +14.4 [+11.4, +17.0] | +0.036 [−0.003, +0.076] | −0.249 [−0.374, −0.220] | +0.019 [−0.027, +0.052] | |
| | OneJev-27B | +10.7 [+8.0, +13.3] | +0.028 [−0.004, +0.063] | −0.362 [−0.488, −0.336] | −0.038 [−0.081, +0.006] | |
|
|
| Every difference in favour of this model is significant except RoboPulse++ AUROC and JevAny-Robotics-Public against the two 27B judges (ties). |
| Per-benchmark reports with all slices are in `eval/comparison/`. |
|
|
| For reference, published numbers under the benchmarks' own protocols: RoboRewardBench leaderboard — RoboReward-8B 0.665, GPT-5 mini 0.691, |
| GPT-5 0.811, RoboReward-4B 0.845, Gemini 3 Pro (preview) 0.851, Qwen3-VL-8B 0.892; RoboPulse paper — Gemini 3 Pro Preview 0.66, Qwen3-VL-8B 0.59, |
| Qwen3-VL-4B 0.56. |
|
|
| ### Baselines |
|
|
| - **Zero-shot Qwen3.5-4B / Qwen3.8-27B (constrained choice)**: the untrained base VLM with the same judge prompt; options are lettered and the answer is |
| restricted to the option letters (one forward pass, probabilities = softmax over the letter logits). This is the same scoring as vLLM |
| structured-output choice decoding (`structured_outputs.choice` + letter logprobs); the robot suites were run with the transformers implementation, |
| JevAny-Multimodal-agentic-Public with vLLM. |
| - **OneJev / Intern-Decision / OmniJev / Jev-Omni**: released checkpoints run with their own inference code inside the same harness and items. |
| - **JevAny-Qwen3.8-27B-LoRA** and the parent **JevAny-Qwen3.5-4B-LoRA**: released JevAny checkpoints, same media settings as this model. |
|
|
| ## General judgement (retention) |
|
|
| | Benchmark | Qwen3.5-4B zero-shot (vLLM choice) | Parent JevAny-4B | This model | This − parent | |
| |---|---:|---:|---:|---| |
| | Transfer v9 (1,046) acc / NLL | — | 78.68% / 0.587 | 78.39% / 0.664 | acc −0.29 [−1.75, +1.05] | |
| | JevBench public (local rebuild, 231) acc | — | 77.92% | 77.92% | 0.00 [−3.10, +2.64] | |
| | JevAny-Multimodal-agentic-Public v0.4 (3,208) skill_role / acc | 0.223 / 53.6% | 0.201 / 52.6% | 0.176 / 51.4% | acc −1.2 (n.s., McNemar p = 0.12) | |
| |
| Accuracy on general (non-robot) judgement is retained within noise; probabilities become over-confident outside robotics |
| (NLL +0.077 on Transfer, +0.163 on JevAny-Multimodal-agentic-Public), so recalibrate before using them there. For general-purpose judging use the parent |
| checkpoint; this one is specialised for robotics. The JevBench number is on a local rebuild of the public suite (same 231 items, different |
| manifest hash) and is not directly comparable to the official 80.09%. |
| |
| ## Training |
| |
| - Init: LoRA adapter + pointer head of `SimpleJev/JevAny-Qwen3.5-4B-LoRA` (delta mode); base revision `851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a`. |
| - 1 epoch, 5,780 optimizer steps, global batch 32, lr 1e-5 (OneCycle), LoRA r=8 on all projection modules, head dim 256 + residual 256, bf16. |
| - Full media: images at native resolution, every frame of the derived video clips (≈1–2 fps, 8–32 frames), up to 16,384 packed tokens. |
| - `training_config.json` / `training_metrics.json` hold the exact arguments; `eval/training_eval_history.jsonl` the in-training evaluations. |
| |
| | step | development acc / NLL (3,026) | calibration acc / NLL (2,228) | |
| |---:|---:|---:| |
| | 0 (parent) | 0.463 / 1.069 | 0.496 / 0.969 | |
| | 1,000 | 0.640 / 0.757 | 0.688 / 0.633 | |
| | 3,000 | 0.723 / 0.650 | 0.778 / 0.499 | |
| | 5,780 (this checkpoint) | 0.733 / 0.638 | 0.786 / 0.475 | |
| |
| ## Limits |
| |
| - **Source overlap.** RoboPulse and RoboRewardBench draw on the same public robot datasets as parts of the training data (GRM, RoboReward train). |
| Training data were decontaminated against all test sets (episode blocklists + perceptual-hash frame dedup), but scenes and robots overlap. |
| - **RoboPulse++ threshold.** Training balanced rising/falling 1:1, while the RoboPulse++ test is 81% rising; at the default 0.5 threshold the model |
| under-predicts "rising" (accuracy 60.4% vs 69.9% for the parent; a constant "rising" answer scores 81.1%). Ranking quality (AUROC) and balanced |
| accuracy are the meaningful metrics here and both improve; set the threshold on your own data. Multi-view (2–3 camera) items and long, slow-progress |
| intervals remain the weakest cases. |
| - **Calibration outside robotics** (see above). |
| |
| ## Usage |
| |
| ```bash |
| pip install -e '.[serve,multimodal]' |
| jevany serve --checkpoint SimpleJev/JevAny-Robotics-4B-LoRA --device cuda --dtype bf16 |
| ``` |
| |
| For video inputs use the same media handling as training/evaluation (native resolution, all frames of short derived clips). |
| The repository contains LoRA adapter weights and JevAny readout metadata, not the base weights; the base model's license and |
| access terms also apply, as do the licenses of the training data sources listed in the dataset card. |
| |
