README.md

---
base_model: Qwen/Qwen3.5-4B
library_name: peft
license: apache-2.0
tags:
- jevany
- lora
- decision-model
- pointer
- robotics
- reward-model
- judge
---

# JevAny-Robotics-4B-LoRA (v0.2, SFT)

A 4B Jev-style (System One) judge for robot manipulation: one forward pass returns a probability for every option. It is a robotics continuation of [`SimpleJev/JevAny-Qwen3.5-4B-LoRA`](https://huggingface.co/SimpleJev/JevAny-Qwen3.5-4B-LoRA)
(pointer readout on [`Qwen/Qwen3.5-4B`](https://huggingface.co/Qwen/Qwen3.5-4B)), fine-tuned on JevAny-Robotics-Train-v0.2 (to be released) — 184,942 judgement questions over
real and simulated robot images and videos (task progress, 1–5 rollout scores, execution success, failure mode, plan checking,
failure localisation, trajectory preference), built only from train-side splits and decontaminated against every test set below.

**Highlights** (every model below was run by us in the same harness, on the same items; dedicated robot reward models
included, see [Dedicated robot reward models](#dedicated-robot-reward-models-same-harness)):

- **No single model wins every benchmark; this one stays near the top on all of them.** Specialists win their own format:
  Robo-Dopamine-GRM-2.0-8B on RoboPulse (90.1% vs 71.1%) and RoboReward-8B on RoboRewardBench (MAE 0.358 vs 0.443).
- **Real robots, human ratings:** on RoboRewardBench's RoboArena split (1,000 real-robot rollouts) it has the lowest error of every model
  we ran, 0.687 (best reward model: Robometer-4B 0.834, paired −0.147 [−0.245, −0.048]; RoboReward-8B 0.968; 27B judges 0.867–0.919).
  When two policies attempted the same task, it orders them like the human raters 88.4% of the time (805 pairs; OneJev-27B 88.8%,
  RoboReward-8B 85.3%, Robo-Dopamine 78.3%, Robometer-4B 75.5%).
- **RoboPulse++: AUROC 0.728**, statistically tied with Robo-Dopamine (0.698–0.730 across its readouts) and the 27B judges (0.692–0.700);
  above Robometer-4B (+0.098 [+0.062, +0.131]).
- **Breadth:** it answers all 13 question families of JevAny-Robotics-Public (skill_role 0.567, micro accuracy 80.4%, tied with the
  strongest 27B judge). The reward models can answer 6 of them; on success detection (5 sets) its mean AUROC is 0.872 vs 0.756 for the best
  reward model (Robometer-4B, paired +0.116 [+0.088, +0.144]); OneJev-27B is higher (0.903).
- **Calibrated probabilities:** on RoboRewardBench its probabilities have the lowest NLL of all models we ran, 1.02 vs 1.39 for
  RoboReward-8B (paired −0.37 [−0.52, −0.22]) at similar exact-score accuracy (62.1% vs 63.5%). Robo-Dopamine and Robometer output no
  per-option probabilities.
- **Speed:** batch 1 on one A100, a median 185 ms per RoboPulse pair with the flash-linear-attention / causal-conv1d kernels (294 ms with
  the reference kernels used for the tables) vs 316 ms for Robo-Dopamine-8B (vLLM) and 732–838 ms for 27B judges on two GPUs.

## Robot benchmarks

### Jev-style judges and base VLMs

| Model | Size | RoboPulse acc ↑ | RoboPulse AUROC ↑ | RoboPulse++ AUROC ↑ | RoboPulse++ bal. acc ↑ | RoboRewardBench MAE ↓ | JevAny-Robotics-Public skill_role ↑ |
|---|---:|---:|---:|---:|---:|---:|---:|
| **JevAny-Robotics-4B-LoRA (this model)** | 4B | **71.1** | **0.806** | **0.728** | **65.9** | **0.443** | 0.567 |
| JevAny-Qwen3.5-4B-LoRA (parent) | 4B | 52.7 | 0.588 | 0.641 | 59.1 | 1.057 | 0.149 |
| Qwen3.5-4B zero-shot (constrained choice) | 4B | 49.9 | 0.519 | 0.592 | 50.0 | 1.261 | 0.203 |
| OneJev-4B | 4B | 53.9 | 0.586 | 0.617 | 58.3 | 1.005 | 0.425 |
| Intern-Decision-4B | 4B | 51.6 | 0.547 | 0.653 | 60.5 | 0.820 | 0.303 |
| OmniJev-4B | 4B | 49.7 | 0.536 | 0.556 | 54.6 | 0.811* | 0.264 |
| OneJev-9B | 9B | 57.6 | 0.595 | 0.564 | 54.0 | 0.927 | 0.442 |
| Jev-Omni-12B | 12B | 49.4 | 0.492 | 0.537 | 52.3 | 1.085 | 0.194 |
| Qwen3.8-27B zero-shot (constrained choice) | 27B | 56.7 | 0.604 | 0.569 | 55.2 | 0.992 | 0.472 |
| JevAny-Qwen3.8-27B-LoRA | 27B | 56.6 | 0.627 | 0.692 | 55.7 | 0.692 | 0.549 |
| OneJev-27B | 27B | 60.4 | 0.651 | 0.700 | 62.7 | 0.805 | **0.605** |

RoboPulse: 1,800 pairs, native image resolution. RoboPulse++: 7,720 consecutive-step pairs, native resolution. RoboRewardBench: 2,831 videos, full media,
official group-wise MAE (RoboArena 0.687 / OXE subsets 0.432 for this model). JevAny-Robotics-Public v0.1: 3,990 items, skill_role is the role-balanced
chance-corrected skill. *OmniJev-4B's training data overlaps RoboRewardBench. Every model answered every item.

**Paired comparisons** (this model minus baseline, 95% cluster-bootstrap CI over episodes / task groups):

| Baseline | RoboPulse acc | RoboPulse++ AUROC | RoboRewardBench MAE | JevAny-Robotics-Public skill_role |
|---|---:|---:|---:|---:|
| Parent JevAny-4B | +18.3 [+15.1, +21.4] | +0.087 [+0.049, +0.131] | −0.614 [−0.725, −0.554] | +0.419 [+0.365, +0.463] |
| Qwen3.5-4B zero-shot | +21.2 [+18.1, +24.4] | +0.136 [+0.091, +0.187] | −0.817 [−0.875, −0.732] | +0.364 [+0.318, +0.411] |
| OneJev-4B | +17.2 [+14.1, +20.1] | +0.111 [+0.072, +0.151] | −0.562 [−0.811, −0.541] | +0.142 [+0.100, +0.189] |
| Intern-Decision-4B | +19.4 [+16.5, +22.5] | +0.075 [+0.047, +0.104] | −0.376 [−0.518, −0.341] | +0.264 [+0.218, +0.312] |
| JevAny-27B | +14.4 [+11.4, +17.0] | +0.036 [−0.003, +0.076] | −0.249 [−0.374, −0.220] | +0.019 [−0.027, +0.052] |
| OneJev-27B | +10.7 [+8.0, +13.3] | +0.028 [−0.004, +0.063] | −0.362 [−0.488, −0.336] | −0.038 [−0.081, +0.006] |

Every difference in favour of this model is significant except RoboPulse++ AUROC and JevAny-Robotics-Public against the two 27B judges (ties).
Per-benchmark reports with all slices are in `eval/comparison/`.

### Dedicated robot reward models (same harness)

| Model | Size | RoboPulse acc ↑ | RoboPulse AUROC ↑ | RoboPulse++ AUROC ↑ | RoboPulse++ bal. acc ↑ | RoboRewardBench MAE ↓ | RoboArena MAE ↓ |
|---|---:|---:|---:|---:|---:|---:|---:|
| **JevAny-Robotics-4B-LoRA (this model)** | 4B | 71.1 | 0.806 | **0.728** | 65.9 | 0.443 | **0.687** |
| RoboReward-8B | 8B | 49.9ᵃ | 0.499ᵃ | 0.489ᵃ | 50.0ᵃ | **0.358** | 0.968 |
| RoboReward-4B | 4B | 49.6ᵃ | 0.501ᵃ | 0.507ᵃ | 51.6ᵃ | 0.407 | 0.999 |
| Robometer-4B | 4B | 63.6ᵇ | 0.700ᵇ | 0.630ᵇ | 58.6ᵇ | 0.777ᶜ | 0.834ᶜ |
| Robo-Dopamine-GRM-2.0-8B | 8B | **90.1** | **0.953** | 0.717ᵈ | **68.6**ᵈ | 1.023ᵉ | 0.928ᵉ |

How each model was run (all decisions fixed before scoring the test sets):
- **RoboReward-8B / 4B** (`teetone/RoboReward-*`): model-card prompt verbatim, assistant turn prefilled with `ANSWER: `, softmax over the
  digit tokens 1–5; every frame of the same derived clips. Free generation gives the same RoboRewardBench MAE (0.357 vs 0.358 for 8B,
  0 unparsable). ᵃ RoboReward scores whole rollouts; for the pairwise benchmarks we score the 2-frame video (before, after) minus
  (after, before). It was not trained for pairs, so treat these as an out-of-scope use.
- **Robometer-4B** (`robometer/Robometer-4B`, official code): ᵇ progress difference between the two observations, with the two frames also
  scored in reversed order to remove a position bias. ᶜ 1–5 levels from the last-frame progress, with cut points fitted on a
  label-stratified subset of the RoboReward **validation** split (never the test set).
- **Robo-Dopamine-GRM-2.0-8B** (`tanhuajie2001/Robo-Dopamine-GRM-2.0-8B-Preview`): the inference of PRM-as-a-Judge's `GRMInference`
  (vLLM, same prompt, sampling and model length), its native 8-image layout (reference start / end + three views before and after).
  ᵈ RoboPulse++: hop from the earlier to the later observation, reference start = the episode's first observation, blank goal image
  (the model card's default when no goal image exists), missing views repeated as in the model card; 146 of 23,559 requests exceeded the
  8,192-token limit and were rerun with images downscaled to ≤76,800 px. Its other readouts: forward 0.698, position-debiased 0.730.
  ᵉ RoboRewardBench: hop from the first to the last frame, mapped to 1–5 with cut points fitted on the RoboReward validation subset.
  Robo-Dopamine's RoboPulse lead is not explained by test overlap: on the 939 items whose episodes do not appear in its public
  training data (GRM), it scores 87.5% (this model 68.8%).

**Paired comparisons with the reward models** (this model minus the reward model, 95% cluster-bootstrap CI):

| Reward model | RoboPulse acc | RoboPulse++ AUROC | RoboRewardBench MAE | RoboArena MAE | RoboRewardBench NLL |
|---|---:|---:|---:|---:|---:|
| RoboReward-8B | +21.2 [+17.9, +24.2] | +0.239 [+0.204, +0.272] | +0.085 [+0.039, +0.116] | −0.281 [−0.380, −0.190] | −0.367 [−0.517, −0.220] |
| RoboReward-4B | +21.4 [+18.2, +24.6] | +0.221 [+0.185, +0.256] | +0.036 [+0.005, +0.093] | −0.312 [−0.410, −0.216] | −0.191 [−0.332, −0.060] |
| Robometer-4B | +7.5 [+4.6, +10.3] | +0.098 [+0.062, +0.131] | −0.334 [−0.565, −0.316] | −0.147 [−0.245, −0.048] | – |
| Robo-Dopamine-8B | −19.1 [−21.3, −16.8] | +0.011 [−0.026, +0.045] | −0.580 [−0.773, −0.521] | −0.241 [−0.345, −0.132] | – |

Full reports: `eval/comparison/reward_models_{robopulse,robopulsepp,roboreward,jevany_robotics_public,calibration_ranking_speed}.md`.
Positive RoboRewardBench MAE differences mean the reward model is better. Our harness measures the RoboReward models with *lower*
error than published on the official leaderboard (8B: 0.358 vs 0.665; 4B: 0.407 vs 0.845) on the same 2,831 test videos (HF test split,
revision 469b9af, all labels identical). Free generation reproduces our number and a zero-shot Qwen3-VL-4B-Instruct control lands near its
published value (0.939 vs 1.032), so we report what we measured and do not compare with leaderboard entries.

Ranking policies: on RoboArena sessions where two policies attempted the same task and received different human scores (805 policy
pairs, 134 sessions), the model orders the two policies like the human raters 88.4% of the time (RoboReward-8B 85.3%, RoboReward-4B 83.7%,
Robometer-4B 75.5%, OneJev-27B 88.8%; paired vs RoboReward-8B +3.1 [−0.2, +6.5], vs RoboReward-4B +4.7 [+1.4, +8.0]).

### Baselines

- **Zero-shot Qwen3.5-4B / Qwen3.8-27B (constrained choice)**: the untrained base VLM with the same judge prompt; options are lettered and the answer is
  restricted to the option letters (one forward pass, probabilities = softmax over the letter logits). This is the same scoring as vLLM
  structured-output choice decoding (`structured_outputs.choice` + letter logprobs); the robot suites were run with the transformers implementation,
  JevAny-Multimodal-agentic-Public with vLLM.
- **OneJev / Intern-Decision / OmniJev / Jev-Omni**: released checkpoints run with their own inference code inside the same harness and items.
- **JevAny-Qwen3.8-27B-LoRA** and the parent **JevAny-Qwen3.5-4B-LoRA**: released JevAny checkpoints, same media settings as this model.

## General judgement (retention)

| Benchmark | Qwen3.5-4B zero-shot (vLLM choice) | Parent JevAny-4B | This model | This − parent |
|---|---:|---:|---:|---|
| Transfer v9 (1,046) acc / NLL | — | 78.68% / 0.587 | 78.39% / 0.664 | acc −0.29 [−1.75, +1.05] |
| JevBench public (local rebuild, 231) acc | — | 77.92% | 77.92% | 0.00 [−3.10, +2.64] |
| JevAny-Multimodal-agentic-Public v0.4 (3,208) skill_role / acc | 0.223 / 53.6% | 0.201 / 52.6% | 0.176 / 51.4% | acc −1.2 (n.s., McNemar p = 0.12) |

Accuracy on general (non-robot) judgement is retained within noise; probabilities become over-confident outside robotics
(NLL +0.077 on Transfer, +0.163 on JevAny-Multimodal-agentic-Public), so recalibrate before using them there. For general-purpose judging use the parent
checkpoint; this one is specialised for robotics. The JevBench number is on a local rebuild of the public suite (same 231 items, different
manifest hash) and is not directly comparable to the official 80.09%.

## Training

- Init: LoRA adapter + pointer head of `SimpleJev/JevAny-Qwen3.5-4B-LoRA` (delta mode); base revision `851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a`.
- 1 epoch, 5,780 optimizer steps, global batch 32, lr 1e-5 (OneCycle), LoRA r=8 on all projection modules, head dim 256 + residual 256, bf16.
- Full media: images at native resolution, every frame of the derived video clips (≈1–2 fps, 8–32 frames), up to 16,384 packed tokens.
- `training_config.json` / `training_metrics.json` hold the exact arguments; `eval/training_eval_history.jsonl` the in-training evaluations.

| step | development acc / NLL (3,026) | calibration acc / NLL (2,228) |
|---:|---:|---:|
| 0 (parent) | 0.463 / 1.069 | 0.496 / 0.969 |
| 1,000 | 0.640 / 0.757 | 0.688 / 0.633 |
| 3,000 | 0.723 / 0.650 | 0.778 / 0.499 |
| 5,780 (this checkpoint) | 0.733 / 0.638 | 0.786 / 0.475 |

## Limits

- **Source overlap.** RoboPulse and RoboRewardBench draw on the same public robot datasets as parts of the training data (GRM, RoboReward train).
  Training data were decontaminated against all test sets (episode blocklists + perceptual-hash frame dedup), but scenes and robots overlap.
- **RoboPulse++ threshold.** Training balanced rising/falling 1:1, while the RoboPulse++ test is 81% rising; at the default 0.5 threshold the model
  under-predicts "rising" (accuracy 60.4% vs 69.9% for the parent; a constant "rising" answer scores 81.1%). Ranking quality (AUROC) and balanced
  accuracy are the meaningful metrics here and both improve; set the threshold on your own data. Multi-view (2–3 camera) items and long, slow-progress
  intervals remain the weakest cases.
- **RoboRewardBench's synthetic OXE items.** On relabelled-instruction and clipped-video items the RoboReward models, trained on exactly
  that construction, are clearly better (MAE 0.223 vs 0.297 and 0.418 vs 0.584 for RoboReward-8B vs this model); clipped videos are
  often scored as near-complete.
- **Speed vs RoboReward.** Our full-media protocol gives the model ~2.5× more visual tokens per video than RoboReward's processor budget,
  and Qwen3.5's linear-attention layers are slower than Qwen3-VL attention: per RoboRewardBench video this model takes a median 279 ms
  (fast kernels) vs 142–160 ms for a single RoboReward forward pass and 361 ms for RoboReward-8B's normal generation.
- **Calibration outside robotics** (see above).

## Usage

```bash
# --media is in a pending JevAny PR; until it is merged, use the full-media branch of the fork:
git clone -b full-media https://github.com/BruceSheng1202/JevAny && cd JevAny
pip install -e '.[serve,multimodal]'
jevany serve --checkpoint SimpleJev/JevAny-Robotics-4B-LoRA --device cuda --dtype bf16 \
    --media full --max-state-tokens 16384 --max-packed-tokens 32768
```

**Use `--media full`** (or `JEVANY_MEDIA=full`; also on `jevany decide --checkpoint`): images at native resolution and every frame of a
video, the inputs this model was trained and evaluated with. With it, our evaluation harness returns exactly the probabilities
of the reported runs (checked on 300 RoboRewardBench videos and 300 RoboPulse pairs). The JevAny default (`capped`: the images of a request share 512×512 pixels, a video is 8 frames
within 512×512) costs accuracy, most of all on video:

| | full | capped |
|---|---:|---:|
| RoboRewardBench MAE ↓ (RoboArena MAE) | 0.443 (0.687) | 0.607 (1.289) |
| RoboPulse accuracy / AUROC | 71.1 / 0.806 | 69.4 / 0.786 |

Send short clips like the evaluation's (about 1 frame per second, 8–32 frames); every frame becomes visual tokens. For speed, install the
optional `flash-linear-attention` and `causal-conv1d` kernels (see JevAny's DEPLOYMENT.md): median 185 ms per RoboPulse pair on an A100.

The repository contains LoRA adapter weights and JevAny readout metadata, not the base weights; the base model's license and
access terms also apply, as do the licenses of the training data sources (RoboReward, Robo-Dopamine GRM, Guardian BDV2 / RLBench, DenseReward,
RoboFAC, ViFailback, RoboVQA).