RoboJev-27B

RoboJev-27B estimates whether a robot completed a task from the instruction, ten frames of an attempt, and available final auxiliary-camera views. Median serial inference is 0.26 seconds on one H100.

Paper (PDF)  ·  Launch page  ·  Try your video  ·  Every result  ·  Evaluation data  ·  Code on GitHub

Paper: RoboJev: An Open Vision-Language Judge for Robot Success Detection — Josef Chen, Independent Researcher. LaTeX source · Reproducibility bundle · BibTeX. This is a preprint; no arXiv identifier has been assigned.

The evaluated weights are pinned at e089a9b69a365744b80765057d86583647af95a9. The documentation update does not change those weights.

On the 268 held-out SO-101 runs, RoboJev-27B achieves 95.2% balanced accuracy at a cost of $0.19 per 1,000 answers. GPT-6 Astra, GPT-6.1 Sol, Claude Opus 5.5, and Claude Sonnet 5.5 achieve 91.9%-93.3% balanced accuracy at a cost of $2.08-$9.29 per 1,000 answers. The paired 95% interval for the balanced-accuracy difference from GPT-6 Astra includes zero. Its recorded API cost is about 48 times the estimated GPU compute cost of RoboJev.

The language models all received the same question from four frames of the main camera, while RoboJev-27B read ten frames plus the other cameras. Costs are API list prices from our runs, and one H100 at $2.69 an hour for RoboJev-27B.

How it works

The client sends the task as free text and samples ten primary-camera frames uniformly from the start to the end of the attempt. It can append the last frame from each additional camera. It reads the scores of false and true at one answer position, then applies the released calibration to return P(success). This takes one forward pass without a generated explanation.

Watch it decide

Both attempts were done on an SO-101 arm, on a task that was out of training. RoboJev is asked every half a second, each time with ten frames distributed throughout the video so far. The overlay displays its answer as it updates.

Results

Compared against 4 frontier language models and 2 decision APIs, on the same episodes:

Median serial response time is 0.26 seconds on one H100. The recorded frontier API medians are 5–7 seconds per call. Local model time and API wall-clock time include different overheads.

Our catalogue constructed from the Hub contains 68,555 episodes in 7,641 sampled public LeRobot datasets, approximately 501 hours of video. Grading all of these via RoboJev-27B would take an estimated 4.9 GPU-hours, or $13.16 as opposed to $637 using GPT-6 Astra. We did not regrade the catalogue for this release yet.

Evaluation details

The release contains 20,628 predictions across 29 model/protocol entries on 927 human-labelled episodes. The 65 ambiguous community clips remain available but are excluded from metrics, leaving 862 scored episodes. Entries include earlier releases and different protocols, not 29 independent architectures.

The released RoboJev and the ten-frame JEV baseline use ten primary-camera frames plus available final auxiliary-camera views. Their image hashes and task text match across all 727 SO-101 and unseen-domain episodes. The frontier API comparisons use four primary-camera frames and verbal probability answers. Other entries identify their own protocol, including whole-video input. The frontier API comparisons do not isolate training effects at equal input size; the matched-input JEV comparison does hold the visual evidence fixed. Training-label teachers also overlap the API baseline panel; see Training below.

Community evaluations

The community evaluation set contains 200 episodes from public LeRobot evaluation datasets (SO-100, SO-101, Koch and other arms). Each episode was labeled by one person. 135 out of these have a clear outcome and were scored. All 106 owners represented by these 200 datasets were excluded from community fitting. Labels come from one annotator, without an inter-annotator agreement estimate.

Model	AUROC	Accuracy
RoboJev-27B	0.984	0.933
RoboJev-27B, first release (four frames)	0.961	0.904
JEV-27B-VL	0.947	0.852
GPT-6 Astra (four frames)	0.991	0.926
GPT-6.1 Sol (four frames)	0.990	0.926
Claude Opus 5.5 (four frames)	0.988	0.941

The first-release and community JEV rows above are retained historical aggregates. Their individual predictions are not available in the current release snapshot, so they cannot be independently recomputed from that snapshot. The paper marks this distinction.

SO-101, ArmnetBench

The SO-101 test contains 268 human-labelled runs: 111 successes and 157 failures across eight tasks. Two entire tasks account for 80 runs; the remaining 188 runs are held out by recording session. Other ArmnetBench sessions contribute training data.

Model	AUROC	Balanced accuracy	ECE	AUROC, held-out tasks
RoboJev-27B	0.993	0.952	0.032	0.986
RoboJev-27B, first release (four frames)	0.987	0.924	0.052	0.973
JEV-27B-VL	0.979	0.881	0.114	0.978
TypeSafe Jev 1.14	0.975	0.829	0.177	0.958
Perplexity Decider v1.2	0.954	0.811	0.116	0.947
Inception Mercury Decide	0.938	0.816	0.109	0.949
Liquid d1-ultra	0.918	0.814	0.062	0.878
Robometer-4B (whole video)	0.777	0.677	0.251	0.812

Unseen lab/robot domains, RBM-1M OOD

RBM-1M OOD contains 459 runs (263 successes and 196 failures) from eight held-out lab/robot domains, including Franka, Koch, Trossen, xArm and SO-101 arms. These are eight domain labels, not eight independently verified institutions. The GPT-6.1 Sol entry has 458 recorded answers; its missing answer is excluded.

Model	AUROC	Balanced accuracy	ECE
JEV-27B-VL	0.900	0.794	0.152
RoboJev-27B	0.890	0.782	0.148
Perplexity Decider v1.2	0.885	0.761	0.168
RoboJev-27B, first release (four frames)	0.872	0.761	0.173
Liquid d1-ultra	0.859	0.735	0.143
Inception Mercury Decide	0.859	0.756	0.205
TypeSafe Jev 1.14	0.841	0.721	0.197
Robometer-4B (whole video)	0.818	0.598	0.414

AUROC is the probability that a randomly selected success receives a higher score than a randomly selected failure. The default threshold for both accuracy and balanced accuracy is 0.5. Balanced accuracy is the mean of the recall on successes and the recall on failures. ECE is expected calibration error, with 10 bins.

Against the first release

In comparison with the previous version on the same episodes, in a paired bootstrap with 95% intervals on the difference in AUROC:

Test	This release	First release	Difference [95% interval]
Community evaluations, whole attempt	0.984	0.961	+0.023 [+0.003, +0.049]
SO-101, whole attempt	0.993	0.987	+0.006 [+0.001, +0.014]
Unseen labs, whole attempt	0.890	0.872	+0.018 [+0.002, +0.035]
SO-101, last frames only	0.966	0.938	+0.028 [+0.009, +0.051]
Unseen labs, last frames only	0.776	0.779	unresolved: -0.003 [-0.021, +0.015]

With identical inputs, adaptation improves SO-101 balanced accuracy over JEV-27B-VL by 7.08 percentage points (label-stratified paired-bootstrap 95% interval: +3.04 to +11.45). On unseen domains, the AUROC difference is −0.0092 (−0.0218 to +0.0026): the evidence does not establish an improvement over the base model. For 28 unseen-lab episodes, the four best models in the initial RoboJev release confidently disagree with the label. For 8 of them that we checked, the video cuts off before the outcome. The disputed examples are in benchmark/bench_ood_v1_disputed.jsonl.

Against frontier models

Versus GPT-6 Astra, the unseen-lab AUROC difference is +0.028 (95%CI +0.005 to +0.051). The SO-101 balanced-accuracy difference is +0.019 (−0.016 to +0.052), which does not establish a difference. On the 80 runs from fully held-out tasks, RoboJev reaches 92.5% balanced accuracy; Astra and Sol each reach 96.25%. The complete release includes all 29 recorded model/protocol entries and their available predictions. Full comparison

The cost of inference at $0.192 per 1,000 answers on one H100 at $2.69 an hour, one answer at a time, does not include download times, decoding or waiting time. The shared demo Space includes GPU allocation time which is why it's slower than a single H100.

Usage

Serve the model using vLLM and then call it from the small client in this repo.

vllm serve josefchen/RoboJev-27B --served-model-name robojev --max-model-len 32768 \
  --limit-mm-per-prompt '{"image": 16}' --logprobs-mode processed_logprobs --trust-request-chat-template

from robojev import RoboJev

judge = RoboJev("http://localhost:8000")
judge.done("Put the ring on the peg", "attempt.mp4")          # cuts ten frames from the video
judge.done("Put the ring on the peg", frames)                 # or pass ten frames, start to end
judge.done("Put the ring on the peg", "attempt.mp4",
           views={"wrist": "wrist_last.jpg"})                 # with other cameras' last frames

The done method returns P(completed). robojev.py requires requests, pillow, and ffmpeg (for video files). The model requires 1x 80 GB GPU (bf16).

If you want to draw the probability on a video or live camera input:

python robojev_overlay.py "Put the ring on the peg" attempt.mp4 out.mp4
python robojev_overlay.py "Put the ring on the peg" --camera 0       # requires opencv-python

The lerobot_plugin directory exposes a reward model for recorded episodes and a processor for thresholded terminal rewards in LeRobot. Downstream policy-learning improvements have not been measured. See its README for integration details.

hf download josefchen/RoboJev-27B --include "lerobot_plugin/*" --local-dir robojev
pip install ./robojev/lerobot_plugin

The prompt ends at [decision]:. The client reads the log-probabilities of false and true at that position, adds the inherited biases from decision.json, divides by the validation-fitted temperature and applies a two-way softmax. decision.json also contains the number of frames the model reads.

Grade a LeRobot dataset

With the server up and robojev_grade.py, robojev.py and decision.json all in a single directory:

python robojev_grade.py owner/eval_dataset --server http://localhost:8000 --out results.csv

For each episode, the command prints the episode index, task, P(done), and verdict (Done or Not yet). The summary includes the number graded, the success rate at the threshold (default = 0.5), and how many episodes are worth a look (with P(done) between 0.2 and 0.8). Results can also be saved as CSV with the --out option.

It will pick a 3rd person camera and parse ten frames/episode via HTTP byte ranges. Use --camera to specify a camera, --episodes to specify episodes, and --header for additional server headers. No token is required for public datasets, but set HF_TOKEN when using private ones. LeRobot v2.0 and v2.1 require requests, pillow and ffmpeg, and v3.0 also requires pyarrow to read its Parquet episode metadata.

Files

• robojev.py: client
• robojev_overlay.py: adds a probability overlay on a video/camera feed
• robojev_grade.py: grades episodes from a LeRobot dataset on the Hub
• lerobot_plugin/: LeRobot reward model & reward step
• benchmark/: test sets + disputed episodes + scores for all models
• examples/: the two videos above
• decision.json: config file, prompt format, token ids, biases, temperature, frame count

Weights occupy 54.7 GB in bf16 and are intended for one 80 GB GPU. Initialization is autotrust/JEV-27B-VL: its decision adapter is merged before the robot-specific LoRA. The pinned checkpoint configuration declares Qwen3_5ForConditionalGeneration and model_type=qwen3_5. Earlier release prose used a different foundation-model name; the paper does not claim that unresolved lineage as verified.

Training

The current release fits 52,598 episode records, with the following post-split counts. This is not an independently deduplicated count of physical attempts. The earlier 33,000-example description refers to the first release.

Source family	Fit episode records
Selected RBM-1M processed sources	28,170
ArmnetBench	1,388
Teacher-labelled LeRobot community records	10,216
Community demonstrations	3,718
ViFailback	4,962
DROID successful conversion	4,144
Total	52,598

Annotation and split provenance

The executed annotation pipeline requires GPT-6.1 Sol and GPT-6 Astra, each judging four primary-camera frames. Both must agree on the binary outcome and each must assign a probability at most 0.1 or at least 0.9. If a Claude Opus 5.5 answer is available, it must agree on the binary side and can veto the candidate; it need not independently meet that confidence margin. The soft target averages all available teacher probabilities. The training loader then applies the final 0.1 confidence margin to that mean.

All 11,387 saved unique target values match the executed consensus artifact. The loader reports 11,415 accepted community records before exclusions, leaving an unresolved 28-record discrepancy. Its 282 held-out Koch records map to 272 unique validation IDs. The paper reports these limits and does not equate loader records with unique physical attempts. After validation and robot exclusions, 10,216 community records enter fitting. All 106 owners in the human-labelled community evaluation are excluded from fitting.

The required annotation teachers overlap the frontier API benchmark panel. The benchmark labels are held out, but these comparisons are not independent of teacher-assisted training. Earlier prose describing Sol/Opus consensus on ten frames was incorrect; this card follows the executed run, as does the paper.

Adaptation and calibration

The language component uses rank-32 LoRA, alpha 64 and dropout 0.05, with 233,455,616 trainable parameters. The vision encoder and visual merger stay frozen. Training uses one epoch at learning rate 5e-5, AdamW with zero weight decay, 5% warmup and cosine decay, gradient clipping at 1.0, and seed 7. A per-GPU batch of 1 with accumulation 2 gives an optimizer batch of 14 on the documented seven-GPU run. The source records report approximately 5.5 hours on seven H100s; a shorter resumed-process log is not the total duration.

Original-record cross-entropy uses inverse-class-frequency weights within each source, clipped to [0.25, 4]; a source with only one outcome keeps weight 1. Synthetic negatives receive weight 0.5. These weights apply to cross-entropy, not the KL term. Repetition and augmentation produce 102,172 training examples in the epoch. Eligible episodes use ten-frame evidence with probability 0.6; other branches retain four-frame and final-view inputs. Resizing, JPEG recompression and colour jitter augment the images. Truncated positives and mismatched instructions provide heuristic negatives, which can be noisy.

On selected research sources, DROID and ViFailback, an untempered two-token KL penalty KL(base || adapted) with weight 1 anchors the model to JEV's decision. ArmnetBench and community examples are unanchored. This differs from the four-frame evidence used to create community teacher labels.

The run-specific selection procedure chooses the best held-out community AUROC among nonzero LoRA scales whose AUROC is no more than 0.005 below the base on each protected validation group. Those groups include AutoEval, FINO-Net, RoboArena's 0825 release and the held-out community Koch group. The run-specific validation procedure selects scale 1.0. Benchmarks were inspected during development, so this is not a claim of a fully blind test history. Temperature is fitted on held-out validation predictions; inherited decision-head biases are retained. The released decision.json uses temperature 0.9, tokens false (3721) and true (1802), and biases +0.00011960109 and −0.00011981795 respectively.

Intended use and limitations

Use RoboJev for offline episode review, dataset grading and research into task-conditioned reward signals. The probabilities are model estimates, not measured physical success rates. The model is not a safety controller.

• Ten frames can miss brief events. Long tasks, wrist-only views and recordings that stop before the outcome remain difficult.
• Specialization improves the SO-101 test relative to the same-input base, but unseen-domain AUROC is 0.890 versus the base's 0.900. No universal advantage over frontier models is established.
• Calibration changes with domain: ECE is 0.032 on SO-101 and 0.148 on unseen domains. A confident answer can still be wrong.
• Community labels are partly teacher-generated; the human evaluation has one annotator, with 65 ambiguous clips excluded from metrics. The student shares training provenance with some API baselines.
• The 52,598 fit count is a record count. The complete historical training manifest, the 28-record community discrepancy and part of the training licence/revision chain remain unresolved.
• The catalogue of 68,555 sampled episodes has not been regraded with this checkpoint. Cost extrapolations exclude media processing, networking, allocation, idle time and service overhead.
• The LeRobot integration is released, but no downstream policy-learning gain or real-robot safety guarantee has been demonstrated.

License

The released weights and inference tools declare Apache-2.0. Training data retain their source-specific terms. The release records Apache-2.0 for ArmnetBench; MIT for RoboArena, LIBERO, Meta-World and ViFailback; and CC BY 4.0 for RoboReward, MolmoAct and DROID. Open X-Embodiment has source-specific licences. The copies of SOAR, AutoEval and FINO-Net accessed through RBM-1M did not state an explicit dataset licence in the audit. Community source availability has changed, and the complete historical training licence chain has not been independently reconstructed. The paper's source ledger records those gaps. It does not redistribute the training corpus.

Citation

@unpublished{chen2026robojev,
  title = {RoboJev: An Open Vision-Language Judge for Robot Success Detection},
  author = {Chen, Josef},
  year = {2026},
  note = {Preprint},
  url = {https://huggingface.co/josefchen/RoboJev-27B/resolve/main/paper/RoboJev.pdf}
}

Video credits

The robot clips used in the launch video were taken from these public LeRobot datasets hosted by HuggingFace. Thanks for all your help sharing your evaluation data! If you see that one of these datasets is yours, and would like to have its clip removed, please open a discussion on this repo.

• armnet/armnetbench_v01_lerobot_so101 (ring-on-peg attempts, episodes 37 and 42; the 24 race episodes)
• Aipolabs/eval_so101 (episode 8)
• Ayush-Sharan-1/eval_v1_ACT_c-4g_tC-T_fC-T_d-250525_v-1_059000 (episode 7)
• BranoSandy/eval_act_so100_test_2 (episode 1)
• Deepkar/eval_redblock-pick-place-50ep_act (episode 7)
• EGLima/eval_act_pen_task_v5 (episode 3)
• Hennadiy/eval_so100_50_act (episode 2)
• HenryZhang/eval_Koch_30 (episode 1)
• Jeevan-HM/eval_record-test (episode 7)
• KHandsome/eval_wow_with (episode 5)
• Kazakus/eval_DATASET_NAME_test_2026-04-30_20-33-03 (episode 0)
• Lerori/eval_act_cuda_so101_1 (episode 1)
• Lisette1231/eval_2025120so101smovla_banana_turn (episode 8)
• M3thRocks/eval_act_so101_test (episode 1)
• Marco10pez/eval_Marcos_demoV3 (episode 0)
• Ready321/eval_pi05_redcube_dong17_demo (episode 5)
• RobotisSW/eval_koch_test_vertical (episode 29)
• Sengankou/eval_pi05-T1k10-instRight-idPlace (episode 7)
• ShiangYu/eval_b601_smolvla_v3 (episode 0)
• ShishKebab04/eval_pi0_eraser_so101 (episode 0)
• ThePayload/eval_cobot_teleop_data_v6 (episode 1)
• Whakamua/eval_act_so100_test520250604_195916 (episode 0)
• WoojongKim/eval_smolvla_policy_omx_gelsight_env1_square_20260903_132708 (episode 4)
• YasmineMakni/eval_act_so100_mvt_ball_2 (episode 5)
• Yteria/eval_act_so101_cube_bowl (episode 3)
• airthebear/eval_act_so100_GL (episode 7)
• aivanni/eval_pi05_kettle (episode 3)
• anandsmilesath/eval_pick_block_bowl (episode 0)
• anasnajaa/eval_record_sort_pens_1 (episode 6)
• ankithreddy/eval_repair_2 (episode 0)
• ases200q2/eval_UR10_RTDE_SpaceMouse_EE_20261014-173627 (episode 2)
• asldjflasjdlajflsdkfal/eval_cube_to_box (episode 0)
• azaracla/eval_so101_pick_3dprint (episode 13)
• badwolf256/eval_baseball (episode 3)
• bensprenger/eval_act_right_arm_tape (episode 7)
• bjahoor/eval_green_block (episode 3)
• cupnb/eval_ID_Box_5-4 (episode 3)
• douzi999/eval_act_so100_test_13 (episode 5)
• fleford/eval_test_10 (episode 2)
• fracapuano/eval-1 (episode 8)
• funXedu/eval_so101_act_v2 (episode 0)
• gerotropic/eval_so101_test (episode 8)
• godnpeter/eval_pikachu13 (episode 0)
• hangVLA/eval_red_ball5 (episode 0)
• helper2424/eval_smolval_movet_to_blue_test2 (episode 0)
• hjkso1406/eval_smolvla-so101-4tasks-aug-dpo-b1-s000750_pick-place_25 (episode 0)
• honcabanthan/eval_lerobot_gr00t_v3 (episode 3)
• ilikirobot/eval_picknplace_blue_block_20260331 (episode 38)
• izuluaga/eval_actpolicy (episode 15)
• joooonhocho/eval_so100_doll_in_bowl_2 (episode 1)
• kagachi2018/eval_so101_smolvla_ding_vlapruner (episode 2)
• kd-forge/eval_act_rac (episode 11)
• mchen02/eval_act_so100_test2 (episode 19)
• mikechambers/eval_block_cup_13_3 (episode 0)
• mjay2023/eval_blue_block (episode 0)
• nuaajiaz/eval_act_so100_test (episode 0)
• orofra/eval_bluecube1 (episode 2)
• pbvr/eval_act_so101_004 (episode 4)
• pengjunkun/eval_act_so100_test (episode 0)