| <!doctype html> |
| <html lang="en"> |
| <head><link rel="canonical" href="https://robo-jev.com"> |
| <meta charset="utf-8"> |
| <meta name="viewport" content="width=device-width,initial-scale=1"> |
| <meta name="description" content="RoboJev-27B, an open model that tells you whether a robot attempt worked: benchmarks, demo and data."> |
| <title>RoboJev · Did it work?</title> |
| <link rel="stylesheet" href="style.css"> |
| </head> |
| <body> |
| <div class="wrap"> |
| <header class="nav"> |
| <a class="brand" href="#">RoboJev<span> ◉</span> |
| </a> |
| <nav class="navlinks" aria-label="Resources"> |
| <a href="https://huggingface.co/josefchen/RoboJev-27B" target="_blank" rel="noopener noreferrer">Model</a> |
| <a href="https://robo-jev.com/demo" target="_blank" rel="noopener noreferrer">Try it</a> |
| <a href="https://robo-jev.com/data" target="_blank" rel="noopener noreferrer">Data</a> |
| <a href="https://github.com/josefchen/robojev" target="_blank" rel="noopener noreferrer">Code</a> |
| <a href="#results">Results</a> |
| <a href="#paper">Paper</a> |
| </nav> |
| </header> |
| <main> |
| <section class="hero"> |
| <h1>Did it work?</h1> |
| <p class="intro">RoboJev-27B estimates whether a robot completed its task. Give it the instruction and ten frames, plus any final auxiliary-camera views. It returns P(success) in a median 0.26 seconds on one H100.</p> |
| <div class="actions"> |
| <a class="pill" href="https://robo-jev.com/demo" target="_blank" rel="noopener noreferrer">Try your own video <span class="arrow" aria-hidden="true">↗</span> |
| </a> |
| <a class="pill secondary" href="https://huggingface.co/josefchen/RoboJev-27B" target="_blank" rel="noopener noreferrer">Get the weights</a> |
| <a class="pill secondary" href="paper/RoboJev.pdf" target="_blank" rel="noopener noreferrer">Read the paper (PDF)</a> |
| </div> |
| <figure class="space"> |
| <a href="cost-wide.webp" target="_blank"> |
| <picture> |
| <source media="(max-width:700px)" srcset="cost-tall.webp"> |
| <img class="chart" src="cost-wide.webp" alt="Accuracy against cost: RoboJev-27B at 95.2% balanced accuracy and $0.192 per 1,000 answers; frontier models at 91.9% to 93.3% and $2.08 to $9.29" loading="lazy"> |
| </picture> |
| </a> |
| </figure> |
| </section> |
| <section class="section" id="film"> |
| <div class="bezel"> |
| <video style="display:block;width:100%;border-radius:24px" controls playsinline preload="none" poster="wall-wide.jpg" aria-label="RoboJev launch film" src="launch-wide.mp4"> |
| </video> |
| </div> |
| </section> |
| <section class="section"> |
| <div class="copy"> |
| <h2>Ring on the peg. Ring slips off.</h2> |
| <p>Same instruction, on a task held out of training. The overlay shows RoboJev’s answer every half second, from the frames up to that moment.</p> |
| </div> |
| <div class="video-pair"> |
| <figure class="bezel"> |
| <video controls playsinline preload="metadata" aria-label="Successful ring-on-peg attempt scored by RoboJev" src="https://huggingface.co/josefchen/RoboJev-27B/resolve/main/examples/robojev-ring-successful.mp4"> |
| </video> |
| <figcaption>The ring stays on.</figcaption> |
| </figure> |
| <figure class="bezel"> |
| <video controls playsinline preload="metadata" aria-label="Failed ring-on-peg attempt scored by RoboJev" src="https://huggingface.co/josefchen/RoboJev-27B/resolve/main/examples/robojev-ring-failure.mp4"> |
| </video> |
| <figcaption>The ring slips off. The score rose first, then fell: an early high score is not a verdict.</figcaption> |
| </figure> |
| </div> |
| </section> |
| <section class="section" id="results"> |
| <h2>How it compares</h2> |
| <div class="copy"> |
| <p>95.2% balanced accuracy on 268 held-out SO-101 runs, at an estimated $0.19 per 1,000 answers. 0.890 AUROC on 459 runs from eight held-out lab/robot domains.</p> |
| </div> |
| <div class="charts"> |
| <figure> |
| <a href="board-tall.webp" target="_blank"> |
| <img class="chart" src="board-tall.webp" alt="Scoreboard: RoboJev-27B, GPT-6 Astra, GPT-6.1 Sol, Claude Opus 5.5, Claude Sonnet 5.5, TypeSafe Jev 1.14 and Perplexity Decider" loading="lazy"> |
| </a> |
| </figure> |
| <figure> |
| <a href="speed-tall.webp" target="_blank"> |
| <img class="chart" src="speed-tall.webp" alt="Seconds per answer: RoboJev-27B 0.26, Jev 1.14 0.72, Perplexity 0.90, Sonnet 5.2, Sol 5.6, Astra 6.9, Opus 7.1" loading="lazy"> |
| </a> |
| </figure> |
| </div> |
| <div class="space" id="leaderboard"> |
| <p class="muted" role="status">Loading results…</p> |
| </div> |
| <details> |
| <summary>All 29 model and protocol entries, including earlier variants</summary> |
| <div id="all-models" class="space"> |
| <p class="muted" role="status">Loading results…</p> |
| </div> |
| </details> |
| <div class="copy"> |
| <p> |
| <a href="https://robo-jev.com/data" target="_blank" rel="noopener noreferrer">View individual answers, compare any two models and download the data →</a> |
| </p> |
| <h3>What was measured</h3> |
| <p>SO-101: 268 held-out runs across eight tasks, with 80 runs from two entirely held-out tasks and 188 held out by session. Unseen domains: 459 runs from eight lab/robot domains. Community: 135 clips with a clear outcome, out of 200 labelled by one person. All community evaluation owners are excluded from fitting.</p> |
| <p>RoboJev and the matched JEV baseline use ten primary-camera frames plus available final auxiliary views. Frontier APIs use four primary-camera frames. The table contains 29 recorded model/protocol entries, including earlier variants; the API comparisons use different amounts of visual evidence.</p> |
| <p>Balanced accuracy averages success and failure recall at a 0.5 threshold. AUROC is how often a success scores above a failure.</p> |
| <h3>What the differences show</h3> |
| <p>Against GPT-6 Astra, paired bootstrap, 95% intervals: unseen-domain AUROC +0.028 (+0.005 to +0.051); SO-101 balanced accuracy +0.019 (−0.016 to +0.052), which does not resolve a difference. These API comparisons use unequal visual inputs. All paired results are in <a href="https://huggingface.co/spaces/josefchen/RoboJev-data/resolve/main/summary.json" target="_blank" rel="noopener noreferrer">the metrics file</a>.</p> |
| <p>Where it doesn’t win: Opus 5.5 has higher community accuracy, and Astra, Sol and Opus have higher community AUROC. The base model with the same ten frames scores 0.900 on unseen labs, compared with RoboJev’s 0.890; the paired interval includes zero. From last frames only, the difference from the previous release is also unresolved (0.776 vs 0.779).</p> |
| </div> |
| </section> |
| <section class="section" id="scale"> |
| <div class="copy"> |
| <h2>501 hours is already a lot of watching.</h2> |
| <p>We indexed 68,555 episodes in 7,641 public LeRobot datasets: 501 hours of video. One answer per episode at the measured speed is about 4.9 GPU-hours, $13.16 on an H100 at $2.69 an hour. At GPT-6 Astra’s API price: $637.</p> |
| <p>These are compute estimates, excluding media processing, networking, allocation and idle time. We have not regraded the catalogue with this release.</p> |
| </div> |
| <figure class="space"> |
| <a href="hub-wide.webp" target="_blank"> |
| <picture> |
| <source media="(max-width:700px)" srcset="hub-tall.webp"> |
| <img class="chart" src="hub-wide.webp" alt="Cost to grade 68,555 LeRobot episodes: GPT-6 Astra $637, Opus 5.5 $399, Sonnet 5.5 $244, GPT-6.1 Sol $143, RoboJev-27B $13" loading="lazy"> |
| </picture> |
| </a> |
| </figure> |
| <figure class="space"> |
| <a href="scale-wide.webp" target="_blank"> |
| <picture> |
| <source media="(max-width:700px)" srcset="scale-tall.webp"> |
| <img class="chart" src="scale-wide.webp" alt="Illustrative fleets of 75,000, 890,000 and 6.5 million robots: annual video hours, equivalent human viewing effort and RoboJev-27B GPUs" loading="lazy"> |
| </picture> |
| </a> |
| <figcaption>Illustrative fleet sizes, with eight hours of video per robot per day and 2,000 viewing hours per person per year. One answer per 26-second episode, H100s running around the clock; media and service overhead are excluded.</figcaption> |
| </figure> |
| </section> |
| <section class="section" id="paper" aria-labelledby="paper-heading"> |
| <div class="paper-layout"> |
| <div class="copy"> |
| <h2 id="paper-heading">Read the paper</h2> |
| <p class="paper-title">RoboJev: An Open Vision-Language Judge for Robot Success Detection</p> |
| <p class="muted">Josef Chen · Independent Researcher · October 2026</p> |
| <p>From real robot attempts to a success probability. The paper documents the architecture, training mixture and calibration, then examines where adaptation helps and where it falls short.</p> |
| <p>Five scientific figures accompany the recorded evaluation: 20,628 predictions across 29 model/protocol entries. The source package includes the figures and bibliography; the reproducibility bundle includes the plotting inputs and scripts.</p> |
| <div class="actions"> |
| <a class="pill" href="paper/RoboJev.pdf" target="_blank" rel="noopener noreferrer">Open the PDF <span class="arrow" aria-hidden="true">↗</span> |
| </a> |
| <a class="pill secondary" href="https://huggingface.co/josefchen/RoboJev-27B/resolve/main/paper/RoboJev-arxiv-source.zip" target="_blank" rel="noopener noreferrer">LaTeX source</a> |
| </div> |
| <p> |
| <a href="https://huggingface.co/josefchen/RoboJev-27B/resolve/main/paper/RoboJev-reproducibility.zip" target="_blank" rel="noopener noreferrer">Reproduce the figures and tables</a> · <a href="paper/CITATION.bib" target="_blank" rel="noopener noreferrer">BibTeX citation</a> |
| </p> |
| <p class="muted">Preprint. An arXiv link will be added when an identifier is assigned.</p> |
| </div> |
| <a class="paper-preview" href="paper/RoboJev.pdf" target="_blank" rel="noopener noreferrer" aria-label="Open the RoboJev paper PDF"> |
| <img src="paper/preview.webp" alt="First page of the RoboJev paper, with the abstract and real robot success and failure examples" width="1236" height="1600" loading="lazy"> |
| </a> |
| </div> |
| </section> |
| <section class="section"> |
| <div> |
| <h2>The model</h2> |
| <figure class="space"> |
| <a href="how-wide.webp" target="_blank"> |
| <picture> |
| <source media="(max-width:700px)" srcset="how-tall.webp"> |
| <img class="chart" src="how-wide.webp" alt="Ten front-camera frames plus the overhead and wrist cameras go into RoboJev-27B: P(success) 1.00 for the attempt that lands the ring, 0.02 for the one where it slips off" loading="lazy"> |
| </picture> |
| </a> |
| </figure> |
| <div class="copy"> |
| <p>JEV-27B-VL, adapted with a rank-32 language LoRA on 52,598 training episode records while the vision encoder stays frozen. The record count is not an independently deduplicated count of physical attempts. It reads the task and ten evenly spaced frames, plus the last frame of each other camera, and returns P(success) with the calibration in <code>decision.json</code>. 54.7 GB in bf16; one 80 GB GPU.</p> |
| </div> |
| <pre> |
| <code>vllm serve josefchen/RoboJev-27B --served-model-name robojev \ |
| --max-model-len 32768 --limit-mm-per-prompt '{"image": 16}' \ |
| --logprobs-mode processed_logprobs --trust-request-chat-template |
|
|
| from robojev import RoboJev |
| judge = RoboJev("http://localhost:8000") |
| judge.done("Put the ring on the peg", "attempt.mp4")</code> |
| </pre> |
| <div class="copy"> |
| <p> |
| <code>robojev_grade.py</code> grades a LeRobot dataset, <code>robojev_overlay.py</code> draws the score on a video or camera feed, and the LeRobot plugin adds a reward model and reward step.</p> |
| <h3>Where it can go wrong</h3> |
| <p>Ten frames can miss a brief button press or an object dropped between frames. Long tasks, wrist-only footage and recordings that stop before the outcome are hard. A confident answer can still be wrong.</p> |
| </div> |
| <div class="actions"> |
| <a class="pill" href="https://huggingface.co/josefchen/RoboJev-27B" target="_blank" rel="noopener noreferrer">Model & code <span class="arrow" aria-hidden="true">↗</span> |
| </a> |
| <a class="pill secondary" href="https://robo-jev.com/demo" target="_blank" rel="noopener noreferrer">Interactive demo</a> |
| <a class="pill secondary" href="https://huggingface.co/datasets/josefchen/RoboJev-evals" target="_blank" rel="noopener noreferrer">Evaluation data</a> |
| <a class="pill secondary" href="https://github.com/josefchen/robojev" target="_blank" rel="noopener noreferrer">Code on GitHub</a> |
| </div> |
| </div> |
| </section> |
| </main> |
| <footer class="footer"> |
| <a href="https://huggingface.co/josefchen/RoboJev-27B#video-credits" target="_blank" rel="noopener noreferrer">Video credits and licenses</a> · <a href="https://huggingface.co/datasets/armnet/armnetbench_v01_robometer" target="_blank" rel="noopener noreferrer">ArmnetBench</a> · <a href="https://huggingface.co/datasets/robometer/rbm-1m-ood" target="_blank" rel="noopener noreferrer">Robometer OOD</a> |
| </footer> |
| </div> |
| <script src="results.js"> |
| </script> |
| </body> |
| </html> |
|
|
