hallucinate-bench

Physics-violation scores for generated robot videos. A frozen DINOv3 backbone plus a trained diffusion keypoint head lifts each video to 3D keypoints; detectors run on those keypoints and judge fixed 16-frame segments against thresholds calibrated on real robot motion.

Layout

train/<reader_id>/diffusion/       checkpoint.pt  meta.json  train_log.jsonl
scores/<cell_id>/diffusion/        summary.json  results.jsonl  metrics.csv
                                   segments.csv  run_manifest.json
                                   render/<original name>.mp4   (skeleton overlay)
                                   render/reel/<tree>.mp4       (all clips of one tree)
web/<bundle>/                      1.mp4 ... N.mp4  segments.json

• reader_id = <robot>.<corpus>.<view>, e.g. fourier_gr1.humanoid_sv.sv1_16x9
• cell_id = <corpus>.<view>.<source>, e.g. cosmos.sv1_16x9.cosmos
• web/<bundle>/segments.json is the file to read; the numbered mp4s beside it are the original clips without overlay.

Scoring pipeline

video frames
  -> DINOv3 ViT-L/16 (frozen)
  -> diffusion keypoint head (the trained part, train/.../checkpoint.pt)
  -> per-frame 3D keypoints in metres (GR-1: 6 joints per arm, 2 arms)
  -> one score per frame per detector
  -> 16-frame segments, reduced to one value per segment
  -> compared to a threshold: the 95th percentile of real robot motion
     measured with the same reader

A violation therefore means "worse than 95% of real robot motion".

Detectors

• rigidity reduces by median: a single stretched frame is measurement noise; the segment violates only if the typical frame is stretched.
• jerk reduces by max: one jittery frame flags the segment.

Reading segments.json

Top of the file — the calibration shared by every clip:

"detectors": {
  "rigidity": {"threshold": 43.71,    "units": "mm",     "reduce": "median"},
  "jerk":     {"threshold": 457409.4, "units": "mm/s^3", "reduce": "max"}
}

Per clip, keyed by the mp4 number:

"8": {
  "source": "high/14_Use the right hand to pick up rubik's cube ... .mp4",
  "fps": 16.0, "n_frames": 93, "n_violated": 5,
  "segments": [
    {"start_frame": 64, "end_frame": 79, "n_frames": 16,
     "rigidity": {"value": 50.0,      "ratio": 1.14, "violated": true},
     "jerk":     {"value": 1518272.4, "ratio": 3.32, "violated": true}}
  ]
}

• "8" is 8.mp4 in the same directory; source is the original path in the source dataset.
• ratio = value / threshold. Below 1 is clean, 1-1.5 is borderline, above 2 is a clear violation, above 3 is visible by eye. (self_collision is inverted: ratio below 1 violates.)
• Segment time in the video = start_frame / fps. The example above is seconds 4.0-5.0 of 8.mp4: the arm bones are off their real length by 50 mm (1.14x threshold) and there is a jerk spike at 3.32x the worst real motion.
• n_violated counts segments with at least one violated detector.
• For an annotated view of the same clip, open scores/<cell>/diffusion/render/<source>.mp4.

Example: rank clips by worst jerk

import json
d = json.load(open("segments.json"))
rank = sorted(((max(s["jerk"]["ratio"] for s in v["segments"]), n, v["source"])
               for n, v in d["videos"].items()), reverse=True)
for r, n, src in rank[:5]:
    print(f"{n}.mp4  jerk {r:.2f}x  {src}")