VibeVideoBench: Evaluating Multimodal Agents on Code-Driven Video Generation
Abstract
VibeVideoBench evaluates multimodal agents that generate videos via code, finding that top systems meet core requirements on most tasks but frequently fail due to poor time management.
Diffusion video models produce striking clips, but they cannot reliably render an exact formula, a correct chart, a rule-consistent game or narration synchronized to on-screen events. A different route is emerging: multimodal agents that write code to render every frame and synthesize every sound. We introduce VibeVideoBench, a benchmark for evaluating agents on code-driven video generation. Its 239 self-contained tasks in English and Chinese span 12 categories, from data stories and music-driven pieces to classroom lessons and paper explainers; each asks an agent, running in its native harness inside a sandbox with no image or video generation model, to deliver a finished MP4 with sound within one hour. Evaluation is hierarchical and evidence-based: program checks of the delivery spec, about 13k task-specific rubric checks traced to the requirements they enforce, and five quality dimensions anchored per task, scored by three VLM judge panels. Across nine frontier agent systems and 2,151 scored rollouts, GPT-6 Astra and Claude Opus 5.5 rank first and second with both the judges and human annotators, and the strongest system satisfies every core requirement on 76.3% of tasks. Delivery is a second axis: counting missing videos as zero changes the ranking, and three quarters of the lost videos are final renders that started too late, ran too slowly or never started, a failure of how agents plan their hour. Agents build their own rendering pipelines and inspect their own frames, yet how much they look does not predict quality. The judges' ranking of the systems closely matches blind human preferences.
Community
π¬ VibeVideoBench: Can AI agents make a finished video, entirely with code?
Agents writing code to make videos is a new way to generate video π‘: multimodal agents write code to render every frame and synthesize every sound, so formulas, charts and narration timed to on-screen events come out exact.
Meet VibeVideoBench π, to our knowledge the first benchmark of general multimodal agents on open-domain, code-driven video generation with audio.
π¦ The benchmark
β’ 239 self-contained tasks in English and Chinese across 12 categories: data stories, music-driven pieces, classroom lessons, paper explainers, games, build-and-process videos and more
β’ Agents run fully autonomously in their native harness, in a sandbox with no image or video generation model, only TTS
β’ Deliver a finished MP4 with sound within one hour β±οΈ
π Evaluation: hierarchical and evidence-based
β’ L1 program checks of the delivery spec β
β’ L2 ~13k task-specific rubric checks, each traced to the requirement it enforces π
β’ L3 five quality dimensions, anchored per task π―
β’ Three independent VLM judge panels; audio checks only go to judges that can hear π
π Key findings (9 frontier agent systems, 2,151 scored rollouts)
π₯ GPT-6 Astra and Claude Opus 5.5 rank 1st and 2nd with both the judges and blind human raters
𧩠Far from solved: the best system meets every core requirement on only 76.3% of tasks
π¦ Delivery is a second axis: counting missing videos as zero reshuffles the ranking. Gemini 4 Argon delivers 77% of its videos and drops from 4th to 7th
β° Of 224 runs with no playable video, 169 are final renders that started too late, ran too slowly or never started. The problem is how agents plan their hour
π οΈ Every system keeps its own rendering stack; 95% of runs inspect their own frames, yet how much they look doesn't predict quality (Ο = β0.07)
βοΈ The judges rank systems like humans do (Ο = 0.98) but agree on only 73% of individual pairs: good for ranking systems, not single videos
π Project page: https://vibevideobench.github.io/
Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper