MotionGraphicsBench: Evaluating Motion Graphics Generation with Programmatic Verification

Jiaju Ma^1,2,*    Ashvini Kumar Jindal^2,*    Chuhua Wang²    Sumithra Bhakthavatsalam^2
1Stanford University    ²Figma    ^*Equal contribution

tl;dr: We introduce MotionGraphicsBench, an expert-curated benchmark for evaluating how well AI models can animate static vector graphics. For each of our 100 tasks, we give a model a complete input static SVG and a motion edit prompt and ask it to animate the supplied elements. We evaluate 32 proprietary and open-weight LLMs; only four exceed 50% pass@1 at their highest available reasoning-effort settings. We automate evaluation with deterministic, human-readable verification programs that we construct using a multi-agent pipeline. On 180 held-out animations, our verification programs achieve 93.3% raw agreement with human judgments, compared with 83.3% for a VLM baseline. Explore our tasks and animations at https://motion-graphics-bench.github.io/, find our benchmark and code at https://huggingface.co/datasets/figma-ai/motion-graphics-bench, and read our full paper at https://arxiv.org/abs/2611.10592.

Introduction

The prompt "pelican riding a bicycle" by Simon Willison has become a popular test of how well large language models (LLMs) can generate motion graphics animations. However, creating animations solely from a text prompt requires the model to have two separate capabilities: vector graphics generation, which creates the static graphic, and motion generation, which adds animations to the static graphics. Instead of starting with a text prompt, the more common workflow for creating motion graphics begins with an existing design: a designer supplies the visual elements and asks an animator to add motion while preserving the design. This requires correctly following instructions about poses, trajectories, timing, coordination, and changes in appearance or visibility. Benchmarks such as SVG-Bench and VGBench evaluate static vector graphics generation but do not measure these capabilities. End-to-end text-to-animation tests make motion generation difficult to isolate because failures in graphic creation and motion generation are mixed together. It remains unclear how reliably models can generate motions that combine spatial, appearance, and temporal requirements.

To evaluate motion generation independently of graphic creation, we introduce MotionGraphicsBench, an expert-curated benchmark of 100 motion editing tasks. The tasks cover 17 animation requirement types across spatial structure, appearance and visibility, and temporal sequencing. We set up each task following a designer's workflow: we give a model a complete input static SVG and a motion edit prompt, and then ask it to animate the existing elements. We use task-specific deterministic verification programs to evaluate model-generated animations. Each program contains a series of human-readable checks, each tied to a specific part of the motion edit prompt.

We evaluated 32 proprietary and open-weight LLMs from 12 providers, including models from the GPT, Claude, Gemini, Muse, Grok, Qwen, DeepSeek, GLM, Kimi, Nemotron, and Gemma families across multiple reasoning-effort settings. At their highest available effort settings, only four models exceeded 50% pass@1. GPT-6 Astra at maximum reasoning achieved the highest score of 67.7%. The strongest open-weight settings were GLM 5.3 (max) at 39.8%, Qwen 3.8 (xhigh) at 39.4%, and Qwen 3.8 Flash-Next (xhigh) at 34.0%. For 15 models, we also evaluated pass@k using 32 independent attempts per task, with pass@32 ranging from 38.0% to 95.0%. Our verification programs enabled us to analyze model performance beyond just statistics. We examined their human-readable checks and individual outcomes to identify which parts of a motion edit prompt each animation satisfied or missed. This analysis revealed the three most common failure modes: failing to match requested poses, failing to follow motion trajectories, and failing to coordinate related events correctly.

To construct the benchmark, we derive input static SVGs and human-reviewed motion edit prompts from real-world Lottie animations in LottieAnimation-660K. We build a deterministic, human-readable verification program for each task using a multi-agent pipeline with human-labeled animations. The programs accept different valid implementations and report which checks pass or fail rather than requiring an exact match to one source Lottie animation. On a balanced held-out set of 180 animations, our programs achieve 93.3% agreement with human judgments, compared with 83.3% for a VLM judge based on GPT-6 Astra. Explore our tasks and animations at https://motion-graphics-bench.github.io/, and find the dataset and release code in the Hugging Face repository. See the paper for the full methods and results.

What a task in MotionGraphicsBench looks like

Figure 1. The Scissors animation closes, reopens, and returns to its starting pose, but the blades never become nearly closed. Since every check must pass, this generation is unsuccessful.

MotionGraphicsBench contains 100 tasks that focus on evaluating motion graphics generation. For each task, we give a model two inputs: an input static SVG and a motion edit prompt, and ask it to animate the supplied visual elements rather than create missing ones.

The input static SVG contains every visual element needed for the animation, including geometry that may initially be hidden. It provides a reference appearance but does not necessarily define the animation's first frame. In the Scissors example in Figure 1, it already contains the blades, handles, and central screw. The motion edit prompt describes what should move or change and how those changes should relate in space and time. Designers can often describe their intended motion without knowing the exact parameters needed to produce it. Therefore, our prompts describe animation behaviors in terms of relative spatial and temporal relationships rather than exact keyframes, coordinates, or frame numbers, consistent with research on how people describe spatio-temporal relationships. For Scissors, the motion edit prompt asks the model to rotate the two blades in opposite directions around the screw, bring the blades into a "nearly closed, almost horizontal overlap," pause briefly, then reopen, without specifying exact rotation angles.

Given the input static SVG and motion edit prompt, we ask the model to generate an animation. Vector animations can be created through many means, such as JavaScript libraries, CSS animations, or formats such as Lottie, but these can all be represented as a sequence of SVG frames that each records the animated scene at one point in time. We use these frames as the input to evaluate the animation, so our verification checks do not depend on how the motion was produced.

How we verify an animation

Rendering every output as SVG frames gives us a common input for evaluation, but we still need to determine whether the motion satisfies the prompt. Because a motion edit prompt specifies required relationships rather than an exact trajectory and timing, several animations can satisfy it. Comparing the generated frames with a single reference animation would therefore be too restrictive. A vision-language model (VLM) judge avoids this restriction, but its judgments can vary and it can be difficult to inspect how individual requirements affect the decision. Instead, we use deterministic human-readable verification programs to evaluate generated animations at scale. With these programs, we can reproduce each decision and inspect which prompt requirements an animation satisfies or misses.

Our verification setup is inspired by MoVer, which expresses motion requirements as executable checks over SVG frames. Because each prompt specifies different relationships and events, every task has its own Python verification program. The program takes the rendered SVG frames, measures how the elements' geometry, appearance, and relationships change over time, evaluates the prompt requirements as checks, and returns a verification report with the overall decision, which checks passed or failed, and any execution errors. The program evaluates the rendered behavior, not whether the generated code resembles a reference program. We count a generated animation as successful only if it renders and every check within its verification program passes. Our verification programs run on CPU and do not require a model call for each evaluation. In the Hugging Face repository, we release all 100 verification programs together with the helper and runtime code needed to execute them. The full paper describes the measurement and execution details. We describe how we construct our verification programs in How we build our verification programs and how we use human judgments for construction and evaluation in Alignment with human judgments.

As an example, Figure 1 shows three selected checks from the Scissors verification program: the closing-and-reopening motion, the nearly closed pose, and the original open pose. The excerpt below shows their final tests from the stored verification program.

def ver_opposed_snip_and_reverse(matrix):
    ...  # Check directions and measure closing/reopening intervals.
    return verdict(together, ...)

def ver_nearly_closed_overlap(matrix):
    ...  # Measure blade overlap, orientation, and handle exposure.
    return verdict(
        gap <= uncertainty and horizontal
        and exposure >= -_epsilon(matrix), ...)

def ver_original_open_pose(matrix):
    ...  # Compare the starting and ending poses with the input static SVG.
    return verdict(worst <= _epsilon(matrix), ...)

The nearly closed check does more than test whether the blades move closer. Its final condition combines blade overlap, a roughly horizontal orientation, and the requirement that the halves do not merge completely. In Figure 1, the blades stop too far apart, so this check fails even though the closing-and-reopening and return-pose checks pass. Because every check must pass, the generated animation is unsuccessful.

How models perform on MotionGraphicsBench

We evaluated 32 models from 12 providers across 47 model-setting combinations with different reasoning-effort levels. For the pass@k study, we selected the highest-available-effort setting from each of 15 models and made 32 independent generation attempts per task. We made one attempt per task for the other 32 settings, giving 51,200 attempts in total. The release includes every model-study attempt, with its generated program, recorded model settings, response index, rendered SVG frames when available, and verification outcomes.

For a controlled comparison, we ask every model to generate an animation as a JavaScript program using the GSAP animation library. GSAP has been an industry-standard SVG animation library for over a decade and is used across millions of websites, making its API likely to be well represented in model training data. We use MoVer's GSAP rendering engine to execute each program in a browser and convert it into the SVG frames consumed by our verification programs. Here is an example program produced by Gemini 3.1 Pro at high reasoning effort that renders to the animation shown in Figure 1:

const root = document.querySelector("svg");
const upperHalf = root.querySelector("#scissors-1");
const lowerHalf = root.querySelector("#scissors-2");

const tl = gsap.timeline();

// Use the center screw as the shared rotation pivot.
tl.set([upperHalf, lowerHalf], { rotation: 0, svgOrigin: "1176.172 1080.996" }, 0);

// Rotate both halves toward the closed pose at the same time.
tl.to(upperHalf, { rotation: 22, duration: 1.3, ease: "power2.inOut" }, 0);
tl.to(lowerHalf, { rotation: -22, duration: 1.3, ease: "power2.inOut" }, 0);

// Return both halves to the original open pose.
tl.to(upperHalf, { rotation: 0, duration: 1.3, ease: "power2.inOut" }, 1.7);
tl.to(lowerHalf, { rotation: 0, duration: 1.3, ease: "power2.inOut" }, 1.7);

A generation attempt is one independent request for one animation program. Attempts do not receive verification feedback from earlier generations, so pass@k measures repeated generation rather than iterative correction. See the full paper for the complete generation instructions, rendering procedure, and model settings. We include the generation pipeline code alongside the benchmark in the Hugging Face repository.

pass@1

Figure 2. Pass@1 across all 47 evaluated settings from 32 models. Each bar represents one model-setting combination; hatched bars identify open-weight models. Open the image to read the labels at full size.

pass@1 is the average success rate of one independent generation, weighting each of the 100 tasks equally. Across all 47 settings, pass@1 ranges from 6.0% to 67.7%, with a mean of 30.3%. GPT-6 Astra at maximum reasoning effort has the highest score. Reasoning effort improves pass@1 by 7–13 percentage points in five of six matched within-model comparisons. GPT-5.6 Sol has the largest gain, rising from 35% to 48%, while DeepSeek V4 Flash is the exception, falling from 31% to 29%.

To compare proprietary and open-weight models without counting some models more often than others, we select one highest-effort setting per model. The 16 proprietary models average 38.8%, while the 16 open-weight models average 21.8%. There is also overlap between the groups: GLM 5.3 (max) reaches 39.8% and Qwen 3.8 (xhigh) reaches 39.4%, outperforming several proprietary settings. Performance also varies substantially across tasks. Averaging the 47 settings equally, the median task-level pass@1 is 20.6%: 16 tasks have pass@1 below 5%, while 25 reach at least 50%. Current models therefore follow some motion edit prompts reliably but struggle consistently with others; our failure-mode analysis examines which requested behaviors they most often miss.

pass@k

Figure 3. Pass@k for the 15 models with 32 independent attempts per task. Each curve uses one selected highest-effort setting. This is a smaller study group than the 47 settings in Figure 2.

pass@k is the chance that at least one of k independent attempts succeeds, weighting each task equally. Pass@32 therefore asks whether any of 32 attempts passes every check. Across the 15 models in the pass@k study, mean success rises from 36.7% at pass@1 to 73.7% at pass@32. The pass@32 scores range from 38.0% to 95.0% and exceed pass@1 for every model in the study. Models that perform better on the first attempt plateau earlier. For the five models with the highest pass@1, mean success rises from 56.4% at pass@1 to 88.2% at pass@32, but the final increase from 16 to 32 attempts is only 2.6 percentage points. For the five models with the lowest pass@1, mean success rises from 18.0% to 55.4%, and the same final doubling still adds 6.7 points. Lower-performing models therefore continue to benefit from additional attempts after higher-performing models have begun to level off.

How models fail

Figure 4. Recorded outputs illustrating reference-pose, motion-trajectory, and event-coordination mismatches.

An animation can fail overall while still following parts of the motion edit prompt. Among the 29,632 failed attempts with complete, error-free reports in the repeated-generation group, 93.6% pass at least one check and 31.6% fail exactly one. A single failed check can combine several conditions, so it does not necessarily represent just one missed requirement.

Different generations can satisfy different parts of the motion edit prompt without producing one complete success. For each model and task, we examine all 32 attempts together. In 125 of the resulting 1,500 comparisons (15 models × 100 tasks), every verification check passes in at least one attempt, but no single animation passes them all. These cases span 52 of the 100 tasks and all 15 models in the pass@k study.

Because each verification program contains human-readable checks tied to specific animation requirements, its report shows which requirements in the motion edit prompt an animation satisfies or misses—not only whether it passes overall. Analyzing these check outcomes reveals three common failure modes. A reference-pose mismatch occurs when an element misses a pose shown by the input static SVG or requested by the motion edit prompt; in Scissors, the blades merge instead of remaining distinct at the closed pose. A motion trajectory mismatch occurs when an element follows the wrong path, direction, or distance; in TWS Earbuds, one earbud descends instead of following the requested mirrored upward-and-outward path. An event coordination mismatch occurs when related motions have the wrong order or overlap; in Speed-2, the green status remains visible after the needle leaves its range instead of fading as it departs. Figure 4 shows these examples; see our full paper for their rates, the complete failure taxonomy, and the analysis method.

How we build MotionGraphicsBench

Figure 5. We select 100 source Lottie animations, recover hidden geometry to create complete input static SVGs, and construct motion edit prompts through LLM-assisted drafting and human review. The telephone example recovers ringing arcs that are stored but invisible in the selected frame.

We construct the 100 MotionGraphicsBench tasks from 100 distinct source Lottie animations in LottieAnimation-660K, a collection of real-world motion designs sourced from professional sites such as IconScout and LottieFiles. We use each source Lottie animation to construct the input static SVG and human-reviewed motion edit prompt; evaluated models do not receive it or need to reproduce it exactly. We build these inputs in three stages: 1) select source Lottie animations, 2) prepare input static SVGs, and 3) construct motion edit prompts (Figure 5).

• Source Lottie animation selection. Lottie preserves layers, shapes, transforms, keyframes, and easing curves, allowing us to identify the animated elements and motions. To keep the task focused on motion generation rather than identifying unnamed SVG elements, we retain only animations whose animated layers or objects can be referenced clearly in a motion edit prompt. We rank the candidates by their animated properties, objects, and keyframes. Human reviewers inspect 1,000 candidates near the median complexity so the selected tasks contain nontrivial motion without becoming too complex to be described precisely with natural language. They also remove candidates with duplicate graphics and motions.

• Input static SVG preparation. To model a workflow in which an animator receives an existing design, we choose a representative frame in which the main objects are clearly visible and normally sized, avoiding empty, cropped, or transitional frames. An LLM proposes one frame from five samples, and a human reviewer either confirms it or chooses a better one. The input static SVG must contain every element needed for the requested motion so the task does not require creating missing graphics. Because a Lottie renderer can omit geometry that is inactive or fully transparent in the selected frame, we extend the intervals in which layers are active, disable transparency-based geometry removal at the selected timestamp, and then restore the original visibility of hidden groups. This geometry completion preserves the selected frame's appearance while recovering elements such as the telephone's hidden ringing arcs in Figure 5.

• Motion edit prompt construction. The source Lottie animations do not include natural-language instructions for adding their motion to an input static SVG, so we ask an LLM to draft each motion edit prompt from the completed input static SVG, its PNG rendering, sampled frames from the source Lottie animation, and the source Lottie JSON. The draft names the relevant elements and describes directions, pivots, repetition, ordering, overlap, coordination, timing, and end states. It uses relative spatial and temporal relationships to describe the intended behavior without forcing one exact trajectory or implementation. A human reviewer approves, edits, or requests a targeted revision, removing ambiguity and unsupported claims so the final prompt is clear and complete.

See our full paper for the complete source-selection, input static SVG construction, and motion edit prompt review protocols.

What the tasks in MotionGraphicsBench cover

Figure 6. All 17 requirement types grouped into spatial structure, temporal sequencing, and appearance and visibility. Counts show the number of distinct tasks whose prompts include each type.

To summarize the animation behaviors requested across the benchmark, we organize them into a taxonomy of 17 animation requirement types across spatial structure, temporal sequencing, and appearance and visibility. Each motion edit prompt combines 7–14 types. All 100 tasks combine spatial and temporal requirements, and 65 tasks also include appearance or visibility changes. The benchmark emphasizes the temporal ordering and geometric state constraints needed for coherent motion while also covering deformation, progressive reveal and erasure, and changes to color, fill, and stroke. The source Lottie animations average 3.50 seconds in length, with a standard deviation of 1.89 seconds; the prompts average 193.8 words, with a standard deviation of 57.2 words. See our full paper for definitions and examples of all 17 requirement types.

How we build our verification programs

Because a motion edit prompt can have multiple valid animations, each verification program must accept allowed variation while rejecting animations that violate the prompt. We construct one program per task by writing, testing, and revising its checks against real and synthesized animations. Two GPT-6 Astra-based agents carry out this process: a verification program writer that develops the program and a test case synthesizer that creates additional animations to test it. (Both agents work with an animation matrix, a compact representation of SVG frames adapted from MoVer; see our full paper for details.)

Figure 7. The writer receives execution reports for revision. The synthesizer supplies test cases without direct access to the candidate program or its reports.

To show the agents examples of valid variation and real prompt violations, we build a human-labeled construction set of 1,585 model-generated animations from 25 model settings: 410 approved by humans and 1,175 rejected. For each task, both agents receive its input static SVG, motion edit prompt, source Lottie animation, and assigned human-labeled animations with review notes. The source Lottie animation shows one valid realization, human-approved animations show allowed alternatives, and rejection notes identify missed requirements. The Alignment with human judgments section describes how we collect these labels and separate construction animations from held-out evaluation data.

The verification program writer translates the motion edit prompt into readable Python checks. We provide it with a verification library of reusable functions for measuring geometry, motion, timing, visibility, and relationships between elements across SVG frames. The writer selects measurements and tolerances based on the requested behavior and animation properties, then runs the candidate program on the source Lottie animation and all available construction examples. The resulting reports identify checks that miss a violation or reject a valid animation, guiding the writer's revisions.

The test case synthesizer creates additional animations to test requirements that the human-labeled examples leave untested or violate together. We provide it with an animation transformation library of functions that deterministically edit SVG frame sequences. Starting from the source Lottie animation or a human-approved animation, it can change timing, direction, movement range, position, appearance, or SVG structure. Positive cases preserve every requested behavior while varying an allowed choice; negative cases introduce a targeted violation. The synthesizer can also pair positive and negative cases to test the same motion requirement, with one animation satisfying it and the other violating it. The synthesizer assigns each case a label and an explanation tied to the motion edit prompt. It also receives descriptions of earlier cases, but not the candidate verification program or its execution reports.

The construction loop alternates between testing, adding cases, and revising the program. After the writer tests its initial candidate, the synthesizer can add up to 20 cases per task, with more negative cases when human approvals dominate and more positive cases when rejections dominate. The source Lottie animation, human-approved animations, and positive synthesized cases must pass every check; human-rejected animations and negative cases must fail at least one. While synthesis slots remain, each check must also reject a known negative example. After the budget is exhausted, we record checks without such an example as coverage gaps. The writer can challenge each synthesized case once by citing a prompt clause and frame measurements. The synthesizer can keep, repair, or replace that case while retaining its assigned label and budget slot. Held-out animations and labels remain unavailable throughout construction, and we freeze the completed programs before evaluating our verification programs on them. See our full paper for the complete construction procedure.

Alignment with human judgments

We collect human judgments for two purposes: to guide verification-program construction using human-approved and human-rejected animations, and to evaluate how closely our completed verification programs align with human judgments on animations from new model-generated programs.

We ask models to animate the benchmark's 100 input static SVGs according to their motion edit prompts and collect human judgments for 1,980 generated animations across 29 model settings from seven providers, covering different models and reasoning-effort levels. We onboard the annotators with a calibration task with example animations. We then assign two annotators to independently inspect each animation alongside its input static SVG and motion edit prompt and ask each annotator to judge whether the animation follows its motion edit prompt, not its aesthetic quality or exact similarity to the source Lottie animation; approval requires every requirement to be met, while one violation is enough for rejection. Annotators can reveal the source Lottie animation as one valid example to clarify the motion edit prompt. For each rejection, we ask the annotator to write notes quoting the violated requirement in the motion edit prompt and describing the visible violation. If the two assigned annotators disagree on whether to approve or reject an animation, those same annotators review the animation again and discuss its motion edit prompt. We record a final approval or rejection only after both annotators agree. The human-reviewed animations receive 500 approvals and 1,449 rejections, plus 28 cannot-judge outcomes for technical failures that leave no animation to inspect and three unresolved approval–rejection differences. We include records for all 1,980 human-reviewed animations, the annotators' judgments and written notes, and the available verification outcomes in the MotionGraphicsBench Hugging Face repository.

Evaluating our verification programs

To measure how closely our verification programs align with human judgments on model-generated animations unseen during verification-program construction, we hold out all animations and human labels from four model settings. We use 1,585 human-labeled animations from the other 25 settings to construct our verification programs. For evaluation, we reserve 180 animations generated with the four held-out settings of Gemini 3.1 Pro, Gemma 4 31B IT, GPT-5.5, and Sonnet 5. The 180-animation test set contains 90 human-approved animations and 90 human-rejected animations and covers all 100 benchmark tasks.

We compare our verification programs with a VLM baseline on the same 180 held-out test animations. The VLM baseline uses GPT-6 Astra at maximum reasoning effort and receives each animation's motion edit prompt, a rendering of its input static SVG, and up to 64 evenly spaced, timestamped PNG frames from the generated animation, including the first and last frames. The VLM baseline returns an approval or rejection and assesses each motion edit prompt requirement as satisfied, violated, or unclear. It quotes each requirement, describes what the animation shows at specific times, and notes any uncertainty about its judgment.

Table 1. Alignment of our verification programs and the VLM baseline with human judgments on 180 held-out animations.

On the balanced 180-animation held-out test set, our verification programs achieve 93.3% raw agreement with human judgments, compared with 83.3% for the VLM baseline.

Raw agreement is the fraction of verification-program or VLM decisions that match the final human labels. Approval recall is the fraction of the 90 human-approved animations correctly approved, and rejection recall is the fraction of the 90 human-rejected animations correctly rejected. Our verification programs outperform the VLM baseline on both recall measures, with the larger improvement in identifying human-rejected animations. For animations generated by each of the four held-out model settings, our verification programs also achieve higher raw agreement than the VLM baseline.

We examine the generated animations and verification reports for all 12 cases where our verification programs disagree with human judgments. Across 11 tasks, our verification programs reject nine human-approved animations and approve three human-rejected animations. All three false approvals involve missed requirements for how parts of a graphic move relative to one another. Most false rejections come from timing, coordination, or geometry checks that are stricter than human judgments. Our full paper provides additional evaluation results for our verification programs, the human-review protocol, and the complete disagreement review.

Using MotionGraphicsBench

Explore the 100 tasks and animations at https://motion-graphics-bench.github.io/. We provide the benchmark inputs, our verification programs, recorded model outputs, and human judgments in the Hugging Face repository.

Follow the README's quick start to load a task, render a GSAP program, and inspect which verification checks pass or fail: https://huggingface.co/datasets/figma-ai/motion-graphics-bench#quick-start. The README also explains how to verify SVG frames from other animation systems and load the recorded model outputs.

Cite MotionGraphicsBench

If you use MotionGraphicsBench, please cite our paper:

@article{ma2026motiongraphicsbench,
  title   = {MotionGraphicsBench: Evaluating Motion Graphics Generation with Programmatic Verification},
  author  = {Ma, Jiaju and Jindal, Ashvini Kumar and Wang, Chuhua and Bhakthavatsalam, Sumithra},
  journal = {arXiv preprint arXiv:2611.10592},
  year    = {2026},
  url     = {https://arxiv.org/abs/2611.10592}
}