# MotionGraphicsBench: How to Evaluate AI SVG Animation Beyond a Good-Looking Demo

Figma's new benchmark separates motion instructions from visual taste, which changes how you should compare animation models.

By Paras Madan | 2026-11-11


MotionGraphicsBench evaluates whether AI models animate an existing SVG according to instructions. Described in Figma researchers’ November 10 announcement, it uses 100 tasks and explicit checks for movement, appearance and timing. It is useful for choosing an animation workflow because a visually attractive result can still perform the wrong motion.

That distinction matters when the input is a client's finished design. You may need the logo to move around a particular point, pause and return without changing its geometry. Generating a different, impressive animation does not complete that job.

## What MotionGraphicsBench actually measures

The [research announcement](https://huggingface.co/blog/figma-ai/motion-graphics-bench) supplies a static SVG and motion-edit instructions rather than asking models to invent the whole picture. Its reported evaluation covers 32 models, with only four exceeding 50% pass@1 at their highest tested reasoning settings. GPT-6 Astra at maximum reasoning leads at 67.7%.

Those results do not make a universal ranking of video generators, design tools or creative taste. The test isolates a narrower capability: following motion requirements when the visual elements already exist. This is closer to a production edit than a blank-canvas demonstration, but it is still a specific benchmark.

For your own model choice, preserve that separation. First decide whether you need the model to draw assets, animate supplied assets or do both. Otherwise a stronger illustrator can appear to be a stronger animator simply because its output looks more polished.

## Turn one motion brief into separate checks

Consider an illustrative product animation: a notification dot moves from a bell to an inbox, the inbox opens only after the dot arrives, then both return to their original positions. The model should not receive a vague instruction to make it feel premium and then be judged entirely on taste.

Write down the conditions you would reject independently. Does the dot follow the requested route? Does the inbox open after arrival rather than before? Does the ending preserve the original layout? If one condition fails, identify it instead of assigning an overall score that hides the error.

Keep the aesthetic review as well. The timing may be technically correct while the motion feels unpleasant. **Passing a motion test earns a design review; it does not replace one.** This gives engineers and designers a shared language without pretending that all visual quality is reducible to code.

## Read failed checks before increasing the model budget

The [released dataset](https://huggingface.co/datasets/figma-ai/motion-graphics-bench) includes benchmark tasks and model outputs, with fields for passed checks, failed checks, recorded decisions and human review where available. Its verification operates on ordered SVG frames. The provided renderer handles GSAP programs, but the checks evaluate frames rather than requiring one exact source implementation.

That structure suggests a practical debugging order. If the notification dot takes the wrong path, inspect coordinate systems and transforms. If the inbox opens too early, inspect the timing relationship. If every frame has the wrong appearance, investigate rendering or asset preservation before blaming reasoning quality.

An additional model attempt is useful when it addresses a diagnosed problem. Repeatedly asking for a better animation can produce several attractive variants while leaving the same timing defect intact.

## Compare single attempts and retries separately

The [paper](https://arxiv.org/abs/2611.10592) distinguishes pass@1 from pass@k, where multiple independent generations give the model more opportunities to succeed. That distinction belongs in any buying decision because generating several candidates consumes time and still requires a way to identify the valid one.

For an internal comparison, give each model the same assets and requirements. Record the first result, the number of retries, the total generation cost and the time a person spends reviewing or repairing the output. Do not quietly compare one model's first attempt with another model's best of ten.

Suppose one workflow produces a valid animation immediately but another needs five generations and manual cleanup. Even if their best outputs look similar, they have different economics. The cost of choosing and fixing the result belongs in the comparison alongside the API bill.

## Run verification away from sensitive files

The dataset's documentation warns that its Python verification subprocesses are not a security sandbox. Timeouts do not stop code from accessing files or the network, and browser rendering restrictions do not protect the Python verification stage.

If you try the released pipeline, use an isolated environment without production credentials, personal files or access to internal systems. Inspect the code and dependencies first. Keep generated JavaScript and downloaded verification programs away from your normal working directory until you understand what they execute.

This is especially important for teams that treat evaluation scripts as harmless because they only appear to read output. The permissions granted to the process matter more than the label attached to it.

## Use the benchmark to improve the brief

For a small team, the most valuable change may be writing a better acceptance checklist before generating anything. Preserve the asset, specify what moves, define the timing relationship and decide which failures block delivery. Then let a person judge whether the result is worth showing.

A good-looking animation is easy to celebrate. An animation that reliably respects the original design is easier to put into a product, and that is the standard a useful evaluation should help you enforce.

I’m Paras. I work around AI, engineering and distribution.

I help technical teams turn research and product behavior into clear explanations developers and founders can use.

[Book a 30-minute call](https://cal.com/paras-madan/30min).