Benchmarking coding agents that turn text and reference images into engine-native 3D scenes. Measuring spatial reasoning, task fulfillment and precise control of scene state.
Text-to-Scene 0.877 on this case · paper Figure 7
An open-ended scene description, an asset catalog and the Unreal Engine editor.
Build me a compact, sunlit Copenhagen-style canal district with narrow cobblestone streets and rectangular waterways enclosing dense urban blocks.
Townhouses, canals, bridges and a waterfront promenade — with detailed requirements for materials, layout and street furniture.
Given: an empty level and the content pack’s asset catalog.
Open the saved scene in 3D ↗Task fulfillment, artifact integrity and static physical validity.
Build me a compact, sunlit Copenhagen-style canal district with narrow cobblestone streets and rectangular waterways enclosing dense urban blocks. Arrange roughly thirty attached four- to six-storey townhouses in continuous street and canal-front rows, using slender façades, repetitive white-framed windows, arched doors and carriage entrances, modest cornices, and steep tiled roofs packed with dormers and brick chimneys. Include occasional exposed party walls between staggered building rows. Finish the façades in weathered muted red, mustard yellow, dusty blue, warm grey, beige, and exposed brick, with red, orange, and charcoal roof tiles. Run a principal waterfront street along a stone quay, crossed by smaller streets and linked across the canals by short, gently arched stone-and-metal bridges, including one opening onto a broad paved promenade with rounded landings. Form the canal edges from stepped pale stone, dark timber retaining walls, and iron railings around reflective rippling water. Add regularly spaced leafy trees, ornate black lamps, stone bollards, benches, European road signs, zebra crossings, puddles, grime, and sparse weeds.
Benchmarking coding agents that turn text and reference images into engine-native 3D scenes. Measuring spatial reasoning, task fulfillment and precise control of scene state.
Text-to-Scene 0.877 on this case · paper Figure 7
An open-ended scene description, an asset catalog and the Unreal Engine editor.
Build me a compact, sunlit Copenhagen-style canal district with narrow cobblestone streets and rectangular waterways enclosing dense urban blocks.
Townhouses, canals, bridges and a waterfront promenade — with detailed requirements for materials, layout and street furniture.
Given: an empty level and the content pack’s asset catalog.
Task fulfillment, artifact integrity and static physical validity.
Build me a compact, sunlit Copenhagen-style canal district with narrow cobblestone streets and rectangular waterways enclosing dense urban blocks. Arrange roughly thirty attached four- to six-storey townhouses in continuous street and canal-front rows, using slender façades, repetitive white-framed windows, arched doors and carriage entrances, modest cornices, and steep tiled roofs packed with dormers and brick chimneys. Include occasional exposed party walls between staggered building rows. Finish the façades in weathered muted red, mustard yellow, dusty blue, warm grey, beige, and exposed brick, with red, orange, and charcoal roof tiles. Run a principal waterfront street along a stone quay, crossed by smaller streets and linked across the canals by short, gently arched stone-and-metal bridges, including one opening onto a broad paved promenade with rounded landings. Form the canal edges from stepped pale stone, dark timber retaining walls, and iron railings around reflective rippling water. Add regularly spaced leafy trees, ornate black lamps, stone bollards, benches, European road signs, zebra crossings, puddles, grime, and sparse weeds.
From an empty level and the content pack’s asset catalog, the agent builds the scene a prompt describes. Many realizations are valid; the evaluator judges requirements rather than matching a reference.
14 coding-agent configurations · paper results on the original 95 public cases (20 construction + 75 editing). These results predate the 201-case public release.
Cost is the mean reported cost of one run, estimated from its token usage at the provider's list price (paper Figure 1b; provisional).
Overall = ½ Text-to-Scene + ½ Image-to-Scene over the 95 public cases; its cost is weighted the same way.
| # | Agent configuration | Overall | Text-to-Scene | Image-to-Scene | Cost / case |
|---|---|---|---|---|---|
| 1 | GPT-6 Astra (max)OpenAI | 0.724 | 0.515 | $12.90 | |
| 2 | Gemini 3.8 Flash (high)ParetoGoogle | 0.657 | 0.581 | $1.92 | |
| 3 | Claude Fable 5.1 (max)Anthropic | 0.788 | 0.424 | $10.62 | |
| 4 | Claude Opus 5 (max)Anthropic | 0.718 | 0.468 | $19.48 | |
| 5 | GPT-5.6 Sol (high)OpenAI | 0.707 | 0.393 | $2.97 | |
| 6 | Muse Spark 1.3 (medium)ParetoMeta | 0.646 | 0.358 | $0.09 | |
| 7 | GLM-5.3 Flash (max)open weightsZ.ai | 0.509 | 0.320 | $0.40 | |
| 8 | Qwen 3.8 27B (thinking off)open weightsAlibaba | 0.557 | 0.200 | $0.38 | |
| 9 | Grok 4.6 (high)xAI | 0.567 | 0.184 | $2.29 | |
| 10 | Qwen 3.8 27B (thinking on)open weightsAlibaba | 0.516 | 0.226 | $0.30 | |
| 11 | -Inkling (high)Thinking Machines | 0.425 | 0.198 | $0.48 | |
| 12 | Gemma 4 31B (thinking on)open weightsParetoGoogle | 0.471 | 0.126 | $0.03 | |
| 13 | Gemma 4 31B (thinking off)open weightsParetoGoogle | 0.455 | 0.115 | $0.02 | |
| 14 | DeepSeek V4.1 Flash (high)open weightsDeepSeek | 0.243 | 0.210 | $0.12 |
Quality against the cost of one case · original 95-case public evaluation
Swipe to explore all configurations →
Solid: the frontier — nothing cheaper scores higher. One point per configuration, at its evaluated reasoning effort. Cost is the mean USD per case; overall averages the two task means.
Overall = ½ Text-to-Scene + ½ Image-to-Scene over the 95 public cases; its cost is weighted the same way.
| # | Agent configuration | Overall | Text-to-Scene | Image-to-Scene | Cost / case |
|---|---|---|---|---|---|
| 1 | GPT-6 Astra (max)ParetoOpenAI | 0.724 | 0.515 | $12.90 | |
| 2 | Gemini 3.8 Flash (high)ParetoGoogle | 0.657 | 0.581 | $1.92 | |
| 3 | Claude Fable 5.1 (max)Anthropic | 0.788 | 0.424 | $10.62 | |
| 4 | Claude Opus 5 (max)Anthropic | 0.718 | 0.468 | $19.48 | |
| 5 | GPT-5.6 Sol (high)OpenAI | 0.707 | 0.393 | $2.97 | |
| 6 | Muse Spark 1.3 (medium)ParetoMeta | 0.646 | 0.358 | $0.09 | |
| 7 | GLM-5.3 Flash (max)open weightsZ.ai | 0.509 | 0.320 | $0.40 | |
| 8 | Qwen 3.8 27B (thinking off)open weightsAlibaba | 0.557 | 0.200 | $0.38 | |
| 9 | Grok 4.6 (high)xAI | 0.567 | 0.184 | $2.29 | |
| 10 | Qwen 3.8 27B (thinking on)open weightsAlibaba | 0.516 | 0.226 | $0.30 | |
| 11 | +Inkling (high)Thinking Machines | 0.425 | 0.198 | $0.48 | |
| 12 | Gemma 4 31B (thinking on)open weightsParetoGoogle | 0.471 | 0.126 | $0.03 | |
| 13 | Gemma 4 31B (thinking off)open weightsParetoGoogle | 0.455 | 0.115 | $0.02 | |
| 14 | DeepSeek V4.1 Flash (high)open weightsDeepSeek | 0.243 | 0.210 | $0.12 |
Click a column to sort. Pareto: on the score-against-cost frontier.
Choose a scene. Compare the coding agents. Open an output to inspect the saved scene in 3D, alongside its evaluator scores.
Drag any scene to inspect the saved 3D output. Compare repairs with their corrupted inputs and ground truth, or open a case to explore every agent.
What the evaluation shows across the 14 agents. Every chart below is drawn from the paper's numbers; hover for values.
Despite nearly identical overall scores, Astra is stronger at construction and Gemini at editing.
Errors persist even when the required objects are present: generating the right objects does not ensure that their relationships satisfy the specification.
Astra performs best on Transform repairs, while Gemini is stronger on Lifecycle and Layout operations.
Across all 14 configurations, 35.8% of complete recoveries still contain unintended changes to the surrounding scene.