Benchmarking coding agents that turn text and reference images into engine-native 3D scenes. Measuring spatial reasoning, task fulfillment and precise control of scene state.
Text-to-Scene 0.877 on this case · paper Figure 7
An open-ended scene description, an asset catalog and the Unreal Engine editor.
Build me a compact, sunlit Copenhagen-style canal district with narrow cobblestone streets and rectangular waterways enclosing dense urban blocks.
Townhouses, canals, bridges and a waterfront promenade — with detailed requirements for materials, layout and street furniture.
Given: an empty level and the content pack’s asset catalog.
Task fulfillment, artifact integrity and static physical validity.
Build me a compact, sunlit Copenhagen-style canal district with narrow cobblestone streets and rectangular waterways enclosing dense urban blocks. Arrange roughly thirty attached four- to six-storey townhouses in continuous street and canal-front rows, using slender façades, repetitive white-framed windows, arched doors and carriage entrances, modest cornices, and steep tiled roofs packed with dormers and brick chimneys. Include occasional exposed party walls between staggered building rows. Finish the façades in weathered muted red, mustard yellow, dusty blue, warm grey, beige, and exposed brick, with red, orange, and charcoal roof tiles. Run a principal waterfront street along a stone quay, crossed by smaller streets and linked across the canals by short, gently arched stone-and-metal bridges, including one opening onto a broad paved promenade with rounded landings. Form the canal edges from stepped pale stone, dark timber retaining walls, and iron railings around reflective rippling water. Add regularly spaced leafy trees, ornate black lamps, stone bollards, benches, European road signs, zebra crossings, puddles, grime, and sparse weeds.
From an empty level and the content pack’s asset catalog, the agent builds the scene a prompt describes. Many realizations are valid; the evaluator judges requirements rather than matching a reference.
-Given a corrupted scene and reference views of the original, restore every target actor within 5 cm, 5° and 5%, while preserving the surrounding scene.
-Benchmarking coding agents that turn text and reference images into engine-native 3D scenes. Measuring spatial reasoning, task fulfillment and precise control of scene state.
Text-to-Scene 0.877 on this case · paper Figure 7
An open-ended scene description, an asset catalog and the Unreal Engine editor.
Build me a compact, sunlit Copenhagen-style canal district with narrow cobblestone streets and rectangular waterways enclosing dense urban blocks.
Townhouses, canals, bridges and a waterfront promenade — with detailed requirements for materials, layout and street furniture.
Given: an empty level and the content pack’s asset catalog.
Task fulfillment, artifact integrity and static physical validity.
0.2 × Detailed + 0.6 × Overview + 0.2 × Physical
Build me a compact, sunlit Copenhagen-style canal district with narrow cobblestone streets and rectangular waterways enclosing dense urban blocks. Arrange roughly thirty attached four- to six-storey townhouses in continuous street and canal-front rows, using slender façades, repetitive white-framed windows, arched doors and carriage entrances, modest cornices, and steep tiled roofs packed with dormers and brick chimneys. Include occasional exposed party walls between staggered building rows. Finish the façades in weathered muted red, mustard yellow, dusty blue, warm grey, beige, and exposed brick, with red, orange, and charcoal roof tiles. Run a principal waterfront street along a stone quay, crossed by smaller streets and linked across the canals by short, gently arched stone-and-metal bridges, including one opening onto a broad paved promenade with rounded landings. Form the canal edges from stepped pale stone, dark timber retaining walls, and iron railings around reflective rippling water. Add regularly spaced leafy trees, ornate black lamps, stone bollards, benches, European road signs, zebra crossings, puddles, grime, and sparse weeds.
Repair the corrupted scene using its reference image; preserve all unrelated actors.
Provided reference image ���A translated and rotated pallet stack, plus an unintended duplicate.
Inspect the saved repair from the same reference camera.
Compare agents and evaluator views ↗Restore each target within 5 cm, 5° and 5%, without unintended edits.
0.8 × Repair F1 + 0.2 × Physical
Repair the current outdoor scene so it matches the provided reference image. Make only the minimum changes needed, and preserve all unrelated actors and properties.
14 coding-agent configurations · paper results on the original 95 public cases (20 construction + 75 editing). These results predate the 201-case public release.
Quality against the cost of one case · original 95-case public evaluation
Swipe to explore all configurations →
Solid: the frontier — nothing cheaper scores higher. One point per configuration, at its evaluated reasoning effort. Cost is the mean USD per case; overall averages the two task means.
Overall = ½ Text-to-Scene + ½ Image-to-Scene over the 95 public cases; its cost is weighted the same way.
| # | Agent configuration | Overall | Text-to-Scene | Image-to-Scene | Cost / case |
|---|---|---|---|---|---|
| 1 | GPT-6 Astra (max)ParetoOpenAI | 0.724 | 0.515 | $12.90 | |
| 2 | Gemini 3.8 Flash (high)ParetoGoogle | 0.657 | 0.581 | $1.92 | |
| 3 | Claude Fable 5.1 (max)Anthropic | 0.788 | 0.424 | $10.62 | |
| 4 | Claude Opus 5 (max)Anthropic | 0.718 | 0.468 | $19.48 | |
| 5 | GPT-5.6 Sol (high)OpenAI | 0.707 | 0.393 | $2.97 | |
| 6 | Muse Spark 1.3 (medium)ParetoMeta | 0.646 | 0.358 | $0.09 | |
| 7 | GLM-5.3 Flash (max)open weightsZ.ai | 0.509 | 0.320 | $0.40 | |
| 8 | Qwen 3.8 27B (thinking off)open weightsAlibaba | 0.557 | 0.200 | $0.38 | |
| 9 | Grok 4.6 (high)xAI | 0.567 | 0.184 | $2.29 | |
| 10 | Qwen 3.8 27B (thinking on)open weightsAlibaba | 0.516 | 0.226 | $0.30 | |
| 11 | -Inkling (high)Thinking Machines | 0.425 | 0.198 | $0.48 | |
| 12 | Gemma 4 31B (thinking on)open weightsParetoGoogle | 0.471 | 0.126 | $0.03 | |
| 13 | Gemma 4 31B (thinking off)open weightsParetoGoogle | 0.455 | 0.115 | $0.02 | |
| 14 | DeepSeek V4.1 Flash (high)open weightsDeepSeek | 0.243 | 0.210 | $0.12 |
Overall = ½ Text-to-Scene + ½ Image-to-Scene over the 95 public cases; its cost is weighted the same way.
| # | Agent configuration | Overall | Text-to-Scene | Image-to-Scene | Cost / case |
|---|---|---|---|---|---|
| 1 | GPT-6 Astra (max)ParetoOpenAI | 0.724 | 0.515 | $12.90 | |
| 2 | Gemini 3.8 Flash (high)ParetoGoogle | 0.657 | 0.581 | $1.92 | |
| 3 | Claude Fable 5.1 (max)Anthropic | 0.788 | 0.424 | $10.62 | |
| 4 | Claude Opus 5 (max)Anthropic | 0.718 | 0.468 | $19.48 | |
| 5 | GPT-5.6 Sol (high)OpenAI | 0.707 | 0.393 | $2.97 | |
| 6 | Muse Spark 1.3 (medium)ParetoMeta | 0.646 | 0.358 | $0.09 | |
| 7 | GLM-5.3 Flash (max)open weightsZ.ai | 0.509 | 0.320 | $0.40 | |
| 8 | Qwen 3.8 27B (thinking off)open weightsAlibaba | 0.557 | 0.200 | $0.38 | |
| 9 | Grok 4.6 (high)xAI | 0.567 | 0.184 | $2.29 | |
| 10 | Qwen 3.8 27B (thinking on)open weightsAlibaba | 0.516 | 0.226 | $0.30 | |
| 11 | +Inkling (high)Thinking Machines | 0.425 | 0.198 | $0.48 | |
| 12 | Gemma 4 31B (thinking on)open weightsParetoGoogle | 0.471 | 0.126 | $0.03 | |
| 13 | Gemma 4 31B (thinking off)open weightsParetoGoogle | 0.455 | 0.115 | $0.02 | |
| 14 | DeepSeek V4.1 Flash (high)open weightsDeepSeek | 0.243 | 0.210 | $0.12 |
Click a column to sort. Pareto: on the score-against-cost frontier.
The leaderboard, one scene through the evaluator, two scenes compared, and what the benchmark finds, in 90 seconds.
Benchmarking Coding Agents for Constructing and Editing 3D Scenes
Frontier coding agents can now write and execute code that authors 3D environments, but whether they reliably -understand 3D structure and precisely control scene state remains unclear. The generated 3D scene is a persistent, executable artifact: a convincing -render can hide incorrect spatial relations, intersecting objects, or unintended modifications. We introduce Code4Scene, a benchmark of 190 Unreal -Engine cases built from human-assembled scenes that evaluates coding agents on two complementary settings under a shared execution interface. -Construction tests scene-level spatial reasoning from open-ended language specifications, where many realizations are valid; editing tests precise -control of scene state, where the agent must recover the target scene from reference images while preserving everything else.
-Rather than scoring code or rendered views, Code4Scene evaluates the generated engine-native scene for task fulfillment, artifact integrity, and -static physical validity, with edits additionally compared against withheld ground truth. Across 14 coding-agent configurations on the 95-case public -set, construction and editing performance are strongly correlated but not interchangeable (Spearman ρ = 0.78): Claude Fable 5.1 leads construction, -Gemini 3.8 Flash leads editing, and GPT-6 Astra narrowly leads overall. Spatial Composition is the weakest construction category for every agent, -while editing remains imprecise: the best Repair F1 is only 0.527, and 35.8% of edits that fully recover the target still introduce unintended -changes elsewhere in the scene. These results expose a gap between plausible 3D generation and reliable spatial reasoning and state control.
-@misc{ye2026code4scene,
title = {Code4Scene: Benchmarking Coding Agents for Constructing and Editing 3D Scenes},
@@ -84,4 +62,4 @@ changes elsewhere in the scene. These results expose a gap between plausible 3D
SimWorld Home
Part of the SimWorld project.
-