Benchmarking coding agents that turn text and reference images into engine-native 3D scenes. Measuring spatial reasoning, task fulfillment and precise control of scene state.
An open-ended scene description, an asset catalog and the Unreal Engine editor.
Build me a compact, sunlit Copenhagen-style canal district with narrow cobblestone streets and rectangular waterways enclosing dense urban blocks.
Townhouses, canals, bridges and a waterfront promenade — with detailed requirements for materials, layout and street furniture.
Given: an empty level and the content pack’s asset catalog.
Task fulfillment, artifact integrity and static physical validity.
0.2 × Detailed + 0.6 × Overview + 0.2 × Physical
Build me a compact, sunlit Copenhagen-style canal district with narrow cobblestone streets and rectangular waterways enclosing dense urban blocks. Arrange roughly thirty attached four- to six-storey townhouses in continuous street and canal-front rows, using slender façades, repetitive white-framed windows, arched doors and carriage entrances, modest cornices, and steep tiled roofs packed with dormers and brick chimneys. Include occasional exposed party walls between staggered building rows. Finish the façades in weathered muted red, mustard yellow, dusty blue, warm grey, beige, and exposed brick, with red, orange, and charcoal roof tiles. Run a principal waterfront street along a stone quay, crossed by smaller streets and linked across the canals by short, gently arched stone-and-metal bridges, including one opening onto a broad paved promenade with rounded landings. Form the canal edges from stepped pale stone, dark timber retaining walls, and iron railings around reflective rippling water. Add regularly spaced leafy trees, ornate black lamps, stone bollards, benches, European road signs, zebra crossings, puddles, grime, and sparse weeds.
Repair the corrupted scene using its reference image; preserve all unrelated actors.
Provided reference image ↗A translated and rotated pallet stack, plus an unintended duplicate.
Inspect the saved repair from the same reference camera.
Compare agents and evaluator views ↗Restore each target within 5 cm, 5° and 5%, without unintended edits.
0.8 × Repair F1 + 0.2 × Physical
Repair the current outdoor scene so it matches the provided reference image. Make only the minimum changes needed, and preserve all unrelated actors and properties.
Benchmarking coding agents that turn open-ended scene descriptions into engine-native 3D scenes in Unreal Engine. Measuring spatial reasoning, task fulfillment and physical validity.
An open-ended scene description, an asset catalog and the Unreal Engine editor.
Build me a compact, sunlit Copenhagen-style canal district with narrow cobblestone streets and rectangular waterways enclosing dense urban blocks.
Townhouses, canals, bridges and a waterfront promenade — with detailed requirements for materials, layout and street furniture.
Given: an empty level and the content pack’s asset catalog.
Task fulfillment, artifact integrity and static physical validity.
0.2 × Detailed + 0.6 × Overview + 0.2 × Physical
Build me a compact, sunlit Copenhagen-style canal district with narrow cobblestone streets and rectangular waterways enclosing dense urban blocks. Arrange roughly thirty attached four- to six-storey townhouses in continuous street and canal-front rows, using slender façades, repetitive white-framed windows, arched doors and carriage entrances, modest cornices, and steep tiled roofs packed with dormers and brick chimneys. Include occasional exposed party walls between staggered building rows. Finish the façades in weathered muted red, mustard yellow, dusty blue, warm grey, beige, and exposed brick, with red, orange, and charcoal roof tiles. Run a principal waterfront street along a stone quay, crossed by smaller streets and linked across the canals by short, gently arched stone-and-metal bridges, including one opening onto a broad paved promenade with rounded landings. Form the canal edges from stepped pale stone, dark timber retaining walls, and iron railings around reflective rippling water. Add regularly spaced leafy trees, ornate black lamps, stone bollards, benches, European road signs, zebra crossings, puddles, grime, and sparse weeds.
14 coding-agent configurations · paper results on the original 95 public cases (20 construction + 75 editing). These results predate the 201-case public release.
Quality against the cost of one case · original 95-case public evaluation
Swipe to explore all configurations →
Solid: the frontier — nothing cheaper scores higher. One point per configuration, at its evaluated reasoning effort. Cost is the mean USD per case; overall averages the two task means.
Overall = ½ Text-to-Scene + ½ Image-to-Scene over the 95 public cases; its cost is weighted the same way.
| # | Agent configuration | Overall | Text-to-Scene | Image-to-Scene | Cost / case |
|---|---|---|---|---|---|
| 1 | GPT-6 Astra (max)ParetoOpenAI | 0.724 | 0.515 | $12.90 | |
| 2 | Gemini 3.8 Flash (high)ParetoGoogle | 0.657 | 0.581 | $1.92 | |
| 3 | Claude Fable 5.1 (max)Anthropic | 0.788 | 0.424 | $10.62 | |
| 4 | Claude Opus 5 (max)Anthropic | 0.718 | 0.468 | $19.48 | |
| 5 | GPT-5.6 Sol (high)OpenAI | 0.707 | 0.393 | $2.97 | |
| 6 | Muse Spark 1.3 (medium)ParetoMeta | 0.646 | 0.358 | $0.09 | |
| 7 | GLM-5.3 Flash (max)open weightsZ.ai | 0.509 | 0.320 | $0.40 | |
| 8 | Qwen 3.8 27B (thinking off)open weightsAlibaba | 0.557 | 0.200 | $0.38 | |
| 9 | Grok 4.6 (high)xAI | 0.567 | 0.184 | $2.29 | |
| 10 | Qwen 3.8 27B (thinking on)open weightsAlibaba | 0.516 | 0.226 | $0.30 | |
| 11 | -Inkling (high)Thinking Machines | 0.425 | 0.198 | $0.48 | |
| 12 | Gemma 4 31B (thinking on)open weightsParetoGoogle | 0.471 | 0.126 | $0.03 | |
| 13 | Gemma 4 31B (thinking off)open weightsParetoGoogle | 0.455 | 0.115 | $0.02 | |
| 14 | DeepSeek V4.1 Flash (high)open weightsDeepSeek | 0.243 | 0.210 | $0.12 |
14 coding-agent configurations · paper results on the original 20 public construction cases.
Quality against the cost of one case · original 20-case public evaluation
Swipe to explore all configurations →
Solid: the frontier — nothing cheaper scores higher. One point per configuration, at its evaluated reasoning effort. Cost is the mean USD per case.
20 public cases. Case score = 0.2 × Detailed + 0.6 × Overview + 0.2 × Physical. Detailed and Overview are means of the per-case scores.
| # | Agent configuration | Score | Detailed | Overview | Physical | Cost / case |
|---|---|---|---|---|---|---|
| 1 | Claude Fable 5.1 (max)ParetoAnthropic | 0.725 | 0.772 | 0.898 | $18.16 | |
| 2 | GPT-6 Astra (max)OpenAI | 0.707 | 0.733 | 0.712 | $22.47 | |
| 3 | Claude Opus 5 (max)Anthropic | 0.631 | 0.716 | 0.811 | $34.50 | |
| 4 | GPT-5.6 Sol (high)ParetoOpenAI | 0.632 | 0.710 | 0.772 | $4.07 | |
| 5 | Gemini 3.8 Flash (high)ParetoGoogle | 0.637 | 0.664 | 0.655 | $2.49 | |
| 6 | Muse Spark 1.3 (medium)ParetoMeta | 0.598 | 0.661 | 0.651 | $0.10 | |
| 7 | Grok 4.6 (high)xAI | 0.522 | 0.576 | 0.585 | $4.33 | |
| 8 | Qwen 3.8 27B (thinking off)open weightsAlibaba | 0.507 | 0.530 | 0.688 | $0.49 | |
| 9 | Qwen 3.8 27B (thinking on)open weightsAlibaba | 0.505 | 0.460 | 0.694 | $0.33 | |
| 10 | GLM-5.3 Flash (max)open weightsZ.ai | 0.403 | 0.537 | 0.533 | $0.71 | |
| 11 | Gemma 4 31B (thinking on)open weightsParetoGoogle | 0.412 | 0.416 | 0.694 | $0.03 | |
| 12 | Gemma 4 31B (thinking off)open weightsParetoGoogle | 0.438 | 0.370 | 0.725 | $0.02 | |
| 13 | +Inkling (high)Thinking Machines | 0.393 | 0.337 | 0.722 | $0.20 | |
| 14 | DeepSeek V4.1 Flash (high)open weightsDeepSeek | 0.217 | 0.161 | 0.513 | $0.19 |
Click a column to sort. Pareto: on the score-against-cost frontier.
Drag any scene to inspect the saved 3D output. Compare repairs with their corrupted inputs and ground truth, or open a case to explore every agent.
Drag any scene to inspect the saved 3D output, or open a case to explore every agent.
What the evaluation shows across the 14 agents. Every chart below is drawn from the paper's numbers; hover for values.
Despite nearly identical overall scores, Astra is stronger at construction and Gemini at editing.
Errors persist even when the required objects are present: generating the right objects does not ensure that their relationships satisfy the specification.
Astra performs best on Transform repairs, while Gemini is stronger on Lifecycle and Layout operations.
Across all 14 configurations, 35.8% of complete recoveries still contain unintended changes to the surrounding scene.
Errors persist even when the required objects are present: generating the right objects does not ensure that their relationships satisfy the specification.
@misc{ye2026code4scene,
title = {Code4Scene: Benchmarking Coding Agents for Constructing and Editing 3D Scenes},
@@ -50,12 +48,12 @@ svg.ch .dim{opacity:.42}svg.ch g.row:hover{opacity:1}svg.ch .hot{fill:var(--c4s-
url = {https://arxiv.org/abs/2609.36777},
}