Each agent's result is the engine-native scene it saved, exported from Unreal Engine so you can turn it in the browser. Beside it, how the -evaluator scored that scene. 4 cases, 29 scenes in 3D. Pick a case, then an agent.
·
Benchmarking coding agents that turn open-ended scene descriptions into engine-native 3D scenes in Unreal Engine. Measuring spatial reasoning, task fulfillment and physical validity.
An open-ended scene description, an asset catalog and the Unreal Engine editor.
Build me a compact, sunlit Copenhagen-style canal district with narrow cobblestone streets and rectangular waterways enclosing dense urban blocks.
Townhouses, canals, bridges and a waterfront promenade — with detailed requirements for materials, layout and street furniture.
Given: an empty level and the content pack’s asset catalog.
Task fulfillment, artifact integrity and static physical validity.
0.2 × Detailed + 0.6 × Overview + 0.2 × Physical
Build me a compact, sunlit Copenhagen-style canal district with narrow cobblestone streets and rectangular waterways enclosing dense urban blocks. Arrange roughly thirty attached four- to six-storey townhouses in continuous street and canal-front rows, using slender façades, repetitive white-framed windows, arched doors and carriage entrances, modest cornices, and steep tiled roofs packed with dormers and brick chimneys. Include occasional exposed party walls between staggered building rows. Finish the façades in weathered muted red, mustard yellow, dusty blue, warm grey, beige, and exposed brick, with red, orange, and charcoal roof tiles. Run a principal waterfront street along a stone quay, crossed by smaller streets and linked across the canals by short, gently arched stone-and-metal bridges, including one opening onto a broad paved promenade with rounded landings. Form the canal edges from stepped pale stone, dark timber retaining walls, and iron railings around reflective rippling water. Add regularly spaced leafy trees, ornate black lamps, stone bollards, benches, European road signs, zebra crossings, puddles, grime, and sparse weeds.
Benchmarking coding agents that turn open-ended scene descriptions into engine-native 3D scenes in Unreal Engine. Measuring spatial reasoning, task fulfillment and physical validity.
An open-ended scene description, an asset catalog and the Unreal Engine editor.
Build me a compact, sunlit Copenhagen-style canal district with narrow cobblestone streets and rectangular waterways enclosing dense urban blocks.
Townhouses, canals, bridges and a waterfront promenade — with detailed requirements for materials, layout and street furniture.
Given: an empty level and the content pack’s asset catalog.
Task fulfillment, artifact integrity and static physical validity.
0.2 × Detailed + 0.6 × Overview + 0.2 × Physical
Build me a compact, sunlit Copenhagen-style canal district with narrow cobblestone streets and rectangular waterways enclosing dense urban blocks. Arrange roughly thirty attached four- to six-storey townhouses in continuous street and canal-front rows, using slender façades, repetitive white-framed windows, arched doors and carriage entrances, modest cornices, and steep tiled roofs packed with dormers and brick chimneys. Include occasional exposed party walls between staggered building rows. Finish the façades in weathered muted red, mustard yellow, dusty blue, warm grey, beige, and exposed brick, with red, orange, and charcoal roof tiles. Run a principal waterfront street along a stone quay, crossed by smaller streets and linked across the canals by short, gently arched stone-and-metal bridges, including one opening onto a broad paved promenade with rounded landings. Form the canal edges from stepped pale stone, dark timber retaining walls, and iron railings around reflective rippling water. Add regularly spaced leafy trees, ornate black lamps, stone bollards, benches, European road signs, zebra crossings, puddles, grime, and sparse weeds.
Repair the corrupted scene using its reference image; preserve all unrelated actors.
Provided reference image ↗A translated and rotated pallet stack, plus an unintended duplicate.
Inspect the saved repair from the same reference camera.
Compare agents and evaluator views ↗Restore each target within 5 cm, 5° and 5%, without unintended edits.
0.8 × Repair F1 + 0.2 × Physical
Repair the current outdoor scene so it matches the provided reference image. Make only the minimum changes needed, and preserve all unrelated actors and properties.
Research Track · 20 representative scenes · 14 coding-agent configurations.
Text-to-scene generation with frontier models is a long-horizon agentic task with substantial inference cost and runtime, which makes repeated full-scale evaluation difficult for many research groups. The Research Track supports affordable, reproducible experimentation; the Full Benchmark provides the more comprehensive evaluation for final model comparison and leaderboard reporting.
Quality against the cost of one case · Research Track, 20 scenes
Swipe to explore all configurations →
Solid: the frontier — nothing cheaper scores higher. One point per configuration, at its evaluated reasoning effort. Cost is the mean USD per case.
Research Track · 20 representative scenes · 14 coding-agent configurations.
Text-to-scene generation with frontier models is a long-horizon agentic task with substantial inference cost and runtime, which makes repeated full-scale evaluation difficult for many research groups. The Research Track supports affordable, reproducible experimentation; the Full Benchmark provides the more comprehensive evaluation for final model comparison and leaderboard reporting.
Quality against the cost of one case · Research Track, 20 scenes
Swipe to explore all configurations →
Solid: the frontier — nothing cheaper scores higher. One point per configuration, at its evaluated reasoning effort. Cost is the mean USD per case.
Research Track, 20 scenes. Case score = 0.2 × Detailed + 0.6 × Overview + 0.2 × Physical. Detailed and Overview are means of the per-case scores.
| # | Agent configuration | Score | Detailed | Overview | Physical | Cost / case |
|---|---|---|---|---|---|---|
| 1 | Claude Fable 5.1 (max)ParetoAnthropic | 0.725 | 0.772 | 0.898 | $18.16 | |
| 2 | GPT-6 Astra (max)OpenAI | 0.707 | 0.733 | 0.712 | $22.47 | |
| 3 | Claude Opus 5 (max)Anthropic | 0.631 | 0.716 | 0.811 | $34.50 | |
| 4 | GPT-5.6 Sol (high)ParetoOpenAI | 0.632 | 0.710 | 0.772 | $4.07 | |
| 5 | Gemini 3.8 Flash (high)ParetoGoogle | 0.637 | 0.664 | 0.655 | $2.49 | |
| 6 | Muse Spark 1.3 (medium)ParetoMeta | 0.598 | 0.661 | 0.651 | $0.10 | |
| 7 | Grok 4.6 (high)xAI | 0.522 | 0.576 | 0.585 | $4.33 | |
| 8 | Qwen 3.8 27B (thinking off)open weightsAlibaba | 0.507 | 0.530 | 0.688 | $0.49 | |
| 9 | Qwen 3.8 27B (thinking on)open weightsAlibaba | 0.505 | 0.460 | 0.694 | $0.33 | |
| 10 | GLM-5.3 Flash (max)open weightsZ.ai | 0.403 | 0.537 | 0.533 | $0.71 | |
| 11 | Gemma 4 31B (thinking on)open weightsParetoGoogle | 0.412 | 0.416 | 0.694 | $0.03 | |
| 12 | Gemma 4 31B (thinking off)open weightsParetoGoogle | 0.438 | 0.370 | 0.725 | $0.02 | |
| 13 | -Inkling (high)Thinking Machines | 0.393 | 0.337 | 0.722 | $0.20 | |
| 14 | DeepSeek V4.1 Flash (high)open weightsDeepSeek | 0.217 | 0.161 | 0.513 | $0.19 |
Click a column to sort. Pareto: on the score-against-cost frontier.
Drag any scene to inspect the saved 3D output, or open a case to explore every agent.
Drag any scene to inspect the saved 3D output, or open a case to explore every agent.
What the evaluation shows across the 14 agents. Every chart below is drawn from the paper's numbers; hover for values.
Errors persist even when the required objects are present: generating the right objects does not ensure that their relationships satisfy the specification.
Errors persist even when the required objects are present: generating the right objects does not ensure that their relationships satisfy the specification.
@misc{ye2026code4scene,
title = {Code4Scene: Benchmarking Coding Agents for Constructing and Editing 3D Scenes},
@@ -56,4 +57,4 @@ svg.ch .dim{opacity:.42}svg.ch g.row:hover{opacity:1}svg.ch .hot{fill:var(--c4s-
SimWorld Home
Part of the SimWorld project.
-