-
-
+
+
+Code4Scene: Benchmarking Coding Agents for Constructing and Editing 3D Scenes
+
+
+
+
+
+
+
+
Code4Scene: Benchmarking Coding Agents for Constructing and Editing 3D Scenes
+
190 Unreal Engine cases built from human-assembled scenes. Coding agents write and run code that builds a scene from text, or
+repairs one from reference images, and Code4Scene scores the engine-native scene they save, not their code or a rendered view.
The leaderboard, one scene through the evaluator, two scenes compared, and what the benchmark finds, in 90 seconds.
+
+
+
Overview
Score the scene, not the code or a render
+
Coding agents can now operate 3D engines: they write and run code, inspect the result and revise the scene. Code4Scene is a
+benchmark of 190 Unreal Engine cases built from human-assembled scenes that evaluates them on two complementary settings under a shared execution
+interface. Construction starts from an empty level and an open-ended text specification. Editing gives the agent a corrupted copy of a
+scene and reference images of the original, and asks it to restore the intended state while leaving everything else untouched.
+
Rather than scoring code or rendered views, Code4Scene evaluates the engine-native scene the agent saved, on task fulfilment, artifact integrity
+and static physical validity, and compares edits against withheld ground truth. Across 14 agent configurations, construction and editing rank the
+agents differently, spatial relations remain the weakest requirement family, and a third of complete repairs still change the surrounding scene.
+
1 · InputPrompt or referenceAn open-ended scene description, or a corrupted scene with reference images of the original.
2 · AgentCoding agentA frontier or open-weight model in its own CLI harness, with the Unreal Engine editor as a tool.
3 · CodeCode in the engineIt writes and runs Python against the editor: spawns, moves and edits actors, inspects, revises.
4 · SceneEngine-native sceneThe saved level (.umap): every actor, transform and material, as the engine holds it.
5 · EvaluatorScored in the engineRequirements located and judged, overview views judged, physics measured, edits diffed against the withheld ground truth.
+
+
Setting 1 · construction
Text-to-Scene
+
From an empty level and the content pack's asset catalog, the agent builds the scene a prompt describes. Many realizations are valid, so the
+evaluator judges requirements rather than matching a reference. 20 public cases.
The agent gets a corrupted copy of a human-assembled scene and reference views of the original. It must restore every target actor (within
+5 cm, 5° and 5%) and change nothing else. 75 public cases: 25 indoor, 50 outdoor.
+
case score = 0.8 × Repair F1 + 0.2 × Physical Safety
+
+
Leaderboard
14 coding agents on the public set
+
Overall = ½ Text-to-Scene + ½ Image-to-Scene, over the 95 public cases. Scores are from the paper's Table 2.
+
Score against cost
+
Each agent's score against what one case cost it. The line is the frontier: no agent to its left scores higher.
+
+
+
Overall
Gemini 3.8 Flash (high)0.619 · $1.59
Muse Spark 1.3 (medium)0.502 · $0.08
Gemma 4 31B (thinking on)0.298 · $0.03
Gemma 4 31B (thinking off)0.285 · $0.02
Text-to-Scene
Claude Fable 5.1 (max)0.788 · $18.16
GPT-5.6 Sol (high)0.707 · $4.07
Gemini 3.8 Flash (high)0.657 · $2.49
Muse Spark 1.3 (medium)0.646 · $0.10
Gemma 4 31B (thinking on)0.471 · $0.03
Gemma 4 31B (thinking off)0.455 · $0.02
Image-to-Scene
Gemini 3.8 Flash (high)0.581 · $1.35
Muse Spark 1.3 (medium)0.358 · $0.07
DeepSeek V4.1 Flash (high)0.210 · $0.05
Gemma 4 31B (thinking on)0.126 · $0.03
Gemma 4 31B (thinking off)0.115 · $0.02
+
Cost is the mean reported cost of one run, estimated from its token usage at the provider's list price (paper Figure 1b; provisional).
+Some runs did not report usage; hover a point for how many of the 95 cases are priced.
+
Text-to-Scene · construction
+
20 public cases. Detailed and Overview are the means of the per-case scores; Score and Physical are from Table 2. Click a column to sort.
Text-to-Scene (left) and Image-to-Scene (right) score per agent, in overall order (paper Table 2). Astra and Gemini tie at 0.619 overall.
Takeaway 2
Spatial Composition remains the weakest requirement family for all 14 agents.
Errors persist even when the required objects are present: generating the right objects does not ensure that their relationships satisfy the specification.
Each row is one agent's four family scores; the orange dot, Spatial Composition, is the lowest in every row (paper Figure 14b).
The objects are there; the relation is not
✓ slab roof 0.95✓ stone coping 0.82✗ slab roof enclosed by the coping 0.78
The judge's own frame of GPT-5.6 Sol on Egyptian Temple. It finds the slab roof and the stone coping, but not the roof enclosed by the coping: the coping is separate low wall segments with visible gaps (ringed).
Mismatch rate when the required objects are present
Across the benchmark (paper Figure 6a).
Takeaway 3
Editing ability varies substantially across repair types.
Astra performs best on Transform repairs, while Gemini is stronger on Lifecycle and Layout operations.
Mean Image-to-Scene score by repair type for four agents; the outlined cell is the best of the four in its row (paper Figure 6b).
Takeaway 4
Recovering the target does not guarantee precise editing.
Across all 14 configurations, 35.8% of complete recoveries still contain unintended changes to the surrounding scene.
@article{code4scene2026,
+ title = {Code4Scene: Benchmarking Coding Agents for Constructing and Editing 3D Scenes},
+ author = {Ye, Xiaokang and Mantri, Siddhant Hitesh and Chen, Zimeng and Zhang, Edward and Zheng, Zhaoxu and Li, Yuanheng and Chen, Yizhao and Huang, Tianyang and Qin, Lianhui},
+ year = {2026}
+}
