File size: 6,387 Bytes
0395907 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 | ---
language:
- en
- zh
pipeline_tag: image-to-3d
tags:
- 3d-reconstruction
- real-to-sim
- embodied-ai
- robotics
- visual-inertial-slam
- agentic-ai
- blender
- glb
---
# AWSM: Agentic World Simulation and Mapping
*Pronounced “awesome”. From real spaces to worlds phygital agents can use (phygital = physical + digital).*
**📖 Read the full interactive article: [phygital-ai.github.io/agentic-world-simulation-and-mapping](https://phygital-ai.github.io/agentic-world-simulation-and-mapping/)**
(中文版: [zh.html](https://phygital-ai.github.io/agentic-world-simulation-and-mapping/zh.html))
AWSM uses large-model agents to turn visual observations into **editable 3D scenes**, grounded by camera-pose and depth estimates, including IMU-informed visual–inertial constraints that tie geometry to physical scale. This repository holds the four frozen reconstructions (Blender `.blend` + `.glb`) from the study, their manifests, and the full metric tables.
| Headline result (M1 → M4) | Before | After | Relative change |
|---|---|---|---|
| Bidirectional surface error | 0.376 m | 0.072 m | ≈ −81% |
| Model-depth AbsRel (180 views) | 16.44% | 7.60% | ≈ −54% |
These are complete-pipeline comparisons in one controlled NVIDIA Isaac Sim scene (World Lobby), not an isolated IMU ablation.
## A scene can look plausible and still be spatially wrong
A reconstruction agent can assemble a convincing room while getting a corner, a distance, or a passage wrong. For an embodied agent those errors change where a destination lies and which route a body can follow. AWSM asks how geometric evidence (pose, depth, metric constraints) can constrain agentic modeling so the result is a usable spatial reference, not just a good-looking render.
## One agentic workflow, four reconstruction routes
The reconstruction agent runs a shared **observe → build → verify → freeze → evaluate** loop: it inspects the evidence, writes Blender Python to build named objects, materials, cameras and collision proxies, renders review views, revises, and freezes the scene before any ground truth is shown.
| Route | Inputs | Geometry source |
|---|---|---|
| **M1** | 180 sampled RGB frames | Visual inference only, no metric scale |
| **M2** | Full RGB video | ViPE RGB-only poses → pose-conditioned Depth Anything 3 |
| **M3** | Video + IMU + camera–IMU calibration | ORB-SLAM3 monocular-inertial poses → pose-conditioned DA3 |
| **M4** | Video + GT camera poses (diagnostic) | GT poses → pose-conditioned DA3 |
## Results
### 1. Better pose does not mechanically imply better native depth
| Route | ATE (m) | Rotation (°) | Native depth AbsRel | Model depth AbsRel |
|---|---|---|---|---|
| M1 | — | — | — | 16.44% † |
| M2 | 0.167 | 0.31 | 19.06% | 9.29% |
| M3 | 0.121 | 0.19 | 20.38% | 8.37% |
| M4 | GT | GT | 18.87% | 7.60% |
† M1 uses a GT-assisted Sim(3) alignment and is diagnostic only.
M3 has a better trajectory than M2, yet slightly worse native depth. After object-level modeling the order flips: M3's final scene is closer to GT.
### 2. Geometric evidence turns a plausible layout into a more faithful space
| Route | Model → GT (m) | GT → model (m) | **Bidirectional mean (m)** |
|---|---|---|---|
| M1 | 0.345 | 0.407 | **0.376** |
| M2 | 0.132 | 0.103 | **0.118** |
| M3 | 0.069 | 0.095 | **0.082** |
| M4 | 0.069 | 0.074 | **0.072** |
### 3. Inspect the scenes, not just the scores
| M4 frozen model (view 72) | Input GT RGB (view 72) |
|---|---|
|  |  |
The interactive article lets you orbit and zoom each frozen scene side by side with GT.
## Beyond the lobby: a phone-captured Office Café
A qualitative real-capture case study: an object-centric scene, a solved camera trajectory, and physics replays in one artifact. Every object is interactable; the shake replay shows the physical response under strong shaking.
| Procedural model | Strong-shake replay |
|---|---|
|  |  |
## Map-based embodied demo
In NVIDIA Isaac Sim, the reception desk is marked on the reconstructed map, routes are set, and a drone, humanoid, quadruped and wheeled robot navigate there and line up. Routes are predefined and localization uses simulator pose. The demo illustrates downstream use, not a navigation benchmark.
| Planned routes | Navigation |
|---|---|
|  |  |
▶ Watch the demo video in the [article](https://phygital-ai.github.io/agentic-world-simulation-and-mapping/).
## Files
```
scenes/M1..M4/scene.blend frozen Blender scene (editable objects, materials, cameras)
scenes/M1..M4/scene.glb the same scene as GLB
scenes/M1..M4/manifest.json route manifest with hashes and registration
tables_1_7.json all seven frozen metric tables and evaluation protocols
awsm.bib citation
```
Download one route:
```bash
hf download AWSM01/AWSM --include "scenes/M3/*" --local-dir awsm
```
## Limitations
- One synthetic scene and one engineering run per route; no claim of general superiority or statistical significance.
- The four routes are complete systems, not a single-variable ablation; M2 vs. M3 cannot be attributed to IMU alone.
- M4 uses GT camera poses and is diagnostic, not deployable.
- The 100-view depth and appearance sets contain modeling RGB inputs; they are not a held-out novel-view benchmark.
- The robot demo uses predefined routes and simulator pose.
## Citation
```bibtex
@misc{awsm_2026,
title = {{AWSM}: Agentic World Simulation and Mapping},
author = {{Phygital AI}},
year = {2026},
howpublished = {Interactive research article},
url = {https://phygital-ai.github.io/agentic-world-simulation-and-mapping/},
note = {Reconstruction benchmarks, editable scene assets, and map-based embodied demonstration}
}
```
|