README.md

metadata

title: Real2Sim Explorer
emoji: 🧱
colorFrom: blue
colorTo: indigo
sdk: static
app_file: index.html
pinned: false
short_description: Explore Real2Sim pipelines for robotics and Physical AI

🧱 Real2Sim Explorer

Explore how real environments are reconstructed, structured and converted into simulation-ready worlds for robotics and Physical AI.

The Real2Sim Explorer focuses on one of the most practical applications of World Compilation:

How do we turn a real environment into a simulation that preserves the information needed for learning, planning and control?

A typical pipeline is:

REAL WORLD
   ↓
Capture
   ↓
Reconstruction
   ↓
Object Decomposition
   ↓
Semantics
   ↓
Physical Properties
   ↓
Dynamics
   ↓
Simulation Conversion
   ↓
Domain Randomization
   ↓
Synthetic Data
   ↓
Validation

Why Real2Sim matters

Real-world data is valuable but expensive, slow and difficult to scale.

Simulation is scalable but can diverge from reality.

Real2Sim connects the two.

The goal is not merely to create a visually similar 3D scene. A useful Real2Sim pipeline should preserve enough:

• geometry
• state
• object identity
• physical properties
• dynamics
• actions
• semantics
• uncertainty

to support downstream AI systems.

Real2Sim as World Compilation

REALITY
   ↓
WORLD COMPILER
   ↓
SIMULATION-READY WORLD
   ↓
SYNTHETIC EXPERIENCE
   ↓
WORLD MODEL / POLICY / PLANNER

Included practical examples

• examples/scene_schema.json — a compact simulation-ready world schema
• examples/domain_randomization.json — example randomization ranges
• examples/validation_config.json — example Real2Sim validation protocol

The reality gap

A reconstructed scene can be visually impressive while still being poor for robotics.

The reality gap may come from:

• inaccurate geometry
• wrong friction
• incorrect mass
• weak contact modeling
• missing objects
• unrealistic sensors
• timing differences
• imperfect dynamics

Therefore Real2Sim quality should ultimately be evaluated by downstream transfer, not appearance alone.

Relationship to worldcompiler

worldcompiler covers the broader transformation:

Reality → Structured Representation

Real2Sim is one important branch:

Reality → Structured Representation → Simulation

Cooperation

Cooperation, research and ecosystem partnerships:
agenten@magenta.de

---

Compile the real world into environments AI can safely explore.