File size: 3,339 Bytes
7777e5a | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 | # XROS Shell — a spatial OS for the Meta Quest 3
A working prototype of a **spatial operating system shell** combining three
elements, inspired by the DiOS paper (*An Extended Reality Operating System for
the Metaverse*, arXiv:2201.03256):
1. **Spatial computing** — panels anchored in your real room (passthrough),
driven by a shared world model rather than per-app state.
2. **A terminal** — a command line you use *inside* VR, as the primary interface.
3. **Generative UI** — describe what you want in natural language and the
system generates a panel (currently a rule-based stub; an LLM can be wired
in later).
## Architecture
```
Quest 3 (Unity) Your computer (Python)
┌─────────────────────────┐ ┌──────────────────────────┐
│ WorldPanelManager │◄──spawn/move─┤ xros_server.py │
│ (spatial panels) │ /remove │ ├─ WorldState │
│ SpatialShellTerminal │──command─────►│ └─ SpatialShell │
│ (terminal panel) │◄──output──────│ (spawn/move/ls/gen)│
│ XrosClient (WebSocket) │ │ │
└─────────────────────────┘ └──────────────────────────┘
```
The backend is the "OS brain": a shared world-state store plus a spatial shell
(command interpreter). The Quest app is a thin client that renders the world
and hosts the terminal.
## Backend (Python)
| File | Role |
|---|---|
| `xros_world.py` | World-state store: scene graph + object registry + event bus |
| `xros_shell.py` | Spatial shell: command interpreter (`spawn`, `move`, `ls`, `gen`, `clear`) |
| `xros_server.py` | WebSocket server exposing the world-state + shell API |
| `xros_client.py` | Python client library + demo |
| `test_xros.py` | End-to-end tests (unit + integration) |
Run it:
```bash
pip install -r requirements.txt
python xros_server.py --host 0.0.0.0 --port 8765 # serve on your LAN
python test_xros.py # verify it works
```
Shell commands: `help`, `ls`, `spawn <type> [x y z] [--title T] [--color C]`,
`move <id> x y z`, `rm <id>`, `clear`, `gen <description>`.
## Frontend (Unity, Quest 3)
| File | Role |
|---|---|
| `XrosClient.cs` | WebSocket client (Newtonsoft.Json, no other packages) |
| `SpatialShellTerminal.cs` | Terminal panel: command input + output log |
| `WorldPanelManager.cs` | Renders world events as spatial panels |
| `UNITY_SETUP.md` | Step-by-step Unity + Meta XR SDK setup and build guide |
See **UNITY_SETUP.md** for the full assembly instructions (project setup, Meta
XR SDK, passthrough, hand tracking, and building to the Quest 3).
## Roadmap
- **Phase 1 (done):** shared world-state service + spatial shell + terminal.
- **Phase 2:** CV as a system service (shared scene understanding).
- **Phase 3:** agentic interface layer (AIOS-style scheduling/context).
- **Phase 4:** spatial developer experience (Code Cubes / structured editing).
- **Phase 5:** privacy/security at the OS layer (capability-based access).
|