| # XROS Shell — a spatial OS for the Meta Quest 3 |
|
|
| A working prototype of a **spatial operating system shell** combining three |
| elements, inspired by the DiOS paper (*An Extended Reality Operating System for |
| the Metaverse*, arXiv:2201.03256): |
|
|
| 1. **Spatial computing** — panels anchored in your real room (passthrough), |
| driven by a shared world model rather than per-app state. |
| 2. **A terminal** — a command line you use *inside* VR, as the primary interface. |
| 3. **Generative UI** — describe what you want in natural language and the |
| system generates a panel (currently a rule-based stub; an LLM can be wired |
| in later). |
|
|
| ## Architecture |
|
|
| ``` |
| Quest 3 (Unity) Your computer (Python) |
| ┌─────────────────────────┐ ┌──────────────────────────┐ |
| │ WorldPanelManager │◄──spawn/move─┤ xros_server.py │ |
| │ (spatial panels) │ /remove │ ├─ WorldState │ |
| │ SpatialShellTerminal │──command─────►│ └─ SpatialShell │ |
| │ (terminal panel) │◄──output──────│ (spawn/move/ls/gen)│ |
| │ XrosClient (WebSocket) │ │ │ |
| └─────────────────────────┘ └──────────────────────────┘ |
| ``` |
|
|
| The backend is the "OS brain": a shared world-state store plus a spatial shell |
| (command interpreter). The Quest app is a thin client that renders the world |
| and hosts the terminal. |
|
|
| ## Backend (Python) |
|
|
| | File | Role | |
| |---|---| |
| | `xros_world.py` | World-state store: scene graph + object registry + event bus | |
| | `xros_shell.py` | Spatial shell: command interpreter (`spawn`, `move`, `ls`, `gen`, `clear`) | |
| | `xros_server.py` | WebSocket server exposing the world-state + shell API | |
| | `xros_client.py` | Python client library + demo | |
| | `test_xros.py` | End-to-end tests (unit + integration) | |
|
|
| Run it: |
|
|
| ```bash |
| pip install -r requirements.txt |
| python xros_server.py --host 0.0.0.0 --port 8765 # serve on your LAN |
| python test_xros.py # verify it works |
| ``` |
|
|
| Shell commands: `help`, `ls`, `spawn <type> [x y z] [--title T] [--color C]`, |
| `move <id> x y z`, `rm <id>`, `clear`, `gen <description>`. |
|
|
| ## Frontend (Unity, Quest 3) |
|
|
| | File | Role | |
| |---|---| |
| | `XrosClient.cs` | WebSocket client (Newtonsoft.Json, no other packages) | |
| | `SpatialShellTerminal.cs` | Terminal panel: command input + output log | |
| | `WorldPanelManager.cs` | Renders world events as spatial panels | |
| | `UNITY_SETUP.md` | Step-by-step Unity + Meta XR SDK setup and build guide | |
|
|
| See **UNITY_SETUP.md** for the full assembly instructions (project setup, Meta |
| XR SDK, passthrough, hand tracking, and building to the Quest 3). |
| |
| ## Roadmap |
| |
| - **Phase 1 (done):** shared world-state service + spatial shell + terminal. |
| - **Phase 2:** CV as a system service (shared scene understanding). |
| - **Phase 3:** agentic interface layer (AIOS-style scheduling/context). |
| - **Phase 4:** spatial developer experience (Code Cubes / structured editing). |
| - **Phase 5:** privacy/security at the OS layer (capability-based access). |
| |
