UNITY_SETUP.md

XROS Shell — Unity (Quest 3) Setup Guide

This guide assembles the Unity frontend for the spatial OS shell. The backend (Python) runs on your computer; the Quest 3 app is a thin client that renders spatial panels and hosts the terminal.

Architecture

Quest 3 (Unity)                          Your computer (Python)
┌─────────────────────────┐              ┌──────────────────────────┐
│ WorldPanelManager       │◄──spawn/move─┤  xros_server.py         │
│  (spatial panels)       │   /remove     │   ├─ WorldState         │
│ SpatialShellTerminal    │──command─────►│   └─ SpatialShell       │
│  (terminal panel)       │◄──output──────┤      (spawn/move/ls/gen)│
│ XrosClient (WebSocket)  │              │                          │
└─────────────────────────┘              └──────────────────────────┘

1. Prerequisites

• Unity 2022.3 LTS (or newer) with Android Build Support module.
• Meta XR All-in-One SDK (or Core SDK) — via the Unity Package Manager.
• Quest 3 in Developer Mode (Meta Horizon app → Settings → Developer).
• Python 3.11+ on your computer, with websockets installed (pip install websockets).

2. Create the project

1. New project → 3D (URP) template.
2. File → Build Settings → Android → Switch Platform.
3. Edit → Project Settings → Player → Other Settings:
   • Set Minimum API Level to 29+ (Quest 3 requires 32).
   • Set Scripting Backend to IL2CPP, Target Architectures ARM64.
4. Package Manager → add Newtonsoft Json (com.unity.nuget.newtonsoft-json) — required by XrosClient.cs.
5. Package Manager → add TextMeshPro (import TMP Essentials when prompted).

3. Import Meta XR SDK

1. Package Manager → Add package by name: com.meta.xr.sdk.all (All-in-One) — or com.meta.xr.sdk.core for the minimum.
2. Edit → Project Settings → XR Plug-in Management → install OpenXR, enable it for Android, and add Meta Quest Support.
3. Edit → Project Settings → Oculus → enable Passthrough and Hand Tracking (or use the Meta XR setup wizard).

4. Build the scene

1. Delete the default camera. Add OVRCameraRig (Meta XR → OVRCameraRig).
2. Add an OVRPassthroughLayer to the rig's CenterEyeAnchor for passthrough.
3. Add OVRHand components (left + right) to the rig for hand tracking.
4. Create an empty GameObject "XROS" and add the three scripts:
   • XrosClient.cs
   • WorldPanelManager.cs
   • SpatialShellTerminal.cs

5. Terminal panel UI

1. Under the rig, create a Canvas (World Space), size ~0.8 × 0.5 m, positioned ~1.5 m in front of the user.
2. Add a TMP_InputField (the command line) and a TMP_Text (the output log) as children.
3. On SpatialShellTerminal, assign client (the XrosClient), input, and output. Wire the input field's OnSubmit to Submit().

6. Panel prefab

1. Create a Quad (or a small World-Space Canvas) with a TMP_Text child named "Title".
2. Save it as a prefab and assign it to WorldPanelManager.panelPrefab.

7. Point the client at your backend

1. Find your computer's LAN IP (ipconfig / ifconfig).
2. On XrosClient, set serverUrl to ws://<your-lan-ip>:8765.
3. Start the backend on your computer: python xros_server.py --host 0.0.0.0 --port 8765
4. Ensure the Quest and your computer are on the same Wi-Fi network.

8. Build & run

1. File → Build Settings → Build And Run with the Quest connected via USB (or use Quest Link / Air Link for fast iteration).
2. Put on the headset. You should see passthrough, the terminal panel, and your hands.
3. Type in the terminal:
   • help
   • spawn panel --title "Hello world"
   • gen a clock for the wall
   • ls

A panel should appear in the room for each spawn/gen, driven by the shared world model on your computer.

Troubleshooting

• No connection: check the LAN IP, that the backend is running with --host 0.0.0.0, and that both devices are on the same network.
• No passthrough: enable Passthrough in Project Settings → Oculus, and add the OVRPassthroughLayer.
• No hands: enable Hand Tracking and add OVRHand components.
• JSON errors: confirm Newtonsoft Json is installed in the Package Manager.