UNITY_SETUP.md

# XROS Shell — Unity (Quest 3) Setup Guide

This guide assembles the Unity frontend for the spatial OS shell. The backend
(Python) runs on your computer; the Quest 3 app is a thin client that renders
spatial panels and hosts the terminal.

## Architecture

```
Quest 3 (Unity)                          Your computer (Python)
┌─────────────────────────┐              ┌──────────────────────────┐
│ WorldPanelManager       │◄──spawn/move─┤  xros_server.py         │
│  (spatial panels)       │   /remove     │   ├─ WorldState         │
│ SpatialShellTerminal    │──command─────►│   └─ SpatialShell       │
│  (terminal panel)       │◄──output──────┤      (spawn/move/ls/gen)│
│ XrosClient (WebSocket)  │              │                          │
└─────────────────────────┘              └──────────────────────────┘
```

## 1. Prerequisites

- **Unity 2022.3 LTS** (or newer) with **Android Build Support** module.
- **Meta XR All-in-One SDK** (or Core SDK) — via the Unity Package Manager.
- **Quest 3** in **Developer Mode** (Meta Horizon app → Settings → Developer).
- **Python 3.11+** on your computer, with `websockets` installed
  (`pip install websockets`).

## 2. Create the project

1. New project → **3D (URP)** template.
2. **File → Build Settings → Android → Switch Platform**.
3. **Edit → Project Settings → Player → Other Settings**:
   - Set **Minimum API Level** to 29+ (Quest 3 requires 32).
   - Set **Scripting Backend** to IL2CPP, **Target Architectures** ARM64.
4. **Package Manager** → add **Newtonsoft Json**
   (`com.unity.nuget.newtonsoft-json`) — required by `XrosClient.cs`.
5. **Package Manager** → add **TextMeshPro** (import TMP Essentials when prompted).

## 3. Import Meta XR SDK

1. **Package Manager → Add package by name**:
   `com.meta.xr.sdk.all` (All-in-One) — or `com.meta.xr.sdk.core` for the minimum.
2. **Edit → Project Settings → XR Plug-in Management** → install **OpenXR**,
   enable it for Android, and add **Meta Quest Support**.
3. **Edit → Project Settings → Oculus** → enable **Passthrough** and
   **Hand Tracking** (or use the Meta XR setup wizard).

## 4. Build the scene

1. Delete the default camera. Add **OVRCameraRig** (Meta XR → OVRCameraRig).
2. Add an **OVRPassthroughLayer** to the rig's CenterEyeAnchor for passthrough.
3. Add **OVRHand** components (left + right) to the rig for hand tracking.
4. Create an empty GameObject **"XROS"** and add the three scripts:
   - `XrosClient.cs`
   - `WorldPanelManager.cs`
   - `SpatialShellTerminal.cs`

## 5. Terminal panel UI

1. Under the rig, create a **Canvas** (World Space), size ~0.8 × 0.5 m,
   positioned ~1.5 m in front of the user.
2. Add a **TMP_InputField** (the command line) and a **TMP_Text** (the output
   log) as children.
3. On `SpatialShellTerminal`, assign `client` (the XrosClient), `input`, and
   `output`. Wire the input field's **OnSubmit** to `Submit()`.

## 6. Panel prefab

1. Create a **Quad** (or a small World-Space Canvas) with a **TMP_Text** child
   named "Title".
2. Save it as a prefab and assign it to `WorldPanelManager.panelPrefab`.

## 7. Point the client at your backend

1. Find your computer's **LAN IP** (`ipconfig` / `ifconfig`).
2. On `XrosClient`, set `serverUrl` to `ws://<your-lan-ip>:8765`.
3. Start the backend on your computer:
   `python xros_server.py --host 0.0.0.0 --port 8765`
4. Ensure the Quest and your computer are on the **same Wi-Fi network**.

## 8. Build & run

1. **File → Build Settings → Build And Run** with the Quest connected via USB
   (or use **Quest Link** / **Air Link** for fast iteration).
2. Put on the headset. You should see passthrough, the terminal panel, and
   your hands.
3. Type in the terminal:
   - `help`
   - `spawn panel --title "Hello world"`
   - `gen a clock for the wall`
   - `ls`

A panel should appear in the room for each `spawn`/`gen`, driven by the shared
world model on your computer.

## Troubleshooting

- **No connection**: check the LAN IP, that the backend is running with
  `--host 0.0.0.0`, and that both devices are on the same network.
- **No passthrough**: enable Passthrough in Project Settings → Oculus, and add
  the OVRPassthroughLayer.
- **No hands**: enable Hand Tracking and add OVRHand components.
- **JSON errors**: confirm Newtonsoft Json is installed in the Package Manager.