README with repository metadata

README.md

@@ -0,0 +1,183 @@
+---
+license: mit
+tags:
+- unity
+- upm
+- unity-package
+- xr
+- virtual-reality
+- llama.cpp
+- offline
+- voice-control
+- agent
+---
+
+# Maverick XR Agent for Unity
+
+The Unity package for [Maverick 4B](https://huggingface.co/ErenAta00/Maverick-4B-Unity-XR-Agent-GGUF).
+
+Let people control your Unity scene in plain English, typed or spoken, with a small language model that runs **on
+the same machine, offline**. "Put the red mug on the table", "turn on the lamp", "where is the stapler?": the model
+reads the objects you marked as interactive, calls the right tool, and Unity executes it. When the words fit two
+objects it asks which one; when an object is in another room it searches there; when a request is impossible it says
+so instead of acting.
+
+> **Status: release candidate (1.0.0-rc.2).** Tested on Windows 11 x64, Unity 2022.3 LTS, NVIDIA RTX
+> 3050 Ti Laptop (4 GB). See [Limitations](#limitations) before building on it.
+
+- **11 tools:** grab, release, place, move_by, rotate, set_state, press, highlight, teleport, find_objects,
+  ask_clarification.
+- **Understands** synonyms and general words ("the fruit"), colours, spatial references ("the one next to the chair",
+  "the second from the left"), "this / that" with the user's gaze, the object in hand ("put it on the shelf"), filler
+  words and speech-recognition noise.
+- **Safe by default:**
+  - D5b: a call to a tool or a value that does not exist is turned into a refusal before it reaches your scene; an
+    unknown object id is generated again under a grammar.
+  - Substitute guard: when the user names an object that is only in another room and the model tries to act on a
+    different object here, the call is not executed and the model is told where the named object is.
+- **Private and free to run:** llama.cpp on `127.0.0.1`; no account, no cloud call, no per-request cost.
+
+## Requirements
+| | |
+|---|---|
+| Unity | 2022.3 LTS (tested 2022.3.62f3) |
+| OS | Windows x64 (tested). llama.cpp also runs on Linux and macOS; `LocalModelServer` is written for them but untested |
+| GPU | ~3.1 GB free VRAM; CPU fallback works but is slow |
+| Disk | 2.5 GB for the model, plus ~100 MB for llama.cpp |
+
+## Install
+1. **Package:** Window → Package Manager → **+** → *Add package from git URL…* →
+   `https://huggingface.co/ErenAta00/Maverick-Unity.git` (or *Add package from disk…* → this folder's `package.json`).
+2. **Model files:** Tools → Scene Agent → **Open Model Folder** (creates `Assets/StreamingAssets/SceneAgent/`) and put
+   there:
+   - `llama-server.exe` and the DLLs next to it, from a llama.cpp release for Windows (CUDA 12 build for NVIDIA GPUs,
+     Vulkan build for other GPUs, CPU build otherwise). Copy the whole folder's contents, not only the .exe.
+   - The model: `Maverick-4B-Unity-XR-Agent-Q4_K_M.gguf` (2.5 GB) from
+     https://huggingface.co/ErenAta00/Maverick-4B-Unity-XR-Agent-GGUF.
+     The server takes the first `.gguf` it finds there, or set `LocalModelServer.modelPath`.
+
+StreamingAssets is copied into a build as it is, so a built game runs without installing anything.
+
+## Quick start (5 minutes)
+1. **Tools → Scene Agent → Set Up Scene.** Adds a *Scene Agent* object with everything wired: `SceneWorld` (your
+   objects, as the model sees them), `SceneAgent` (the conversation with the model), `LocalModelServer` (starts
+   llama.cpp), `SceneAgentSession` and `SceneAgentChatBox` (an on-screen box to try it).
+2. **Build or open your scene.** Name objects with plain words: *Red Mug*, *Table*, *Floor Lamp*, *Book*.
+3. **Select the objects** the agent may use → **Tools → Scene Agent → Make Selected Interactive.** Each gets a
+   `SceneItem`: an id (`mug_1`), a label (`mug`) and a colour (`red`) from its name, a room, and what it stands on (from
+   the renderers' bounds). They move under the *Scene Agent* object.
+4. **Tools → Scene Agent → Validate Scene.** Fix what it reports (duplicate ids, labels the model does not know,
+   missing model files).
+5. **Play.** Type `put the red mug on the table` and press Enter. When the agent asks "Which mug do you mean?", click
+   one. F1 hides the box.
+
+## Use it from your code
+```csharp
+using SceneAgent;
+
+public class MyVoiceUI : MonoBehaviour
+{
+    public SceneAgentSession session;          // on the Scene Agent object
+
+    void Start()
+    {
+        session.Line += text => Debug.Log(text);                        // "You: …", "→ call  result", "Agent: …"
+        session.Finished += r => { if (r.clarificationQuestion != null) ShowChoices(session.Candidates); };
+        session.world.ToolExecuted += (tool, args, result) =>
+        {
+            if (tool == "teleport") MoveCameraTo(session.world.playerRoom);  // react in your own way
+        };
+    }
+
+    public void OnSpeech(string text) => StartCoroutine(session.Command(text));   // typed or recognised speech
+    public void OnPick(string itemId) => StartCoroutine(session.Answer(itemId));  // the user's answer to "which one?"
+
+    void ShowChoices(System.Collections.Generic.List<string> ids) { /* your UI */ }
+    void MoveCameraTo(string room) { /* your rig */ }
+}
+```
+- **Lower level:** `StartCoroutine(agent.Run(text, result => …))` returns an `AgentResult`: the calls, their results,
+  the final text or the clarification question, an error, and the time.
+- **Rooms:** give each `SceneItem` a `room`. The model sees the player's room only; it finds objects elsewhere with
+  `find_objects` and moves the player with `teleport` (which sets `SceneWorld.playerRoom`; you move the camera in
+  `ToolExecuted`).
+- **Gaze:** `SceneAgentGaze` (added by Set Up Scene) sets `SceneWorld.gaze` to the object under the mouse pointer, or at
+  the centre of the camera (XR head gaze), so "this / that" resolve. Objects need colliders.
+- **Clarification:** when the agent asks "Which toolbox do you mean?", the user can click a candidate (`session.Answer(id)`)
+  or just type "the red one": a short reply while a question is pending answers it; a new command starts over.
+
+## What objects can do
+Each label maps to what it affords (can be picked up, is a surface, a container, has on/off or open/closed states, is
+fixed), from the model's catalogue of about 200 everyday object types. For a label the catalogue does not know, tick
+`SceneItem.overrideAffordances` and set the flags, or rename it to a common word; Validate Scene lists every such
+label.
+
+## Performance
+| setting (4 GB laptop GPU) | Maverick 4B |
+|---|---|
+| model alone, first answer | 1.2 s |
+| next to a rendering scene, per command, GPU cool | 0.5–3.6 s |
+| next to a rendering scene, per command, GPU at its thermal limit | 5.9 s |
+
+- **Cap the frame rate.** An uncapped renderer on the same GPU slowed the model about 5×. `LocalModelServer`
+  sets `Application.targetFrameRate` to `capFrameRate` (30) while the model runs on the GPU; set 0 in XR, where the
+  headset sets the frame rate.
+- **Heat:** long sessions on a laptop hold the GPU at its thermal limit and slow everything; cooling helps more than
+  any setting.
+
+## Test and QA
+- **Session logs:** `SceneAgentSession.logDirectory` (set by Set Up Scene to `SceneAgentSessions`) writes every
+  command, the scene the model saw, the calls and the answer to `Application.persistentDataPath/SceneAgentSessions`.
+- **Scripted runs:** add `SceneAgentScriptRunner`, give it a text file:
+  ```
+  pick up the red mug
+  #answer mug_1
+  put it on the table
+  #look lamp_2
+  turn that on
+  ```
+  It writes `report.json` (calls, answers, errors, time per step, optional screenshots). From a build, in any scene with
+  a Scene Agent (the runner adds itself): `Game.exe -sceneAgentScript commands.txt -sceneAgentOut results -sceneAgentShots`.
+
+## Speech input
+Any speech-to-text works: pass the text to `session.Command`. For a fully offline app use a local recogniser (for
+example whisper.cpp); cloud dictation sends audio off the device. The model was trained on speech-recognition noise
+("um put er the mug…"), not on typing errors.
+
+## Limitations
+- **English only; the 11 tools only.** No creating or deleting objects, materials, lighting, physics or code.
+- **Acting on an object in another room:** it searches other rooms for fetch/find requests ("bring me the book",
+  "where is…"), but "turn on the lamp" when the lamp is elsewhere is answered with "there is no lamp here", and "put
+  the drill on the workbench" with the drill elsewhere made the model reach for a similar object here (the hammer, the
+  screwdriver). The substitute guard blocks that call; the user should say the room or go there first.
+- **Placing puts objects at the target's centre:** two things put on the same shelf overlap. Arrange them in
+  `ToolExecuted` if it matters in your scene.
+- **Questions about the scene** ("what is on the workbench?") are answered from the scene data but were not evaluated
+  and can be wrong (in one test it listed an object that had been moved).
+- **Room size:** a room of 90 objects worked; 170 exceeded the 4096-token context (a clear error, no crash). Keep rooms
+  under about 60 objects for multi-step commands, or raise `contextSize` (more VRAM).
+- **Other languages:** trained on English only. A Turkish command worked in one test; do not rely on it.
+- **Typos** can be read as unknown objects ("lapm").
+- **Infeasible requests (4B):** it sometimes tries the action, Unity refuses it, then it explains.
+- **It sees the scene as data, not the camera.** Keep labels and colours accurate.
+- Not validated for real robots or safety-critical control.
+
+Full evaluation: 21 of 22 pre-registered capability targets, 83.6–86.6% on real human instructions (ALFRED-TC, our
+adaptation), 94.5–97.3% on 440 commands through this C# code.
+
+## Troubleshooting
+| symptom | fix |
+|---|---|
+| "llama-server not found" | Tools → Scene Agent → Open Model Folder; copy the llama.cpp release there (all files) |
+| "no model (.gguf) found" | put one `.gguf` in the same folder, or set `LocalModelServer.modelPath` |
+| model loads on the CPU | the GPU build of llama.cpp does not match your GPU or VRAM is full; see the log (`LocalModelServer.LogPath`) |
+| every answer is slow | cap the frame rate; check GPU temperature |
+| "the scene and request need N tokens…" | too many objects in the room: split it, or raise `LocalModelServer.contextSize` |
+| "not done: … is in the storage, not in this room" | the substitute guard: the user named an object in another room |
+| "object … is not visible" | the object's `room` differs from `SceneWorld.playerRoom` |
+| grab/place refused on your object | its label is not in the catalogue: `overrideAffordances`, or a common word |
+| objects jump when first moved | `SceneWorld.initFromTransforms` must be on (Set Up Scene turns it on) |
+
+## License
+Package code: MIT (`LICENSE.md`). The model weights are a fine-tune of Qwen3 (Apache-2.0). llama.cpp is MIT. See
+`Third Party Notices.md`.