ErenAta00
/

Maverick-Unity

@@ -14,170 +14,162 @@ tags:
 # Maverick XR Agent for Unity
-The Unity package for [Maverick-4B-Unity-XR-Agent](https://huggingface.co/ErenAta00/Maverick-4B-Unity-XR-Agent-GGUF).
-Let people control your Unity scene in plain English, typed or spoken, with a small language model that runs **on
-the same machine, offline**. "Put the red mug on the table", "turn on the lamp", "where is the stapler?": the model
-reads the objects you marked as interactive, calls the right tool, and Unity executes it. When the words fit two
-objects it asks which one; when an object is in another room it searches there; when a request is impossible it says
-so instead of acting.
-> **Status: release candidate (1.0.0-rc.2).** Tested on Windows 11 x64, Unity 2022.3 LTS, NVIDIA RTX
-> 3050 Ti Laptop (4 GB). See [Limitations](#limitations) before building on it.
-- **11 tools:** grab, release, place, move_by, rotate, set_state, press, highlight, teleport, find_objects,
-  ask_clarification.
-- **Understands** synonyms and general words ("the fruit"), colours, spatial references ("the one next to the chair",
-  "the second from the left"), "this / that" with the user's gaze, the object in hand ("put it on the shelf"), filler
-  words and speech-recognition noise.
-- **Safe by default:**
-  - D5b: a call to a tool or a value that does not exist is turned into a refusal before it reaches your scene; an
-    unknown object id is generated again under a grammar.
-  - Substitute guard: when the user names an object that is only in another room and the model tries to act on a
-    different object here, the call is not executed and the model is told where the named object is.
-- **Private and free to run:** llama.cpp on `127.0.0.1`; no account, no cloud call, no per-request cost.
 ## Requirements
-| | |
-|---|---|
-| Unity | 2022.3 LTS (tested 2022.3.62f3) |
-| OS | Windows x64 (tested). llama.cpp also runs on Linux and macOS; `LocalModelServer` is written for them but untested |
-| GPU | ~3.1 GB free VRAM; CPU fallback works but is slow |
-| Disk | 2.5 GB for the model, plus ~100 MB for llama.cpp |
-## Install
-1. **Package:** Window → Package Manager → **+** → *Add package from git URL…* →
-   `https://huggingface.co/ErenAta00/Maverick-Unity.git` (or *Add package from disk…* → this folder's `package.json`).
-2. **Model files:** Tools → Scene Agent → **Open Model Folder** (creates `Assets/StreamingAssets/SceneAgent/`) and put
-   there:
-   - `llama-server.exe` and the DLLs next to it, from a llama.cpp release for Windows (CUDA 12 build for NVIDIA GPUs,
-     Vulkan build for other GPUs, CPU build otherwise). Copy the whole folder's contents, not only the .exe.
-   - The model: `Maverick-4B-Unity-XR-Agent-Q4_K_M.gguf` (2.5 GB) from
-     https://huggingface.co/ErenAta00/Maverick-4B-Unity-XR-Agent-GGUF.
-     The server takes the first `.gguf` it finds there, or set `LocalModelServer.modelPath`.
-StreamingAssets is copied into a build as it is, so a built game runs without installing anything.
-## Quick start (5 minutes)
-1. **Tools → Scene Agent → Set Up Scene.** Adds a *Scene Agent* object with everything wired: `SceneWorld` (your
-   objects, as the model sees them), `SceneAgent` (the conversation with the model), `LocalModelServer` (starts
-   llama.cpp), `SceneAgentSession` and `SceneAgentChatBox` (an on-screen box to try it).
-2. **Build or open your scene.** Name objects with plain words: *Red Mug*, *Table*, *Floor Lamp*, *Book*.
-3. **Select the objects** the agent may use → **Tools → Scene Agent → Make Selected Interactive.** Each gets a
-   `SceneItem`: an id (`mug_1`), a label (`mug`) and a colour (`red`) from its name, a room, and what it stands on (from
-   the renderers' bounds). They move under the *Scene Agent* object.
-4. **Tools → Scene Agent → Validate Scene.** Fix what it reports (duplicate ids, labels the model does not know,
-   missing model files).
-5. **Play.** Type `put the red mug on the table` and press Enter. When the agent asks "Which mug do you mean?", click
-   one. F1 hides the box.
-## Use it from your code
 ```csharp
 using SceneAgent;
-public class MyVoiceUI : MonoBehaviour
 {
-    public SceneAgentSession session;          // on the Scene Agent object
     void Start()
     {
-        session.Line += text => Debug.Log(text);                        // "You: …", "→ call  result", "Agent: …"
-        session.Finished += r => { if (r.clarificationQuestion != null) ShowChoices(session.Candidates); };
         session.world.ToolExecuted += (tool, args, result) =>
         {
-            if (tool == "teleport") MoveCameraTo(session.world.playerRoom);  // react in your own way
         };
     }
-    public void OnSpeech(string text) => StartCoroutine(session.Command(text));   // typed or recognised speech
-    public void OnPick(string itemId) => StartCoroutine(session.Answer(itemId));  // the user's answer to "which one?"
-    void ShowChoices(System.Collections.Generic.List<string> ids) { /* your UI */ }
-    void MoveCameraTo(string room) { /* your rig */ }
 }
 ```
-- **Lower level:** `StartCoroutine(agent.Run(text, result => …))` returns an `AgentResult`: the calls, their results,
-  the final text or the clarification question, an error, and the time.
-- **Rooms:** give each `SceneItem` a `room`. The model sees the player's room only; it finds objects elsewhere with
-  `find_objects` and moves the player with `teleport` (which sets `SceneWorld.playerRoom`; you move the camera in
-  `ToolExecuted`).
-- **Gaze:** `SceneAgentGaze` (added by Set Up Scene) sets `SceneWorld.gaze` to the object under the mouse pointer, or at
-  the centre of the camera (XR head gaze), so "this / that" resolve. Objects need colliders.
-- **Clarification:** when the agent asks "Which toolbox do you mean?", the user can click a candidate (`session.Answer(id)`)
-  or just type "the red one": a short reply while a question is pending answers it; a new command starts over.
-## What objects can do
-Each label maps to what it affords (can be picked up, is a surface, a container, has on/off or open/closed states, is
-fixed), from the model's catalogue of about 200 everyday object types. For a label the catalogue does not know, tick
-`SceneItem.overrideAffordances` and set the flags, or rename it to a common word; Validate Scene lists every such
-label.
-## Performance
-| setting (4 GB laptop GPU) | Maverick-4B-Unity-XR-Agent |
 |---|---|
-| model alone, first answer | 1.2 s |
-| next to a rendering scene, per command, GPU cool | 0.5–3.6 s |
-| next to a rendering scene, per command, GPU at its thermal limit | 5.9 s |
-- **Cap the frame rate.** An uncapped renderer on the same GPU slowed the model about 5×. `LocalModelServer`
-  sets `Application.targetFrameRate` to `capFrameRate` (30) while the model runs on the GPU; set 0 in XR, where the
-  headset sets the frame rate.
-- **Heat:** long sessions on a laptop hold the GPU at its thermal limit and slow everything; cooling helps more than
-  any setting.
-## Test and QA
-- **Session logs:** `SceneAgentSession.logDirectory` (set by Set Up Scene to `SceneAgentSessions`) writes every
-  command, the scene the model saw, the calls and the answer to `Application.persistentDataPath/SceneAgentSessions`.
-- **Scripted runs:** add `SceneAgentScriptRunner`, give it a text file:
-  ```
-  pick up the red mug
-  #answer mug_1
-  put it on the table
-  #look lamp_2
-  turn that on
-  ```
-  It writes `report.json` (calls, answers, errors, time per step, optional screenshots). From a build, in any scene with
-  a Scene Agent (the runner adds itself): `Game.exe -sceneAgentScript commands.txt -sceneAgentOut results -sceneAgentShots`.
-## Speech input
-Any speech-to-text works: pass the text to `session.Command`. For a fully offline app use a local recogniser (for
-example whisper.cpp); cloud dictation sends audio off the device. The model was trained on speech-recognition noise
-("um put er the mug…"), not on typing errors.
 ## Limitations
-- **English only; the 11 tools only.** No creating or deleting objects, materials, lighting, physics or code.
-- **Acting on an object in another room:** it searches other rooms for fetch/find requests ("bring me the book",
-  "where is…"), but "turn on the lamp" when the lamp is elsewhere is answered with "there is no lamp here", and "put
-  the drill on the workbench" with the drill elsewhere made the model reach for a similar object here (the hammer, the
-  screwdriver). The substitute guard blocks that call; the user should say the room or go there first.
-- **Placing puts objects at the target's centre:** two things put on the same shelf overlap. Arrange them in
-  `ToolExecuted` if it matters in your scene.
-- **Questions about the scene** ("what is on the workbench?") are answered from the scene data but were not evaluated
-  and can be wrong (in one test it listed an object that had been moved).
-- **Room size:** a room of 90 objects worked; 170 exceeded the 4096-token context (a clear error, no crash). Keep rooms
-  under about 60 objects for multi-step commands, or raise `contextSize` (more VRAM).
-- **Other languages:** trained on English only. A Turkish command worked in one test; do not rely on it.
-- **Typos** can be read as unknown objects ("lapm").
-- **Infeasible requests (4B):** it sometimes tries the action, Unity refuses it, then it explains.
-- **It sees the scene as data, not the camera.** Keep labels and colours accurate.
-- Not validated for real robots or safety-critical control.
-Full evaluation: 21 of 22 pre-registered capability targets, 83.6–86.6% on real human instructions (ALFRED-TC, our
-adaptation), 94.5–97.3% on 440 commands through this C# code.
 ## Troubleshooting
-| symptom | fix |
 |---|---|
-| "llama-server not found" | Tools → Scene Agent → Open Model Folder; copy the llama.cpp release there (all files) |
-| "no model (.gguf) found" | put one `.gguf` in the same folder, or set `LocalModelServer.modelPath` |
-| model loads on the CPU | the GPU build of llama.cpp does not match your GPU or VRAM is full; see the log (`LocalModelServer.LogPath`) |
-| every answer is slow | cap the frame rate; check GPU temperature |
-| "the scene and request need N tokens…" | too many objects in the room: split it, or raise `LocalModelServer.contextSize` |
-| "not done: … is in the storage, not in this room" | the substitute guard: the user named an object in another room |
-| "object … is not visible" | the object's `room` differs from `SceneWorld.playerRoom` |
-| grab/place refused on your object | its label is not in the catalogue: `overrideAffordances`, or a common word |
-| objects jump when first moved | `SceneWorld.initFromTransforms` must be on (Set Up Scene turns it on) |
-## License
-Package code: MIT (`LICENSE.md`). The model weights are a fine-tune of Qwen3 (Apache-2.0). llama.cpp is MIT. See
-`Third Party Notices.md`.

 # Maverick XR Agent for Unity
+Control a Unity scene with typed or spoken English. This package runs
+[Maverick-4B-Unity-XR-Agent](https://huggingface.co/ErenAta00/Maverick-4B-Unity-XR-Agent-GGUF) on the player's own
+machine through llama.cpp: no account, no internet connection and no API key.
+You choose which objects the agent may use. For each command the model reads those objects and calls one of 11 tools,
+which Unity then carries out: "put the red mug on the table", "turn on the lamp", "where is the stapler?". When a
+command could mean more than one object, it asks which one. When it cannot do something, it says so.
 ## Requirements
+- Unity 2022.3 LTS or later
+- Windows x64 (Linux and macOS are untested)
+- About 3 GB of free GPU memory. Without a GPU the model runs on the CPU, but slowly.
+- About 3 GB of disk space for the model and llama.cpp
+## Installation
+1. In Unity, open **Window > Package Manager**, click **+**, choose **Add package from git URL** and enter:
+   ```text
+   https://huggingface.co/ErenAta00/Maverick-Unity.git
+   ```
+2. Choose **Tools > Scene Agent > Open Model Folder**. This creates `Assets/StreamingAssets/SceneAgent/`.
+3. Copy two things into that folder:
+   - **llama.cpp for Windows** from the [llama.cpp releases](https://github.com/ggml-org/llama.cpp/releases): the CUDA 12
+     build for NVIDIA GPUs, the Vulkan build for other GPUs, or the CPU build. Copy every file in the archive, not only
+     `llama-server.exe`.
+   - **The model** `Maverick-4B-Unity-XR-Agent-Q4_K_M.gguf` from the
+     [model page](https://huggingface.co/ErenAta00/Maverick-4B-Unity-XR-Agent-GGUF/tree/main).
+Unity copies `StreamingAssets` into your build, so players do not need to install anything.
+## Quick start
+1. Choose **Tools > Scene Agent > Set Up Scene**. This adds a *Scene Agent* object with all components connected,
+   including a chat box for testing.
+2. Give your objects plain names, such as *Red Mug*, *Table* or *Floor Lamp*.
+3. Select the objects the agent may use and choose **Tools > Scene Agent > Make Selected Interactive**. Each one gets a
+   `SceneItem` with an id (`mug_1`), a label (`mug`), a colour (`red`), a room and the object it stands on.
+4. Choose **Tools > Scene Agent > Validate Scene** and fix anything it reports.
+5. Enter Play mode, type `put the red mug on the table` and press Enter. If the agent asks which mug you mean, click
+   one. F1 hides the chat box.
+## Using it from code
 ```csharp
+using System.Collections.Generic;
 using SceneAgent;
+using UnityEngine;
+public class VoiceCommands : MonoBehaviour
 {
+    public SceneAgentSession session;   // on the Scene Agent object
     void Start()
     {
+        session.Line += text => Debug.Log(text);
+        session.Finished += result =>
+        {
+            if (result.clarificationQuestion != null) ShowChoices(session.Candidates);
+        };
         session.world.ToolExecuted += (tool, args, result) =>
         {
+            if (tool == "teleport") MoveRigTo(session.world.playerRoom);
         };
     }
+    // Typed text or the output of a speech recogniser.
+    public void OnCommand(string text) => StartCoroutine(session.Command(text));
+    // The user's answer to "Which one do you mean?".
+    public void OnPick(string itemId) => StartCoroutine(session.Answer(itemId));
+    void ShowChoices(List<string> itemIds) { /* your UI */ }
+    void MoveRigTo(string room) { /* your camera rig */ }
 }
 ```
+- **Rooms.** Give each `SceneItem` a `room`. The model sees only the player's room. It searches other rooms with
+  `find_objects` and moves the player with `teleport`, which sets `SceneWorld.playerRoom`; move your camera rig in
+  `ToolExecuted`.
+- **"This" and "that".** `SceneAgentGaze` points them at the object under the mouse, or at the centre of the view in
+  XR. The objects need colliders.
+- **Follow-up answers.** When the agent asks which object you mean, the user can pick one (`session.Answer`) or reply
+  in words, such as "the red one".
+- **Speech.** Pass the recognised text to `session.Command`. For a fully offline app, use a local recogniser such as
+  whisper.cpp.
+## Tools
+| Tool | What it does |
 |---|---|
+| `grab`, `release` | Picks up or drops an object with the left, right or both hands |
+| `place` | Puts an object on a surface or in a container |
+| `move_by`, `rotate` | Moves an object by a given distance or turns it |
+| `set_state` | Switches on or off, opens, closes, locks or unlocks |
+| `press` | Presses a button |
+| `highlight` | Highlights objects |
+| `teleport` | Moves the player to another room |
+| `find_objects` | Searches every room for objects |
+| `ask_clarification` | Asks which object the user means |
+Unity refuses a call that does not fit the object, such as picking up a table, and tells the model why. What each
+object can do comes from its label and a catalogue of about 200 everyday objects. For a label outside the catalogue,
+turn on `SceneItem.overrideAffordances` and set the flags yourself; **Validate Scene** lists these labels.
+## Components
+| Component | Purpose | Main settings |
+|---|---|---|
+| `SceneWorld` | Describes the scene to the model and carries out the tools | `rooms`, `playerRoom`, `ToolExecuted` |
+| `SceneItem` | Marks an object the agent may use | `itemId`, `label`, `color`, `room`, `overrideAffordances` |
+| `SceneAgent` | Sends commands to the model and runs its tool calls | `maxTurns`, `fallbackDecoding`, `guardSubstitutes` |
+| `LocalModelServer` | Starts and stops llama.cpp with the game | `modelPath`, `contextSize`, `preferGpu`, `capFrameRate` |
+| `SceneAgentSession` | Holds one conversation, including follow-up questions | `logDirectory` |
+| `SceneAgentGaze` | Resolves "this" and "that" | |
+| `SceneAgentChatBox` | On-screen chat for testing | `toggleKey`, `visibleLines` |
+| `SceneAgentScriptRunner` | Plays a text file of commands and writes a report | `script`, `outputDirectory` |
+`fallbackDecoding` checks every answer before it reaches your scene: a call to a tool or value that does not exist is
+turned into a refusal. `guardSubstitutes` stops the model from acting on a similar object when the one the user named
+is in another room. Leave both on.
+## Performance
+- **Cap the frame rate.** An uncapped renderer on the same GPU can make the model several times slower.
+  `LocalModelServer` caps rendering at 30 frames per second while the model runs on the GPU. In XR, set `capFrameRate`
+  to 0 and let the headset set the frame rate.
+- **Keep rooms small.** Under about 60 interactive objects per room, multi-step commands fit the default 4,096-token
+  context. For larger rooms, raise `LocalModelServer.contextSize`, which needs more GPU memory.
 ## Limitations
+- English only, and only the 11 tools. The agent cannot create or delete objects or change materials, lighting or
+  physics.
+- It finds and fetches objects in other rooms ("bring me the book"), but when the lamp is in another room, "turn on
+  the lamp" gets "there is no lamp here". The user should name the room or go there first.
+- Placed objects go to the centre of the target, so two objects placed on the same shelf overlap. Arrange them in
+  `ToolExecuted` if this matters in your scene.
+- The model reads the scene as data, not through the camera. Keep object names and colours accurate.
+- Not intended for robots or safety-critical control.
 ## Troubleshooting
+| Problem | Fix |
 |---|---|
+| "llama-server not found" | Copy every file of the llama.cpp release into the model folder. |
+| "no model (.gguf) found" | Put the `.gguf` file in the model folder, or set `LocalModelServer.modelPath`. |
+| The model runs on the CPU | The llama.cpp build does not match your GPU, or GPU memory is full. See the log at `LocalModelServer.LogPath`. |
+| Every answer is slow | Cap the frame rate and check the GPU temperature. |
+| "the scene and request need N tokens" | The room has too many objects. Split it or raise `LocalModelServer.contextSize`. |
+| "object ... is not visible" | The object's `room` is not `SceneWorld.playerRoom`. |
+| A call on your object is refused | Its label is not in the catalogue. Use `overrideAffordances` or a more common name. |
+## Licence
+The package is released under the [MIT License](LICENSE.md). The model is released under Apache-2.0, like its base
+model, Qwen3-4B. llama.cpp is released under the MIT License.