diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000000000000000000000000000000000000..f045bb3b6847d243df186ca0e60c2fa540f6dd19 --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,44 @@ +# Changelog + +## [1.0.0-rc.2] - 2026-09-28 (not published) +### Changed +- Renamed to **Maverick XR Agent** (`com.mcbuxrlab.maverick`) for the Maverick 4B model; the C# namespaces and + component names are unchanged. +- Install from Git: `https://huggingface.co/ErenAta00/Maverick-Unity.git`. +- Licence holder: Eren Ata, MCBU XRLab (MIT). + +## [1.0.0-rc.1] - 2026-09-27 (not published) +### Added +- `LocalModelServer` in the package (was in the demo): finds llama-server and the `.gguf` in + `StreamingAssets/SceneAgent`, GPU first with CPU fallback, `capFrameRate` while the model runs on the GPU, + `ModeChanged` event. +- `SceneAgentSession`: commands, clarification answers with the trained follow-up, optional JSON-lines session log. +- `SceneAgentChatBox`: on-screen chat to try the agent without writing UI. +- `SceneAgentScriptRunner`: plays command scripts and writes a report (Editor, build, or command line). +- D5b answer checking (`SceneGrammar.Check`, `SceneGrammar.Schema`), on by default, verified against the evaluation's + Python check on 59,294 cases. +- Editor menu Tools → Scene Agent: Set Up Scene, Make Selected Interactive, Validate Scene, Open Model Folder. +- `SceneWorld.initFromTransforms` (positions read from the scene at Start), `SceneWorld.ToolExecuted` event. +- `SceneItem` affordance overrides for labels outside the catalogue. +- The v3 system prompt ships in `Resources/SceneAgent`. +- README, reference documentation, MIT license, third-party notices. +- `SceneAgentGaze`: "this / that" from the mouse pointer or the camera centre. +- Typed clarification answers ("the red one") in `SceneAgentSession`. +- Substitute guard (`SceneAgent.guardSubstitutes`). +- The model server is tied to the app's lifetime on Windows (a job object: no llama-server left after a crash) and + waits for the old server to exit on a scene reload. +- `-sceneAgentScript` works in any scene with a session. +- Readable tool lines in the chat (`place(hammer_1, toolbox_1, in) ok`); raw JSON with `SceneAgentSession.rawToolLines`. +- `SceneAgentScriptRunner` gives the gaze back to `SceneAgentGaze` when the script ends. +### Fixed +- Play mode in the Editor: an unassigned `systemPromptAsset` threw `UnassignedReferenceException` on the first command. +- Make Selected Interactive: duplicated objects (Ctrl+D) get a fresh id; nested objects move directly under the Scene + Agent object and their supports are inferred from their own renderers; devices get a default state. +- Validate Scene: a model path pointing nowhere is an error; no dialogs in batch mode. +- A context overflow gives a readable message instead of the raw HTTP 400. +### Changed +- `SceneAgent` defaults: D5b on, strict D5 off; waits for its `LocalModelServer` when one is set. + +## [0.1.0] +- SceneWorld (tool semantics mirroring the training simulator), SceneAgent (chat loop), strict D5 grammar, the Unity + E2E harness. diff --git a/CHANGELOG.md.meta b/CHANGELOG.md.meta new file mode 100644 index 0000000000000000000000000000000000000000..c29da60a8f8c28318bf130646d46c8a7efb7ad0a --- /dev/null +++ b/CHANGELOG.md.meta @@ -0,0 +1,7 @@ +fileFormatVersion: 2 +guid: 88b114c1c231c3b4d856afc169a4b9d9 +TextScriptImporter: + externalObjects: {} + userData: + assetBundleName: + assetBundleVariant: diff --git a/Documentation~/SceneAgent.md b/Documentation~/SceneAgent.md new file mode 100644 index 0000000000000000000000000000000000000000..d72569c541f4712df82c62c1efd61cbff87766a0 --- /dev/null +++ b/Documentation~/SceneAgent.md @@ -0,0 +1,98 @@ +# Scene Agent: reference + +Start with the package `README.md` (install, quick start). This page lists what each part does and how the model sees +your scene. + +## How a command runs +1. `SceneWorld.BuildSceneJson()` lists the player's room: every `SceneItem` there, the room names, the player's hands + and gaze. +2. `SceneAgent.Run` sends the system prompt, the scene and the request to the local model (OpenAI-compatible + `/v1/chat/completions` on llama.cpp, temperature 0). +3. The answer is either text (done) or tool calls. With D5b (default) each answer is checked first (below). +4. Each call runs through `SceneWorld.Execute`, which changes the logical state, moves the GameObject + (`SceneWorld.Sync`) and raises `ToolExecuted`. A call Unity refuses (`ToolError`) goes back to the model as + `{"ok": false, "error": "…"}`. +5. Results go back to the model; up to `maxTurns` (4) rounds. `ask_clarification` ends the turn with a question; the + user's pick comes back as the next request (`SceneAgentSession.Answer` builds it). + +## What the model sees +```json +{"player": {"room": "kitchen", "gaze": null, "left_hand": null, "right_hand": "mug_1"}, + "locations": ["kitchen", "living_room", "office"], + "visible_objects": [ + {"id": "counter_1", "label": "counter", "on": "floor", "pos": [-1.6, 0.0, 1.2]}, + {"id": "mug_2", "label": "mug", "color": "red", "on": "counter_1", "pos": [-1.6, 0.9, 1.2]}, + {"id": "fridge_1", "label": "fridge", "color": "white", "on": "floor", "state": "closed", "pos": [1.9, 0.0, 2.4]}]} +``` +Positions are room-local metres of the object's bottom, rounded to 0.1 m. Only the player's room is listed. + +## Components +| component | fields you may set | +|---|---| +| `SceneWorld` | `rooms`, `playerRoom`, `gaze`, `leftHand` / `rightHand`, `leftHandAnchor` / `rightHandAnchor` (where held objects are shown), `initFromTransforms` (read positions from the scene at Start), `roomOrigin` (code: where each room sits in world space) | +| `SceneItem` | `itemId` (unique, e.g. `mug_1`), `label` (plain noun), `color`, `room`, `on` (support id or `floor`), `state` (e.g. `off`, `closed`), affordance override flags | +| `SceneAgent` | `world`, `server`, `endpoint`, `systemPromptAsset` (leave empty), `maxTurns`, `fallbackDecoding` (D5b, on), `constrainedDecoding` (strict D5, off) | +| `LocalModelServer` | `serverExe`, `modelPath`, `port`, `contextSize`, `preferGpu`, `cpuThreads`, `capFrameRate`, `launchOnStart`, `extraArgs` | +| `SceneAgentSession` | `agent`, `world`, `logDirectory`; `Command(text)`, `Answer(itemId)`, events `Line`, `Finished`, `PendingRequest`, `Candidates` | +| `SceneAgentChatBox` | `visibleLines`, `toggleKey` (F1), `examples` | +| `SceneAgentScriptRunner` | `script`, `outputDirectory`, `screenshots`, `quitWhenDone`; command line `-sceneAgentScript`, `-sceneAgentOut`, `-sceneAgentShots` | + +## Tools +| tool | arguments | Unity refuses when | +|---|---|---| +| `grab` | object_id, hand (left/right/both) | not graspable, already held, hand not free | +| `release` | hand | nothing in that hand (the object drops to the floor) | +| `place` | object_id, target_id, relation (on/in), orientation (default/upright/horizontal/vertical/upside_down) | object not carryable; target not a surface (on) or not a container / closed (in) | +| `move_by` | object_id, direction (left/right/forward/backward/up/down), distance, unit (m/cm) | held or fixed object | +| `rotate` | object_id, degrees, axis (yaw/pitch/roll) | held or fixed object | +| `set_state` | object_id, state (on/off/open/closed/half_open/locked/unlocked) | the object has no such state | +| `press` | object_id, hold_seconds | not a button | +| `highlight` | object_ids | an id not in the room | +| `teleport` | location_id | not one of `rooms` (held objects come along) | +| `find_objects` | label, color, room | never; returns the matches in every room | +| `ask_clarification` | question, candidate_ids | never; sets `SceneWorld.clarify` and ends the turn | + +Every tool acts on the player's room only: an object elsewhere is "not visible" until `teleport`. + +## Affordances +`SceneCatalog` (generated from the training simulator) maps about 200 labels to what they afford: grasp, surface +height, container, fixed, states, button. `SceneItem.overrideAffordances` replaces the lookup for one item with its +own flags (`grabbable`, `isSurface` + `surfaceHeight`, `isContainer`, `isFixed`, `isButton`, `states`). The model +learned the catalogue's common sense; overrides that contradict it (a grabbable "table") will confuse it. + +## D5b: checking the model's answer +With `fallbackDecoding` on, each answer is checked against the tool schema and the scene (`SceneGrammar.Check`, +byte-for-byte the same verdicts as the evaluation's Python check, 59,294 cases): +- `ok` → executed as it is. +- `invalid_tool` (a tool that does not exist) or `invalid_value` (an argument or value outside the schema, or a room + not in `rooms`) → replaced by the refusal "I can't do that; none of my tools can." +- `invalid_id` (an id not in the scene, the hands, the gaze or earlier tool results) or `malformed` (a call that does + not parse) → generated again with a grammar that only allows valid calls. +Each turn's verdict is in `AgentTurn.d5b`. Strict D5 (`constrainedDecoding`) applies the grammar to every answer; it +was measured to force actions where the model should refuse, so it is off. + +## Substitute guard +`SceneAgent.guardSubstitutes` (on): before an acting call (grab, place, move_by, rotate, set_state, press) runs, if the +acted-on object is not named in the request while the request names an object that exists only in other rooms, the +call is skipped and the model receives `{"ok": false, "error": "not done: 'screwdriver_1' is a screwdriver, but the request +names a drill; the drill (drill_1) is in the storage, not in this room"}`. Naming uses the label or its last word +("desk lamp" is named by "lamp"), with plural endings. It never fires for pronouns ("put it there"). + +## Events +- `SceneWorld.ToolExecuted (string tool, Dictionary args, object result)`: after every accepted call. +- `LocalModelServer.ModeChanged (Mode)`: Starting, Gpu, Cpu, External, Failed, Stopped. +- `SceneAgentSession.Line (string)` and `Finished (AgentResult)`. + +## Script format (SceneAgentScriptRunner) +``` +// comment +pick up the red mug a command +#answer mug_1 answer the pending "which one?" with this id +#look lamp_2 set the gaze ("#look none" clears it) +#wait 2.5 pause, in seconds (not counted in any step's time) +``` + +## Changing the model +Any GGUF of this model family works through `LocalModelServer.modelPath`. The system prompt in +`Resources/SceneAgent/system_prompt_v3.txt` must be the one the model was trained with; a different model needs its +own prompt and, for D5b, the same tool schema (`SceneGrammar.Schema.cs`, generated from `spec/tools_v0.json`). diff --git a/Editor.meta b/Editor.meta new file mode 100644 index 0000000000000000000000000000000000000000..e8a0326913368fd6e200013f563744188dcac36c --- /dev/null +++ b/Editor.meta @@ -0,0 +1,8 @@ +fileFormatVersion: 2 +guid: 4209fd575d4361a4c85651d5cd0638d2 +folderAsset: yes +DefaultImporter: + externalObjects: {} + userData: + assetBundleName: + assetBundleVariant: diff --git a/Editor/AgentE2ETest.cs b/Editor/AgentE2ETest.cs new file mode 100644 index 0000000000000000000000000000000000000000..c2d5ff737799c9ab416d9d9607f6aa62e5f9080e --- /dev/null +++ b/Editor/AgentE2ETest.cs @@ -0,0 +1,94 @@ +// End-to-end test: build real GameObjects for each case, let SceneAgent serialise the scene, query the +// model server, execute the calls on the GameObjects, and compare the final state with gold. +// Unity -batchmode -nographics -projectPath -executeMethod AgentE2ETest.Run +// -e2eInput cases.json -e2eOutput report.json +using System; +using System.Collections.Generic; +using System.IO; +using System.Linq; +using UnityEditor; +using UnityEngine; +using SceneAgent; + +[Serializable] public class E2EObj { public string id, label, color, room, on, relation, orientation, state; public bool has_pos; public double[] pos; public double yaw, pitch, roll; } +[Serializable] public class E2EWorld { public string[] rooms; public E2EObj[] objects; public string room, gaze, left_hand, right_hand; } +[Serializable] public class E2ECase { public string id; public string category; public string utterance; public E2EWorld world; public string gold_sig; public bool gold_text_end; } +[Serializable] public class E2ESuite { public string endpoint; public string model; public string prompt_path; public bool constrained; public bool fallback; public E2ECase[] cases; } +[Serializable] public class E2EResult { public string id, category, error, final_text, clarification, first_output, sig, scene, d5b; public bool ok; public int turns; public double seconds; } +[Serializable] public class E2EReport { public string unity; public int n; public double success_rate; public double mean_seconds; public List results = new List(); } + +public static class AgentE2ETest +{ + static Vector3 SizeFor(string label) + { + switch (label) + { + case "table": case "desk": return new Vector3(1.2f, 0.75f, 0.6f); + case "counter": return new Vector3(1.4f, 0.9f, 0.6f); + case "shelf": return new Vector3(1.0f, 1.2f, 0.35f); + case "door": return new Vector3(0.9f, 2.0f, 0.05f); + case "window": return new Vector3(1.0f, 1.0f, 0.05f); + case "chair": case "stool": case "crate": case "chest": return new Vector3(0.5f, 0.5f, 0.5f); + case "box": case "basket": case "vase": case "drawer": case "tv": case "fan": case "lamp": return new Vector3(0.4f, 0.4f, 0.4f); + default: return Vector3.one * 0.15f; + } + } + + static string Nn(string s) => string.IsNullOrEmpty(s) ? null : s; + + public static void Run() + { + var args = Environment.GetCommandLineArgs(); + string input = null, output = null; + for (int i = 0; i < args.Length - 1; i++) + { + if (args[i] == "-e2eInput") input = args[i + 1]; + if (args[i] == "-e2eOutput") output = args[i + 1]; + } + var suite = JsonUtility.FromJson(File.ReadAllText(input)); + string prompt = File.ReadAllText(suite.prompt_path).Replace("\r\n", "\n"); + var report = new E2EReport { unity = Application.unityVersion, n = suite.cases.Length }; + foreach (var cs in suite.cases) + { + var root = new GameObject("SceneRoot"); + var world = root.AddComponent(); + var agent = root.AddComponent(); + agent.endpoint = suite.endpoint; agent.model = suite.model; agent.systemPrompt = prompt; agent.world = world; + agent.constrainedDecoding = suite.constrained; // D5, off unless the runner asks (--grammar on) + agent.fallbackDecoding = suite.fallback; // D5b (--grammar d5b) + world.playerRoom = cs.world.room; world.gaze = Nn(cs.world.gaze); + world.leftHand = Nn(cs.world.left_hand); world.rightHand = Nn(cs.world.right_hand); + world.rooms = cs.world.rooms.ToList(); + foreach (var o in cs.world.objects) + { + var go = GameObject.CreatePrimitive(PrimitiveType.Cube); + go.name = o.id; + go.transform.SetParent(root.transform, false); + go.transform.localScale = SizeFor(o.label); + var it = go.AddComponent(); + it.itemId = o.id; it.label = o.label; it.color = Nn(o.color); it.room = o.room; it.on = o.on; + it.relation = Nn(o.relation); it.orientation = Nn(o.orientation); it.state = Nn(o.state); + it.hasPos = o.has_pos; it.pos = (double[])o.pos.Clone(); it.yaw = o.yaw; it.pitch = o.pitch; it.roll = o.roll; + } + foreach (var it in world.Items) world.Sync(it); + + string scene = world.BuildSceneJson(); // what the model is shown; compared with the offline prompt + var r = agent.RunBlocking(cs.utterance); + string sig = world.Signature(); + var res = new E2EResult { + id = cs.id, category = cs.category, error = r.error, final_text = r.finalText, clarification = r.clarificationQuestion, sig = sig, turns = r.turns.Count, + seconds = r.seconds, first_output = r.turns.Count > 0 ? r.turns[0].modelOutput : null, scene = scene, + d5b = string.Join(",", r.turns.Select(t => t.d5b ?? "")) }; + res.ok = r.error == null && sig == cs.gold_sig && (!cs.gold_text_end || r.finalText != null); + report.results.Add(res); + // one line per command, for the live dashboard (tools/dashboard/v3_view.py reads unity_e2e.log) + Debug.Log($"[E2E] case {report.results.Count}/{report.n} {res.id} ok={res.ok} turns={res.turns}"); + UnityEngine.Object.DestroyImmediate(root); + } + report.success_rate = report.results.Count(x => x.ok) / (double)Math.Max(1, report.n); + report.mean_seconds = report.results.Average(x => x.seconds); + File.WriteAllText(output, JsonUtility.ToJson(report, true)); + Debug.Log($"[E2E] n={report.n} success={report.success_rate:F3} mean_s={report.mean_seconds:F2}"); + EditorApplication.Exit(0); + } +} diff --git a/Editor/AgentE2ETest.cs.meta b/Editor/AgentE2ETest.cs.meta new file mode 100644 index 0000000000000000000000000000000000000000..05b38592d9927f203f09de726f5bc27dc8924c08 --- /dev/null +++ b/Editor/AgentE2ETest.cs.meta @@ -0,0 +1,11 @@ +fileFormatVersion: 2 +guid: 96e9993592c204748af996724da83344 +MonoImporter: + externalObjects: {} + serializedVersion: 2 + defaultReferences: [] + executionOrder: 0 + icon: {instanceID: 0} + userData: + assetBundleName: + assetBundleVariant: diff --git a/Editor/SceneAgent.Editor.asmdef b/Editor/SceneAgent.Editor.asmdef new file mode 100644 index 0000000000000000000000000000000000000000..10a41d24864eb8cba4c9a3ab70d8788d2413d258 --- /dev/null +++ b/Editor/SceneAgent.Editor.asmdef @@ -0,0 +1,7 @@ +{ + "name": "SceneAgent.Editor", + "rootNamespace": "SceneAgent", + "references": ["SceneAgent.Runtime"], + "includePlatforms": ["Editor"], + "autoReferenced": true +} diff --git a/Editor/SceneAgent.Editor.asmdef.meta b/Editor/SceneAgent.Editor.asmdef.meta new file mode 100644 index 0000000000000000000000000000000000000000..f13db895231acaeb9ad4cf3b22cb05b4a901f026 --- /dev/null +++ b/Editor/SceneAgent.Editor.asmdef.meta @@ -0,0 +1,7 @@ +fileFormatVersion: 2 +guid: 1f3d3340a35b29a4b86a36bb8bf7e845 +AssemblyDefinitionImporter: + externalObjects: {} + userData: + assetBundleName: + assetBundleVariant: diff --git a/Editor/SceneAgentMenu.cs b/Editor/SceneAgentMenu.cs new file mode 100644 index 0000000000000000000000000000000000000000..2e29df14001613e34f67040e4c535e95106e6ad2 --- /dev/null +++ b/Editor/SceneAgentMenu.cs @@ -0,0 +1,260 @@ +// Editor tools for developers: Tools > Scene Agent > +// Set Up Scene one "Scene Agent" object with SceneWorld, SceneAgent, LocalModelServer, a session and the +// chat box, wired together (idempotent) +// Make Selected Interactive SceneItem on every selected object: id, label (and a colour word) from its name, the room, +// what it stands on (from the renderers' bounds); moved under the Scene Agent object +// Validate Scene ids, labels the model's catalogue does not know, supports, rooms, model files +// Open Model Folder StreamingAssets/SceneAgent, where llama-server and the .gguf go +using System.Collections.Generic; +using System.IO; +using System.Linq; +using System.Text; +using System.Text.RegularExpressions; +using UnityEditor; +using UnityEngine; + +namespace SceneAgent.EditorTools +{ + public static class SceneAgentMenu + { + const string Root = "Tools/Scene Agent/"; + /// Automation (tests, build scripts): no modal dialogs, results only in the Console. + public static bool Quiet; + static readonly string[] Colours = { "red", "green", "blue", "yellow", "white", "black", "gray", "grey", "orange", + "purple", "pink", "brown", "silver", "gold", "beige" }; + + static T Ensure(GameObject g) where T : Component + { + var c = g.GetComponent(); + return c != null ? c : Undo.AddComponent(g); + } + + [MenuItem(Root + "Set Up Scene", priority = 1)] + public static void SetUpScene() + { + var world = Object.FindObjectOfType(); + if (world == null) + { + var go = new GameObject("Scene Agent"); + Undo.RegisterCreatedObjectUndo(go, "Set Up Scene Agent"); + world = go.AddComponent(); + world.initFromTransforms = true; + } + var root = world.gameObject; + var agent = Ensure(root); + var server = Ensure(root); + var session = Ensure(root); + Ensure(root); + var gaze = Ensure(root); + Undo.RecordObjects(new Object[] { agent, session, gaze }, "Set Up Scene Agent"); + gaze.world = world; + gaze.session = session; + agent.world = world; + agent.server = server; + session.agent = agent; + session.world = world; + if (string.IsNullOrEmpty(session.logDirectory)) session.logDirectory = "SceneAgentSessions"; + EditorUtility.SetDirty(root); + Selection.activeGameObject = root; + Debug.Log("[SceneAgent] Scene set up on \"" + root.name + "\". Next: select the objects the agent may use and run " + + "Tools > Scene Agent > Make Selected Interactive, then Validate Scene. " + ModelFilesStatus(server)); + } + + [MenuItem(Root + "Make Selected Interactive", priority = 2)] + public static void MakeSelectedInteractive() + { + var world = Object.FindObjectOfType(); + if (world == null) { SetUpScene(); world = Object.FindObjectOfType(); } + string room = !string.IsNullOrEmpty(world.playerRoom) ? world.playerRoom : world.rooms.FirstOrDefault() ?? "room"; + var selected = new HashSet(Selection.gameObjects); + // ids taken by items that are not being (re)made now; a duplicate (Ctrl+D copies the SceneItem and its id) + // gets a fresh id below + var used = new HashSet(world.Items.Where(i => !string.IsNullOrEmpty(i.itemId) && !selected.Contains(i.gameObject)).Select(i => i.itemId)); + var made = new List(); + foreach (var go in Selection.gameObjects) + { + if (go == world.gameObject || go.GetComponent() != null) continue; + var it = Ensure(go); + Undo.RecordObject(it, "Make Interactive"); + if (string.IsNullOrEmpty(it.label)) + { + var (colour, label) = LabelFromName(go.name); + it.label = label; + if (string.IsNullOrEmpty(it.color)) it.color = colour; + } + if (string.IsNullOrEmpty(it.itemId) || used.Contains(it.itemId)) it.itemId = NextId(it.label, used); + else used.Add(it.itemId); + if (string.IsNullOrEmpty(it.state)) it.state = DefaultState(it); + if (string.IsNullOrEmpty(it.room)) it.room = room; + // directly under the Scene Agent object: SceneWorld.Sync re-parents items there anyway, and an item nested in + // another item would make that one's bounds (and so what stands on it) wrong + if (go.transform.parent != world.transform) Undo.SetTransformParent(go.transform, world.transform, "Make Interactive"); + made.Add(it); + } + Undo.RecordObject(world, "Make Interactive"); + foreach (var r in made.Select(i => i.room).Distinct()) if (!world.rooms.Contains(r)) world.rooms.Add(r); + if (string.IsNullOrEmpty(world.playerRoom)) world.playerRoom = room; + InferSupports(world); + var unknown = made.Where(i => !i.overrideAffordances && !KnownLabel(i.label)).Select(i => $"{i.itemId} ('{i.label}')").ToList(); + Debug.Log($"[SceneAgent] {made.Count} interactive: " + string.Join(", ", made.Select(i => $"{i.itemId} on {i.on}")) + + (unknown.Count > 0 ? $". Not in the model's catalogue (rename to a common word, or tick overrideAffordances): {string.Join(", ", unknown)}" : "")); + } + + [MenuItem(Root + "Make Selected Interactive", true)] + static bool CanMake() => Selection.gameObjects.Length > 0; + + [MenuItem(Root + "Validate Scene", priority = 3)] + public static void ValidateScene() + { + var sb = new StringBuilder(); + int errors = 0, warnings = 0; + void E(string m) { errors++; sb.AppendLine("ERROR " + m); } + void W(string m) { warnings++; sb.AppendLine("warning " + m); } + var world = Object.FindObjectOfType(); + if (world == null) + { + Debug.LogWarning("[SceneAgent] Validate Scene: no SceneWorld in the scene: run Tools > Scene Agent > Set Up Scene."); + if (!Application.isBatchMode && !Quiet) EditorUtility.DisplayDialog("Scene Agent", "No SceneWorld in the scene: run Tools > Scene Agent > Set Up Scene.", "OK"); + return; + } + var agent = world.GetComponent() != null ? world.GetComponent() : Object.FindObjectOfType(); + if (agent == null) E("no SceneAgent component"); + else if (agent.world == null) E("SceneAgent.world is not set"); + if (Resources.Load("SceneAgent/system_prompt_v3") == null && (agent == null || agent.systemPromptAsset == null)) + E("the system prompt is missing (package Resources/SceneAgent/system_prompt_v3.txt)"); + var server = agent != null && agent.server != null ? agent.server : Object.FindObjectOfType(); + if (server == null) W("no LocalModelServer: the agent will call " + (agent != null ? agent.endpoint : "?") + " (start a server yourself)"); + else sb.AppendLine("model " + ModelFilesStatus(server)); + if (server != null) + { + string exe = LocalModelServer.Resolve(server.serverExe), model = server.ResolveModel(exe); + if (exe == null || !File.Exists(exe)) E("llama-server not found: " + exe); + if (model == null || !File.Exists(model)) E("model file not found: " + (model ?? "no .gguf next to llama-server")); + } + + var items = Object.FindObjectsOfType(true); + if (items.Length == 0) E("no SceneItem: select objects and run Make Selected Interactive"); + foreach (var g in items.GroupBy(i => i.itemId ?? "").Where(g => g.Count() > 1)) E($"duplicate id '{g.Key}' on {string.Join(", ", g.Select(i => i.name))}"); + var ids = new HashSet(items.Select(i => i.itemId)); + foreach (var it in items) + { + string n = $"'{it.name}'"; + if (string.IsNullOrEmpty(it.itemId)) E($"{n} has no id"); + else if (!Regex.IsMatch(it.itemId, "^[a-z0-9_]+$")) W($"{n}: id '{it.itemId}' should look like label_1 (lower case, digits, _)"); + if (string.IsNullOrEmpty(it.label)) E($"{n} has no label"); + else if (!it.overrideAffordances && !KnownLabel(it.label)) + W($"{n}: label '{it.label}' is not in the model's catalogue; Unity will refuse grab/place/set_state on it " + + "unless you tick overrideAffordances (or use a common word)"); + if (!it.transform.IsChildOf(world.transform)) E($"{n} is not under '{world.name}' (SceneWorld only sees its children)"); + if (string.IsNullOrEmpty(it.state) && DefaultState(it) != null) + W($"{n}: has states ({string.Join("/", StatesOf(it))}) but none is set; the model will not know whether it is {DefaultState(it)}"); + if (string.IsNullOrEmpty(it.room)) E($"{n} has no room"); + else if (world.rooms.Count > 0 && !world.rooms.Contains(it.room)) E($"{n}: room '{it.room}' is not in SceneWorld.rooms"); + if (!string.IsNullOrEmpty(it.on) && it.on != "floor" && !it.on.EndsWith("_hand") && it.on != "both_hands" && !ids.Contains(it.on)) + E($"{n}: stands on unknown id '{it.on}'"); + } + foreach (var g in items.GroupBy(i => i.room ?? "")) + if (g.Count() > 60) W($"room '{g.Key}' has {g.Count()} objects; the model sees one room at a time and 4096 tokens hold about 60 (split the room or raise contextSize)"); + if (!world.initFromTransforms) W("SceneWorld.initFromTransforms is off: positions come from each SceneItem.pos, not from where the objects stand"); + + string head = $"{items.Length} objects, {world.rooms.Count} room(s), {errors} error(s), {warnings} warning(s)"; + Debug.Log("[SceneAgent] Validate Scene: " + head + "\n" + sb); + if (!Application.isBatchMode && !Quiet) + EditorUtility.DisplayDialog("Scene Agent · Validate Scene", head + (errors + warnings > 0 ? "\n\nDetails in the Console." : "\n\nReady to press Play."), "OK"); + } + + [MenuItem(Root + "Open Model Folder", priority = 20)] + public static void OpenModelFolder() + { + string dir = Path.Combine(Application.streamingAssetsPath, "SceneAgent"); + Directory.CreateDirectory(dir); + AssetDatabase.Refresh(); + EditorUtility.RevealInFinder(dir); + } + + // ------------------------------------------------------------------ helpers + static string ModelFilesStatus(LocalModelServer s) + { + string exe = LocalModelServer.Resolve(s.serverExe); + string model = s.ResolveModel(exe); + if (exe != null) exe = Path.GetFullPath(exe); + if (model != null) model = Path.GetFullPath(model); + bool e = exe != null && File.Exists(exe), m = model != null && File.Exists(model); + return (e ? $"llama-server: {exe}" : $"llama-server MISSING ({exe})") + " · " + + (m ? $"model: {Path.GetFileName(model)} ({new FileInfo(model).Length / 1e9:F2} GB)" : "model (.gguf) MISSING") + + (e && m ? "" : " → Tools > Scene Agent > Open Model Folder, and see the package README 'Model files'."); + } + + static string[] StatesOf(SceneItem it) => it.overrideAffordances ? (it.states ?? new string[0]) + : SceneCatalog.States.TryGetValue(it.label ?? "", out var s) ? s : new string[0]; + + /// The state a device starts in when none is set: off, else closed, else its first state. + public static string DefaultState(SceneItem it) + { + var s = StatesOf(it); + return s.Length == 0 ? null : s.Contains("off") ? "off" : s.Contains("closed") ? "closed" : s[0]; + } + + static readonly HashSet Known = new HashSet(SceneCatalog.Grasp.Concat(SceneCatalog.Top.Keys) + .Concat(SceneCatalog.Container).Concat(SceneCatalog.Fixed).Concat(SceneCatalog.States.Keys).Concat(SceneCatalog.Button)); + + public static bool KnownLabel(string label) => label != null && Known.Contains(label); + + /// "RedMug (2)" → ("red", "mug"); "Floor_Lamp" → ("", "floor lamp"). + public static (string colour, string label) LabelFromName(string name) + { + string s = Regex.Replace(name, @"\s*\(\d+\)\s*$", ""); // "Mug (2)" + s = Regex.Replace(s, "([a-z])([A-Z])", "$1 $2"); // "RedMug" + s = Regex.Replace(s, @"[_\-.]+", " "); + s = Regex.Replace(s, @"\s*\d+$", "").Trim().ToLowerInvariant(); // "mug 2" + var words = s.Split(new[] { ' ' }, System.StringSplitOptions.RemoveEmptyEntries).ToList(); + string colour = ""; + if (words.Count > 1 && Colours.Contains(words[0])) { colour = words[0] == "grey" ? "gray" : words[0]; words.RemoveAt(0); } + return (colour, string.Join(" ", words)); + } + + static string NextId(string label, HashSet used) + { + string b = Regex.Replace((label ?? "item").ToLowerInvariant(), "[^a-z0-9]+", "_").Trim('_'); + int n = 1; + while (used.Contains($"{b}_{n}")) n++; + used.Add($"{b}_{n}"); + return $"{b}_{n}"; + } + + /// What each item stands on: the highest other item whose top is just under its bottom and whose + /// footprint holds its centre; otherwise the floor. + public static void InferSupports(SceneWorld world) + { + var items = world.Items.Where(i => i.gameObject.activeInHierarchy).ToList(); + var bounds = items.ToDictionary(i => i, BoundsOf); + foreach (var a in items) + { + if (a.on != null && (a.on.EndsWith("_hand") || a.on == "both_hands")) continue; + var ba = bounds[a]; + SceneItem best = null; + foreach (var b in items) + { + if (b == a || b.room != a.room) continue; + var bb = bounds[b]; + bool under = bb.max.y <= ba.min.y + 0.05f && bb.max.y >= ba.min.y - 0.15f; + bool holds = ba.center.x >= bb.min.x && ba.center.x <= bb.max.x && ba.center.z >= bb.min.z && ba.center.z <= bb.max.z; + if (under && holds && (best == null || bb.max.y > bounds[best].max.y)) best = b; + } + Undo.RecordObject(a, "Infer supports"); + a.on = best != null ? best.itemId : "floor"; + if (best != null && string.IsNullOrEmpty(a.relation)) a.relation = "on"; + } + } + + static Bounds BoundsOf(SceneItem i) + { + // its own renderers only: not those of other items parented under it + var rs = i.GetComponentsInChildren().Where(r => r.GetComponentInParent() == i).ToArray(); + if (rs.Length == 0) return new Bounds(i.transform.position, i.transform.lossyScale); + var b = rs[0].bounds; + foreach (var r in rs.Skip(1)) b.Encapsulate(r.bounds); + return b; + } + } +} diff --git a/Editor/SceneAgentMenu.cs.meta b/Editor/SceneAgentMenu.cs.meta new file mode 100644 index 0000000000000000000000000000000000000000..e6729f6fd6dd68de5100881bdc2de5edc1cbefe6 --- /dev/null +++ b/Editor/SceneAgentMenu.cs.meta @@ -0,0 +1,11 @@ +fileFormatVersion: 2 +guid: af0a3b6f63517c24c825d827886fff16 +MonoImporter: + externalObjects: {} + serializedVersion: 2 + defaultReferences: [] + executionOrder: 0 + icon: {instanceID: 0} + userData: + assetBundleName: + assetBundleVariant: diff --git a/LICENSE.md b/LICENSE.md new file mode 100644 index 0000000000000000000000000000000000000000..ed6ab3a972963a1c712525d1c83be2b763e0a132 --- /dev/null +++ b/LICENSE.md @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2026 Eren Ata, MCBU XRLab + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/LICENSE.md.meta b/LICENSE.md.meta new file mode 100644 index 0000000000000000000000000000000000000000..2f555ceb5cb3dcdb4dcb163adc9070bb589504c7 --- /dev/null +++ b/LICENSE.md.meta @@ -0,0 +1,7 @@ +fileFormatVersion: 2 +guid: a89b7561ac2a0b54493ca7132f4097f8 +TextScriptImporter: + externalObjects: {} + userData: + assetBundleName: + assetBundleVariant: diff --git a/README.md.meta b/README.md.meta new file mode 100644 index 0000000000000000000000000000000000000000..530d3acf17e8f4d4d49f9a91c286dc1599de40b5 --- /dev/null +++ b/README.md.meta @@ -0,0 +1,7 @@ +fileFormatVersion: 2 +guid: 9731bc90f1cd16b4ca19c31b70160a0c +TextScriptImporter: + externalObjects: {} + userData: + assetBundleName: + assetBundleVariant: diff --git a/Runtime.meta b/Runtime.meta new file mode 100644 index 0000000000000000000000000000000000000000..2347fc9780a8d290311067e8368b58d101f94d47 --- /dev/null +++ b/Runtime.meta @@ -0,0 +1,8 @@ +fileFormatVersion: 2 +guid: 97273a565059476489b4f3f642df153c +folderAsset: yes +DefaultImporter: + externalObjects: {} + userData: + assetBundleName: + assetBundleVariant: diff --git a/Runtime/LocalModelServer.cs b/Runtime/LocalModelServer.cs new file mode 100644 index 0000000000000000000000000000000000000000..25045f9e3b2d06304f0ec4f5dcbaeb4cf97eb3a9 --- /dev/null +++ b/Runtime/LocalModelServer.cs @@ -0,0 +1,248 @@ +// Starts the local llama.cpp server with the model when the game starts and stops it when the game ends, so the player +// installs nothing and nothing leaves the machine (it listens on 127.0.0.1 only). GPU first; if that server does not +// come up, the same model on the CPU. A server already answering on the port is used as it is. +// +// Default layout (relative paths are looked up in StreamingAssets, then next to the project / the built game): +// StreamingAssets/SceneAgent/llama-server.exe (+ the DLLs of the same llama.cpp release) +// StreamingAssets/SceneAgent/.gguf (modelPath empty = the first .gguf in that folder) +using System; +using System.Collections; +using System.Diagnostics; +using System.IO; +using System.Linq; +using System.Net.Http; +using System.Runtime.InteropServices; +using System.Text; +using System.Threading.Tasks; +using UnityEngine; +using Debug = UnityEngine.Debug; + +namespace SceneAgent +{ + public class LocalModelServer : MonoBehaviour + { + public enum Mode { Stopped, Starting, Gpu, Cpu, External, Failed } + + [Tooltip("llama-server executable. Relative: looked up in StreamingAssets, then next to the project / the game.")] + public string serverExe = "SceneAgent/llama-server.exe"; + [Tooltip("The model (GGUF). Empty: the first .gguf in the server's folder. Relative paths resolve like serverExe.")] + public string modelPath = ""; + public int port = 8091; + [Tooltip("Tokens of context. 4096 holds a room of about 60 objects plus a few turns.")] + public int contextSize = 4096; + public bool preferGpu = true; + public int cpuThreads = 8; + [Tooltip("--cache-ram 0: llama.cpp's host prompt cache (8 GB by default) filled the machine's RAM.")] + public string extraArgs = "--cache-ram 0"; + public float startTimeoutSeconds = 120f; + [Tooltip("While the model runs on the GPU, cap rendering at this frame rate so the renderer does not starve it " + + "(uncapped rendering made the model ~5x slower on a 4 GB laptop GPU). 0 = leave the frame rate alone; " + + "use 0 in XR, where the headset sets the frame rate.")] + public int capFrameRate = 30; + [Tooltip("Start the server when this component starts. Off: call Launch() yourself.")] + public bool launchOnStart = true; + + public Mode mode { get; private set; } = Mode.Stopped; + public string status { get; private set; } = "not started"; + public string Endpoint => $"http://127.0.0.1:{port}/v1/chat/completions"; + public bool Ready => mode == Mode.Gpu || mode == Mode.Cpu || mode == Mode.External; + public bool Failed => mode == Mode.Failed; + /// Raised whenever mode changes (Starting, Gpu, Cpu, External, Failed, Stopped). + public event Action ModeChanged; + public string LogPath => Path.Combine(Application.persistentDataPath, "llama-server.log"); + + Process proc; + StreamWriter log; + static readonly HttpClient Http = new HttpClient { Timeout = TimeSpan.FromSeconds(2) }; + + IEnumerator Start() + { + if (launchOnStart) yield return Launch(); + } + + void SetMode(Mode m) + { + mode = m; + if (m == Mode.Gpu && capFrameRate > 0) + { + QualitySettings.vSyncCount = 0; + Application.targetFrameRate = capFrameRate; + Debug.Log($"[SceneAgent] frame rate capped at {capFrameRate} while the model uses the GPU (LocalModelServer.capFrameRate)"); + } + ModeChanged?.Invoke(m); + } + + /// The first existing file among: the path itself (absolute), StreamingAssets/path, next to the project + /// (Editor) or the built game. + public static string Resolve(string p) + { + if (string.IsNullOrEmpty(p)) return null; + if (Path.IsPathRooted(p)) return p; + var roots = new[] { Application.streamingAssetsPath, Path.GetFullPath(Path.Combine(Application.dataPath, "..")) }; + return roots.Select(r => Path.Combine(r, p)).FirstOrDefault(File.Exists) ?? Path.Combine(roots[0], p); + } + + /// The model file: modelPath, or the first .gguf next to the server. + public string ResolveModel(string exe) + { + if (!string.IsNullOrEmpty(modelPath)) return Resolve(modelPath); + string dir = exe != null ? Path.GetDirectoryName(exe) : null; + return dir != null && Directory.Exists(dir) ? Directory.GetFiles(dir, "*.gguf").OrderBy(f => f).FirstOrDefault() : null; + } + + public IEnumerator Launch() + { + SetMode(Mode.Starting); + // a server already answering on the port (e.g. started by hand) is used as is + var probe = HealthAsync(); + while (!probe.IsCompleted) yield return null; + if (probe.Result) { status = $"using the server already running on :{port}"; SetMode(Mode.External); yield break; } + + string exe = Resolve(serverExe), model = ResolveModel(exe); + if (exe == null || !File.Exists(exe) || model == null || !File.Exists(model)) + { + status = exe == null || !File.Exists(exe) + ? $"llama-server not found: {exe} (see the package README, 'Model files')" + : $"no model (.gguf) found: {model ?? Path.GetDirectoryName(exe)}"; + Debug.LogError("[SceneAgent] " + status); + SetMode(Mode.Failed); + yield break; + } + string common = $"-m \"{model}\" --jinja -c {contextSize} --host 127.0.0.1 --port {port} {extraArgs}"; + if (preferGpu) + { + status = "loading the model on the GPU…"; + yield return StartAndWait(exe, common + " -ngl 99", cpuOnly: false); + if (mode == Mode.Gpu) yield break; + Stop(); + SetMode(Mode.Starting); + } + status = "loading the model on the CPU…"; + yield return StartAndWait(exe, common + $" --device none --no-op-offload -ngl 0 -t {cpuThreads}", cpuOnly: true); + if (!Ready) { status = $"the model server did not start (see {LogPath})"; SetMode(Mode.Failed); } + } + + IEnumerator StartAndWait(string exe, string args, bool cpuOnly) + { + var psi = new ProcessStartInfo(exe, args) + { + UseShellExecute = false, CreateNoWindow = true, + RedirectStandardOutput = true, RedirectStandardError = true, + WorkingDirectory = Path.GetDirectoryName(exe), + }; + if (cpuOnly) psi.EnvironmentVariables["CUDA_VISIBLE_DEVICES"] = "-1"; + try + { + log = new StreamWriter(LogPath, append: false, Encoding.UTF8) { AutoFlush = true }; + proc = new Process { StartInfo = psi, EnableRaisingEvents = true }; + // the pipes must be drained, or the server blocks once the buffer is full + proc.OutputDataReceived += (_, e) => Write(e.Data); + proc.ErrorDataReceived += (_, e) => Write(e.Data); + proc.Start(); + ChildProcessGuard.Attach(proc); // dies with this process, even after a crash + proc.BeginOutputReadLine(); + proc.BeginErrorReadLine(); + Debug.Log($"[SceneAgent] model server started ({(cpuOnly ? "CPU" : "GPU")}), log: {LogPath}"); + } + catch (Exception e) + { + status = "could not start llama-server: " + e.Message; + Debug.LogError("[SceneAgent] " + status); + yield break; + } + float t0 = Time.realtimeSinceStartup; + while (Time.realtimeSinceStartup - t0 < startTimeoutSeconds) + { + if (proc.HasExited) break; + var h = HealthAsync(); + while (!h.IsCompleted) yield return null; + if (h.Result) + { + status = $"model ready on the {(cpuOnly ? "CPU" : "GPU")}"; + SetMode(cpuOnly ? Mode.Cpu : Mode.Gpu); + yield break; + } + yield return new WaitForSecondsRealtime(1f); + } + } + + void Write(string line) + { + if (line == null) return; + lock (this) { try { log?.WriteLine(line); } catch (ObjectDisposedException) { } } + } + + async Task HealthAsync() + { + try + { + var r = await Http.GetAsync($"http://127.0.0.1:{port}/health").ConfigureAwait(false); + return r.IsSuccessStatusCode; + } + catch { return false; } + } + + public void Stop() + { + // wait for the exit: a scene reload starts the next server at once, and it must find the port free + try { if (proc != null && !proc.HasExited) { proc.Kill(); proc.WaitForExit(3000); } } catch (Exception) { } + proc?.Dispose(); + proc = null; + lock (this) { log?.Dispose(); log = null; } + if (mode != Mode.External && mode != Mode.Stopped) SetMode(Mode.Stopped); + } + + void OnApplicationQuit() => Stop(); + void OnDestroy() => Stop(); + } + + /// Puts the model server in a Windows job object that is closed, and its processes killed, when this + /// process ends in any way (quit, crash, the Editor closing). Elsewhere: nothing (Stop still ends the server). + static class ChildProcessGuard + { +#if UNITY_STANDALONE_WIN || UNITY_EDITOR_WIN + [StructLayout(LayoutKind.Sequential)] + struct BasicLimits + { + public long PerProcessUserTimeLimit, PerJobUserTimeLimit; + public uint LimitFlags; + public UIntPtr MinimumWorkingSetSize, MaximumWorkingSetSize; + public uint ActiveProcessLimit; + public UIntPtr Affinity; + public uint PriorityClass, SchedulingClass; + } + [StructLayout(LayoutKind.Sequential)] + struct IoCounters { public ulong R, W, O, RB, WB, OB; } + [StructLayout(LayoutKind.Sequential)] + struct ExtendedLimits + { + public BasicLimits Basic; + public IoCounters Io; + public UIntPtr ProcessMemoryLimit, JobMemoryLimit, PeakProcessMemoryUsed, PeakJobMemoryUsed; + } + [DllImport("kernel32.dll", CharSet = CharSet.Unicode)] static extern IntPtr CreateJobObject(IntPtr attributes, string name); + [DllImport("kernel32.dll")] static extern bool SetInformationJobObject(IntPtr job, int infoClass, ref ExtendedLimits info, uint length); + [DllImport("kernel32.dll", SetLastError = true)] static extern bool AssignProcessToJobObject(IntPtr job, IntPtr process); + static IntPtr job; + + public static void Attach(Process p) + { + try + { + if (job == IntPtr.Zero) + { + job = CreateJobObject(IntPtr.Zero, null); + var info = new ExtendedLimits(); + info.Basic.LimitFlags = 0x2000; // JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE + SetInformationJobObject(job, 9, ref info, (uint)Marshal.SizeOf(typeof(ExtendedLimits))); // 9: extended limits + } + if (!AssignProcessToJobObject(job, p.Handle)) + Debug.LogWarning("[SceneAgent] could not tie the model server to this process (error " + Marshal.GetLastWin32Error() + ")"); + } + catch (Exception e) { Debug.LogWarning("[SceneAgent] could not tie the model server to this process: " + e.Message); } + } +#else + public static void Attach(Process p) { } +#endif + } +} diff --git a/Runtime/LocalModelServer.cs.meta b/Runtime/LocalModelServer.cs.meta new file mode 100644 index 0000000000000000000000000000000000000000..2a2c06964fbf549642a963d2a0245c997e5d625b --- /dev/null +++ b/Runtime/LocalModelServer.cs.meta @@ -0,0 +1,11 @@ +fileFormatVersion: 2 +guid: 886cfe192c0362948b23c41866c13b18 +MonoImporter: + externalObjects: {} + serializedVersion: 2 + defaultReferences: [] + executionOrder: 0 + icon: {instanceID: 0} + userData: + assetBundleName: + assetBundleVariant: diff --git a/Runtime/MiniJson.cs b/Runtime/MiniJson.cs new file mode 100644 index 0000000000000000000000000000000000000000..db658fc27a6475e169b54f362843048ab244cdbc --- /dev/null +++ b/Runtime/MiniJson.cs @@ -0,0 +1,150 @@ +// Minimal JSON reader/writer (objects -> Dictionary, arrays -> List, +// numbers -> double, plus string/bool/null) so the agent has no package dependencies. +using System; +using System.Collections; +using System.Collections.Generic; +using System.Globalization; +using System.Text; + +namespace SceneAgent +{ + public static class MiniJson + { + public static object Parse(string s) + { + int i = 0; + var v = ReadValue(s, ref i); + return v; + } + + /// Like Parse, but only whitespace may follow the value (Python's json.loads rejects extra data; D5b's + /// check must judge a tool call exactly as eval/grammar.py does). + public static object ParseStrict(string s) + { + int i = 0; + var v = ReadValue(s, ref i); + Skip(s, ref i); + if (i != s.Length) throw new FormatException("extra data after the JSON value"); + return v; + } + + static void Skip(string s, ref int i) { while (i < s.Length && char.IsWhiteSpace(s[i])) i++; } + + static object ReadValue(string s, ref int i) + { + Skip(s, ref i); + if (i >= s.Length) throw new FormatException("unexpected end of JSON"); + char c = s[i]; + if (c == '{') + { + var d = new Dictionary(); i++; + Skip(s, ref i); + if (s[i] == '}') { i++; return d; } + while (true) + { + Skip(s, ref i); + string key = ReadString(s, ref i); + Skip(s, ref i); + if (s[i] != ':') throw new FormatException("expected ':'"); + i++; + d[key] = ReadValue(s, ref i); + Skip(s, ref i); + if (s[i] == ',') { i++; continue; } + if (s[i] == '}') { i++; return d; } + throw new FormatException("expected ',' or '}'"); + } + } + if (c == '[') + { + var l = new List(); i++; + Skip(s, ref i); + if (s[i] == ']') { i++; return l; } + while (true) + { + l.Add(ReadValue(s, ref i)); + Skip(s, ref i); + if (s[i] == ',') { i++; continue; } + if (s[i] == ']') { i++; return l; } + throw new FormatException("expected ',' or ']'"); + } + } + if (c == '"') return ReadString(s, ref i); + if (s.Substring(i).StartsWith("true")) { i += 4; return true; } + if (s.Substring(i).StartsWith("false")) { i += 5; return false; } + if (s.Substring(i).StartsWith("null")) { i += 4; return null; } + int start = i; + while (i < s.Length && "+-0123456789.eE".IndexOf(s[i]) >= 0) i++; + return double.Parse(s.Substring(start, i - start), CultureInfo.InvariantCulture); + } + + static string ReadString(string s, ref int i) + { + if (s[i] != '"') throw new FormatException("expected string"); + var sb = new StringBuilder(); i++; + while (s[i] != '"') + { + if (s[i] == '\\') + { + i++; + char e = s[i]; + if (e == 'u') { sb.Append((char)Convert.ToInt32(s.Substring(i + 1, 4), 16)); i += 4; } + else sb.Append(e == 'n' ? '\n' : e == 't' ? '\t' : e == 'r' ? '\r' : e == 'b' ? '\b' : e == 'f' ? '\f' : e); + } + else sb.Append(s[i]); + i++; + } + i++; + return sb.ToString(); + } + + public static string Serialize(object v) + { + var sb = new StringBuilder(); + Write(sb, v); + return sb.ToString(); + } + + static void Write(StringBuilder sb, object v) + { + switch (v) + { + case null: sb.Append("null"); break; + case string s: WriteString(sb, s); break; + case bool b: sb.Append(b ? "true" : "false"); break; + case double d: sb.Append(FormatNumber(d)); break; + case float f: sb.Append(FormatNumber(f)); break; + case int n: sb.Append(n.ToString(CultureInfo.InvariantCulture)); break; + case IDictionary d: + sb.Append('{'); bool first = true; + foreach (var kv in d) { if (!first) sb.Append(','); first = false; WriteString(sb, kv.Key); sb.Append(':'); Write(sb, kv.Value); } + sb.Append('}'); break; + case IEnumerable e: + sb.Append('['); bool f1 = true; + foreach (var x in e) { if (!f1) sb.Append(','); f1 = false; Write(sb, x); } + sb.Append(']'); break; + default: WriteString(sb, v.ToString()); break; + } + } + + // Python's json.dumps prints integral floats as "2.0"; keep the same shape as the training data. + public static string FormatNumber(double d) + { + string s = d.ToString("R", CultureInfo.InvariantCulture); + return s.Contains(".") || s.Contains("E") ? s : s + ".0"; + } + + static void WriteString(StringBuilder sb, string s) + { + sb.Append('"'); + foreach (char c in s) + { + if (c == '"') sb.Append("\\\""); + else if (c == '\\') sb.Append("\\\\"); + else if (c == '\n') sb.Append("\\n"); + else if (c < ' ') sb.Append("\\u").Append(((int)c).ToString("x4")); + else sb.Append(c); + } + sb.Append('"'); + } + } +} diff --git a/Runtime/MiniJson.cs.meta b/Runtime/MiniJson.cs.meta new file mode 100644 index 0000000000000000000000000000000000000000..87cc7153871f95a61ed4490478a26d3227aebaa3 --- /dev/null +++ b/Runtime/MiniJson.cs.meta @@ -0,0 +1,11 @@ +fileFormatVersion: 2 +guid: 61e217efc401cb446921912c87d7770d +MonoImporter: + externalObjects: {} + serializedVersion: 2 + defaultReferences: [] + executionOrder: 0 + icon: {instanceID: 0} + userData: + assetBundleName: + assetBundleVariant: diff --git a/Runtime/PyMath.cs b/Runtime/PyMath.cs new file mode 100644 index 0000000000000000000000000000000000000000..861151dc9b74f02e8f4ec96a19e010ea5b0f2453 --- /dev/null +++ b/Runtime/PyMath.cs @@ -0,0 +1,35 @@ +// Python's round(x, n) and f"{x:.nf}" in C#, so the scene text and the tool effects match the training data byte for +// byte. Python rounds the exact binary value of x, ties to even; .NET's Math.Round(x, n) scales by 10^n first, so 0.45 +// (binary 0.45000000000000001...) became 0.4 where the training scenes say 0.5, and -0.04 became 0.0 where they say +// -0.0 (v2 Unity reference on the v3 sample: 81 + 37 of 440 scenes differed from the offline prompt). +using System; +using System.Globalization; +using System.Numerics; + +namespace SceneAgent +{ + public static class PyMath + { + /// Python round(x, n) for n >= 0: exact, ties to even, sign kept (round(-0.04, 1) == -0.0). + public static double Round(double x, int n) + { + if (double.IsNaN(x) || double.IsInfinity(x)) return x; + long bits = BitConverter.DoubleToInt64Bits(x); + bool neg = bits < 0; + int e = (int)((bits >> 52) & 0x7FF); + long m = bits & 0xFFFFFFFFFFFFFL; + if (e == 0) e = 1; else m |= 1L << 52; + e -= 1075; // |x| = m * 2^e exactly + BigInteger num = new BigInteger(m) * BigInteger.Pow(10, n), den = BigInteger.One; + if (e > 0) num <<= e; else den <<= -e; + BigInteger q = BigInteger.DivRem(num, den, out BigInteger r); + int c = (r * 2).CompareTo(den); + if (c > 0 || (c == 0 && !q.IsEven)) q += 1; + double v = (double)q / Math.Pow(10, n); // q and 10^n are exact doubles: the division is correctly rounded + return neg ? -v : v; + } + + /// Python f"{x:.{n}f}". + public static string Fixed(double x, int n) => Round(x, n).ToString("F" + n, CultureInfo.InvariantCulture); + } +} diff --git a/Runtime/PyMath.cs.meta b/Runtime/PyMath.cs.meta new file mode 100644 index 0000000000000000000000000000000000000000..4606aa538415ab12a5f80b66201b095a555c92f6 --- /dev/null +++ b/Runtime/PyMath.cs.meta @@ -0,0 +1,11 @@ +fileFormatVersion: 2 +guid: aa67273f40b6679448917c8a06ebe741 +MonoImporter: + externalObjects: {} + serializedVersion: 2 + defaultReferences: [] + executionOrder: 0 + icon: {instanceID: 0} + userData: + assetBundleName: + assetBundleVariant: diff --git a/Runtime/Resources.meta b/Runtime/Resources.meta new file mode 100644 index 0000000000000000000000000000000000000000..96bae1d66df9d1e3f70f3dac068407e87c888860 --- /dev/null +++ b/Runtime/Resources.meta @@ -0,0 +1,8 @@ +fileFormatVersion: 2 +guid: 4e30c3b99c50b4d499baeb3c87541721 +folderAsset: yes +DefaultImporter: + externalObjects: {} + userData: + assetBundleName: + assetBundleVariant: diff --git a/Runtime/Resources/SceneAgent.meta b/Runtime/Resources/SceneAgent.meta new file mode 100644 index 0000000000000000000000000000000000000000..24766dba69cecd7c2d4123d70870abeda0efcd29 --- /dev/null +++ b/Runtime/Resources/SceneAgent.meta @@ -0,0 +1,8 @@ +fileFormatVersion: 2 +guid: 06871ae947abbc942a04f192fb97ac7f +folderAsset: yes +DefaultImporter: + externalObjects: {} + userData: + assetBundleName: + assetBundleVariant: diff --git a/Runtime/Resources/SceneAgent/system_prompt_v3.txt b/Runtime/Resources/SceneAgent/system_prompt_v3.txt new file mode 100644 index 0000000000000000000000000000000000000000..7fe92da14e05f0bed21f407df34581670a1fb13f --- /dev/null +++ b/Runtime/Resources/SceneAgent/system_prompt_v3.txt @@ -0,0 +1,16 @@ +You are an assistant embedded in a 3D Unity scene. Turn the user's request into tool calls that use object ids exactly as listed in the scene; never invent an id or a tool. Only objects in the user's current room are listed. Match objects by meaning: a synonym or a more general word can name a listed object, and details the scene does not record (a colour when no colours are listed, or words like sliced, wooden, empty) do not rule an object out. If exactly one listed object fits, act on it. If several fit equally well, call ask_clarification with their ids. If none fits, use find_objects to look in the other rooms. place moves an object directly, so there is no need to grab it first. If a request is impossible or needs a tool you do not have, call no tool and briefly explain why. Answer questions about the scene in plain text. + +# Tools +grab(object_id:str, hand:left|right|both) +release(hand:left|right|both) +place(object_id:str, target_id:str, relation?:on|in, orientation?:default|upright|horizontal|vertical|upside_down) +move_by(object_id:str, direction:left|right|forward|backward|up|down, distance:num, unit:m|cm) +rotate(object_id:str, degrees:num, axis?:yaw|pitch|roll) +set_state(object_id:str, state:on|off|open|closed|half_open|locked|unlocked) +press(object_id:str, hold_seconds?:num) +highlight(object_ids:list) +teleport(location_id:str) +find_objects(label:str, color?:str, room?:str) +ask_clarification(question:str, candidate_ids:list) + +Reply with one {"name": ..., "arguments": {...}} block per call, or plain text. \ No newline at end of file diff --git a/Runtime/Resources/SceneAgent/system_prompt_v3.txt.meta b/Runtime/Resources/SceneAgent/system_prompt_v3.txt.meta new file mode 100644 index 0000000000000000000000000000000000000000..49d4455d80073d34e9d2e62dddc102574884f9b7 --- /dev/null +++ b/Runtime/Resources/SceneAgent/system_prompt_v3.txt.meta @@ -0,0 +1,7 @@ +fileFormatVersion: 2 +guid: 6a38ee93718310f469efc52b22f6a1b9 +TextScriptImporter: + externalObjects: {} + userData: + assetBundleName: + assetBundleVariant: diff --git a/Runtime/SceneAgent.Runtime.asmdef b/Runtime/SceneAgent.Runtime.asmdef new file mode 100644 index 0000000000000000000000000000000000000000..ac0472249a5117c362350f4a1958f9a9905119d7 --- /dev/null +++ b/Runtime/SceneAgent.Runtime.asmdef @@ -0,0 +1,6 @@ +{ + "name": "SceneAgent.Runtime", + "rootNamespace": "SceneAgent", + "references": [], + "autoReferenced": true +} diff --git a/Runtime/SceneAgent.Runtime.asmdef.meta b/Runtime/SceneAgent.Runtime.asmdef.meta new file mode 100644 index 0000000000000000000000000000000000000000..5ac347d4fbdd74d1cfbfcf1e49b706c868424bc3 --- /dev/null +++ b/Runtime/SceneAgent.Runtime.asmdef.meta @@ -0,0 +1,7 @@ +fileFormatVersion: 2 +guid: 9fc737831bcf4da4e9b861e32b778d92 +AssemblyDefinitionImporter: + externalObjects: {} + userData: + assetBundleName: + assetBundleVariant: diff --git a/Runtime/SceneAgent.cs b/Runtime/SceneAgent.cs new file mode 100644 index 0000000000000000000000000000000000000000..3352b582483450ce154dc64725434d550cba2ae0 --- /dev/null +++ b/Runtime/SceneAgent.cs @@ -0,0 +1,255 @@ +// Talks to a local OpenAI-compatible server (llama.cpp llama-server with the fine-tuned GGUF, +// Unsloth Studio's /v1 endpoint, Ollama, ...), executes the returned tool calls on the SceneWorld +// and feeds tool results back until the model answers in text, asks the user a clarifying question +// (ask_clarification hands the turn back: the answer arrives as the next request), or runs out of turns. +// +// The loop is a coroutine: HTTP runs on a worker thread, tool calls run on the main thread +// (Unity APIs are main-thread only). Play mode: StartCoroutine(agent.Run(...)). +// Batch/editor code can drive it synchronously with RunBlocking(...). +using System; +using System.Collections; +using System.Collections.Generic; +using System.Globalization; +using System.Linq; +using System.Net.Http; +using System.Text; +using System.Text.RegularExpressions; +using System.Threading; +using System.Threading.Tasks; +using UnityEngine; + +namespace SceneAgent +{ + [Serializable] + public class AgentTurn + { + public string modelOutput; + public List calls = new List(); + public List results = new List(); + /// D5b only: the verdict on the free output (SceneGrammar.Check) and what was done about it. + public string d5b; + } + + public class AgentResult + { + public List turns = new List(); + public string finalText; + /// Set when the model asked the user a question (ask_clarification); the turn ends there. + public string clarificationQuestion; + public string error; + public double seconds; + } + + public class SceneAgent : MonoBehaviour + { + public string endpoint = "http://127.0.0.1:8081/v1/chat/completions"; + public string model = "unity-scene-agent"; + [Tooltip("The model's system prompt. Empty: the one shipped with the package (Resources/SceneAgent/system_prompt_v3), " + + "the exact prompt the v3 model was trained and evaluated with. Change it only with a model trained on yours.")] + public TextAsset systemPromptAsset; + [Tooltip("Optional: the LocalModelServer that runs the model. Run() then waits until it is ready and uses its endpoint.")] + public LocalModelServer server; + [NonSerialized] public string systemPrompt; + public int maxTurns = 4; + public SceneWorld world; + [Tooltip("Strict D5: constrain decoding to the 11 tools, ids that exist and the trained call format (SceneGrammar.cs). " + + "Needs llama-server. The shipped v3 setting is D5b below, not this.")] + public bool constrainedDecoding = false; + [Tooltip("D5b (the shipped v3 setting): the model answers freely; SceneGrammar.Check then turns a call to a tool or " + + "value outside the schema into a refusal and generates an unknown id or a broken call again under the " + + "grammar. Used when constrainedDecoding is off.")] + public bool fallbackDecoding = true; + [Tooltip("Safety: when the request names an object that is only in another room and the model acts on a different " + + "object here instead, that call is not executed; the model is told where the named object is. (Seen in the " + + "user test: 'put the drill on the workbench' with the drill in another room moved the screwdriver.)")] + public bool guardSubstitutes = true; + + static readonly HttpClient Http = new HttpClient { Timeout = TimeSpan.FromSeconds(120) }; + static readonly Regex ToolCallRe = new Regex(@"\s*(.*?)\s*", RegexOptions.Singleline); + static readonly Regex ThinkRe = new Regex(@".*?\s*", RegexOptions.Singleline); + // Same rule as eval/toolcall.py HANDOFF: these calls end the agent turn. + static readonly HashSet Handoff = new HashSet { "ask_clarification" }; + + static string defaultPrompt; + static string DefaultPrompt => defaultPrompt ??= Resources.Load("SceneAgent/system_prompt_v3")?.text.Replace("\r\n", "\n"); + // no ?. on systemPromptAsset: in the Editor an unassigned field is a "fake null" Unity object, and ?. would call + // .text on it (UnassignedReferenceException on the first command in Play mode; found by the visible user test) + string Prompt => systemPrompt ?? (systemPromptAsset != null ? systemPromptAsset.text.Replace("\r\n", "\n") : null) ?? DefaultPrompt + ?? throw new InvalidOperationException("no system prompt (the package's Resources/SceneAgent/system_prompt_v3 is missing)"); + + public IEnumerator Run(string utterance, Action done) + { + var t0 = DateTime.UtcNow; + var res = new AgentResult(); + if (server != null) + { + while (!server.Ready && !server.Failed) yield return null; + if (server.Failed) + { + res.error = "model server not available: " + server.status; + res.seconds = (DateTime.UtcNow - t0).TotalSeconds; + done(res); + yield break; + } + endpoint = server.Endpoint; + } + string sceneJson = world.BuildSceneJson(); + var messages = new List { + Msg("system", Prompt), + Msg("user", "Scene: " + sceneJson + "\nRequest: " + utterance) }; + var toolResults = new List(); // ids returned here become nameable (grammar) + for (int turn = 0; turn < maxTurns && res.error == null; turn++) + { + var req = new Dictionary { + { "model", model }, { "messages", messages }, { "temperature", 0.0 }, { "max_tokens", 192 } }; + if (constrainedDecoding) req["grammar"] = SceneGrammar.Build(sceneJson, toolResults); + string body = MiniJson.Serialize(req); + var task = Task.Run(() => Post(body)); // worker thread: no Unity API inside + while (!task.IsCompleted) yield return null; + if (task.IsFaulted) { res.error = task.Exception?.GetBaseException().Message; break; } + string output = task.Result, d5b = null; + if (!constrainedDecoding && fallbackDecoding) + { + d5b = SceneGrammar.Check(output, sceneJson, toolResults); + if (d5b == "invalid_tool" || d5b == "invalid_value") output = SceneGrammar.Refusal; + else if (d5b == "invalid_id" || d5b == "malformed") + { + req["grammar"] = SceneGrammar.Build(sceneJson, toolResults); + string again = MiniJson.Serialize(req); + var retry = Task.Run(() => Post(again)); + while (!retry.IsCompleted) yield return null; + if (retry.IsFaulted) { res.error = retry.Exception?.GetBaseException().Message; break; } + output = retry.Result; + } + } + var at = new AgentTurn { modelOutput = output, d5b = d5b }; + res.turns.Add(at); + var blocks = ToolCallRe.Matches(output).Cast().Select(m => m.Groups[1].Value).ToList(); + if (blocks.Count == 0) + { + res.finalText = Clean(output); + break; + } + var toolMsgs = new List(); + string question = null; + foreach (var block in blocks) + { + Dictionary c; + try { c = (Dictionary)MiniJson.Parse(block); } + catch (Exception e) { res.error = "unparseable tool call: " + e.Message; break; } + string name = Convert.ToString(c.TryGetValue("name", out var n) ? n : ""); + var args = c.TryGetValue("arguments", out var a) && a is Dictionary d ? d : new Dictionary(); + object result; + string guard = SubstituteGuard(utterance, name, args); + if (guard != null) + { + Debug.Log("[SceneAgent] call not executed: " + guard); + result = new Dictionary { { "ok", false }, { "error", guard } }; + } + else + { + try { result = world.Execute(name, args); } + catch (ToolError e) { result = new Dictionary { { "ok", false }, { "error", e.Message } }; } + } + if (Handoff.Contains(name)) + question = Convert.ToString(args.TryGetValue("question", out var q) ? q : "") ?? ""; + string resultJson = PyJson(result); + at.calls.Add(block); + at.results.Add(resultJson); + toolResults.Add(resultJson); + toolMsgs.Add(new Dictionary { { "role", "tool" }, { "name", name }, { "content", resultJson } }); + } + // history: the model's own call text, then tool results (Qwen3 renders them as ) + messages.Add(Msg("assistant", Clean(output))); + messages.AddRange(toolMsgs); + if (question != null) { res.clarificationQuestion = question; break; } + } + res.seconds = (DateTime.UtcNow - t0).TotalSeconds; + done(res); + } + + /// Synchronous driver for editor/batch code (main thread). + public AgentResult RunBlocking(string utterance) + { + AgentResult result = null; + var it = Run(utterance, r => result = r); + while (it.MoveNext()) Thread.Sleep(2); + return result; + } + + static readonly HashSet ActingTools = new HashSet { "grab", "place", "move_by", "rotate", "set_state", "press" }; + + /// Why this call must not run (see guardSubstitutes), or null. It fires only when the acted-on object is + /// not named in the request while the request names an object that exists only in other rooms. + public string SubstituteGuard(string utterance, string tool, Dictionary args) + { + if (!guardSubstitutes || world == null || !ActingTools.Contains(tool ?? "")) return null; + string id = args.TryGetValue("object_id", out var v) && v != null ? Convert.ToString(v, CultureInfo.InvariantCulture) : null; + var item = id == null ? null : world.Items.FirstOrDefault(x => x.itemId == id); + if (item == null) return null; + if (string.IsNullOrEmpty(item.label)) return null; + // the same rules as tests/acceptance/guard_offline.py, which measured them on every call the four v3 models + // made on the sealed and dev sets (13,438 calls): 4 firings, all on wrong actions, no false alarm + string text = " " + Regex.Replace(utterance.ToLowerInvariant(), "[^a-z0-9]+", " ") + " "; + var words = text.Split(new[] { ' ' }, StringSplitOptions.RemoveEmptyEntries); + string joined = string.Concat(words); + IEnumerable WordsOf(string label) => label.Split(' ').Where(w => w.Length >= 3); + bool Phrase(string label) => text.Contains(" " + label + " ") || text.Contains(" " + label + "s ") || text.Contains(" " + label + "es "); + // the acted-on object: any sign the user meant it ("box" -> toolbox, "note book" -> notebook, "keys" -> key chain) + bool Loosely(SceneItem o) + { + string h = o.label.Split(' ').Last(); + return Phrase(o.label) || WordsOf(o.label).Any(w => words.Contains(w) || words.Contains(w + "s") || words.Contains(w + "es")) + || words.Any(w => w.Length >= 3 && h.EndsWith(w)) || joined.Contains(o.label.Replace(" ", "")); + } + if (Loosely(item)) return null; + // an object elsewhere counts only when named by its full label, and only if nothing here shares a word with it + var hereWords = new HashSet(world.Items.Where(y => y.room == world.playerRoom && !string.IsNullOrEmpty(y.label)) + .SelectMany(y => WordsOf(y.label))); + var elsewhere = world.Items.FirstOrDefault(x => x.room != world.playerRoom && !string.IsNullOrEmpty(x.label) + && Phrase(x.label) && !WordsOf(x.label).Any(hereWords.Contains)); + if (elsewhere == null) return null; + return $"not done: '{id}' is a {item.label}, but the request names a {elsewhere.label}; " + + $"the {elsewhere.label} ({elsewhere.itemId}) is in the {elsewhere.room}, not in this room"; + } + + static string Clean(string s) => ThinkRe.Replace(s, "").Replace("<|im_end|>", "").Trim(); + + static Dictionary Msg(string role, string content) => + new Dictionary { { "role", role }, { "content", content } }; + + string Post(string body) + { + var resp = Http.PostAsync(endpoint, new StringContent(body, Encoding.UTF8, "application/json")).ConfigureAwait(false).GetAwaiter().GetResult(); + string text = resp.Content.ReadAsStringAsync().ConfigureAwait(false).GetAwaiter().GetResult(); + if (!resp.IsSuccessStatusCode) + { + var m = Regex.Match(text, @"request \((\d+) tokens\) exceeds the available context size \((\d+) tokens\)"); + if (m.Success) + throw new Exception($"the scene and request need {m.Groups[1].Value} tokens but the model's context holds " + + $"{m.Groups[2].Value}: put fewer objects in this room, or raise LocalModelServer.contextSize"); + throw new Exception($"HTTP {(int)resp.StatusCode}: {text}"); + } + var root = (Dictionary)MiniJson.Parse(text); + var msg = (Dictionary)((Dictionary)((List)root["choices"])[0])["message"]; + return Convert.ToString(msg.TryGetValue("content", out var c) ? c : "") ?? ""; + } + + // Tool results in training were produced by Python's json.dumps (", " and ": " separators). + public static string PyJson(object v) + { + switch (v) + { + case null: return "null"; + case bool b: return b ? "true" : "false"; + case string s: return MiniJson.Serialize(s); + case double d: return MiniJson.FormatNumber(d); + case IDictionary dict: + return "{" + string.Join(", ", dict.Select(kv => MiniJson.Serialize(kv.Key) + ": " + PyJson(kv.Value))) + "}"; + case IEnumerable e: + return "[" + string.Join(", ", e.Cast().Select(PyJson)) + "]"; + default: return MiniJson.Serialize(Convert.ToString(v, CultureInfo.InvariantCulture)); + } + } + } +} diff --git a/Runtime/SceneAgent.cs.meta b/Runtime/SceneAgent.cs.meta new file mode 100644 index 0000000000000000000000000000000000000000..15eb0bd49edf156f810c2d8dbce8000a7729be6a --- /dev/null +++ b/Runtime/SceneAgent.cs.meta @@ -0,0 +1,11 @@ +fileFormatVersion: 2 +guid: e77941abfd358644fb748664149b43d0 +MonoImporter: + externalObjects: {} + serializedVersion: 2 + defaultReferences: [] + executionOrder: 0 + icon: {instanceID: 0} + userData: + assetBundleName: + assetBundleVariant: diff --git a/Runtime/SceneAgentChatBox.cs b/Runtime/SceneAgentChatBox.cs new file mode 100644 index 0000000000000000000000000000000000000000..691f5d2ef533190e36434dee356b2a7244bf7e88 --- /dev/null +++ b/Runtime/SceneAgentChatBox.cs @@ -0,0 +1,99 @@ +// A minimal on-screen chat for trying the agent without writing any UI (flat screen; for XR, drive SceneAgentSession +// from your own UI or speech input). Type a command, press Enter; when the model asks which object you mean, click it. +// F1 hides / shows the box. +using System.Collections.Generic; +using System.Linq; +using UnityEngine; + +namespace SceneAgent +{ + [RequireComponent(typeof(SceneAgentSession))] + public class SceneAgentChatBox : MonoBehaviour + { + public int visibleLines = 12; + public KeyCode toggleKey = KeyCode.F1; + [Tooltip("Commands offered as one-click examples while the box is empty.")] + public string[] examples = { "put the red mug on the table", "turn on the lamp", "where is the book?" }; + + SceneAgentSession session; + LocalModelServer server; + readonly List lines = new List(); + string input = ""; + bool visible = true; + Vector2 scroll; + GUIStyle wrap; + + void Awake() + { + session = GetComponent(); + session.Line += l => { lines.Add(l); scroll.y = float.MaxValue; }; + } + + void Start() // after every Awake: the session has found its agent by now + { + // no ?? here: Unity objects compare to null through their own operator + server = session.agent != null ? session.agent.server : null; + if (server == null) server = GetComponent(); + if (server == null) server = FindObjectOfType(); + } + + void Send(string text) + { + text = text?.Trim(); + if (string.IsNullOrEmpty(text) || session.Busy) return; + input = ""; + StartCoroutine(session.Command(text)); + } + + void OnGUI() + { + // the toggle through IMGUI events: works with the old Input Manager and with the Input System package + if (Event.current.type == EventType.KeyDown && Event.current.keyCode == toggleKey) { visible = !visible; Event.current.Use(); } + if (!visible) return; + wrap ??= new GUIStyle(GUI.skin.label) { wordWrap = true, richText = false }; + float w = Mathf.Min(Screen.width - 20, 900), h = 60 + visibleLines * 20; + var area = new Rect(10, Screen.height - h - 10, w, h); + GUI.Box(area, ""); + GUILayout.BeginArea(new Rect(area.x + 8, area.y + 6, area.width - 16, area.height - 12)); + + string st = server == null ? "no LocalModelServer: using " + session.agent.endpoint + : server.Ready ? server.status : server.Failed ? "⚠ " + server.status : server.status; + string gaze = string.IsNullOrEmpty(session.world.gaze) ? "" : $" · pointing at: {session.world.gaze}"; + GUILayout.Label($"Scene Agent · room: {session.world.playerRoom}{gaze} · {st}{(session.Busy ? " · thinking…" : "")}"); + + scroll = GUILayout.BeginScrollView(scroll, GUILayout.Height(visibleLines * 20)); + foreach (var l in lines.Skip(Mathf.Max(0, lines.Count - 200))) GUILayout.Label(l, wrap); + if (lines.Count == 0) + { + GUILayout.Label("Type a command in English, e.g.:"); + GUILayout.BeginHorizontal(); + foreach (var ex in examples) if (GUILayout.Button(ex)) input = ex; + GUILayout.EndHorizontal(); + } + GUILayout.EndScrollView(); + + if (session.PendingRequest != null && !session.Busy) + { + GUILayout.BeginHorizontal(); + GUILayout.Label("Click one, or type e.g. 'the red one':", GUILayout.Width(240)); + foreach (var id in session.Candidates) + { + var it = session.world.Items.FirstOrDefault(x => x.itemId == id); + if (it != null && GUILayout.Button($"{session.Describe(it)} ({id})")) StartCoroutine(session.Answer(id)); + } + GUILayout.EndHorizontal(); + } + + GUILayout.BeginHorizontal(); + GUI.SetNextControlName("SceneAgentInput"); + bool enter = Event.current.type == EventType.KeyDown && Event.current.keyCode == KeyCode.Return + && GUI.GetNameOfFocusedControl() == "SceneAgentInput"; + input = GUILayout.TextField(input, GUILayout.ExpandWidth(true)); + GUI.enabled = !session.Busy; + if (GUILayout.Button("Send", GUILayout.Width(70)) || enter) Send(input); + GUI.enabled = true; + GUILayout.EndHorizontal(); + GUILayout.EndArea(); + } + } +} diff --git a/Runtime/SceneAgentChatBox.cs.meta b/Runtime/SceneAgentChatBox.cs.meta new file mode 100644 index 0000000000000000000000000000000000000000..1ddfd12f6c9af6a7e1adb541a871961fa116852a --- /dev/null +++ b/Runtime/SceneAgentChatBox.cs.meta @@ -0,0 +1,11 @@ +fileFormatVersion: 2 +guid: 21283d1231b900f4fb5a1d5c6462ace9 +MonoImporter: + externalObjects: {} + serializedVersion: 2 + defaultReferences: [] + executionOrder: 0 + icon: {instanceID: 0} + userData: + assetBundleName: + assetBundleVariant: diff --git a/Runtime/SceneAgentGaze.cs b/Runtime/SceneAgentGaze.cs new file mode 100644 index 0000000000000000000000000000000000000000..064baec60754d3404581621a8c63a4d2e97ffd1d --- /dev/null +++ b/Runtime/SceneAgentGaze.cs @@ -0,0 +1,47 @@ +// Sets SceneWorld.gaze to the SceneItem the user points at, so "this" and "that" resolve: the object under the mouse +// pointer (flat screen) or at the centre of the camera (head gaze in XR). The objects need colliders. While the agent is +// answering, the gaze is left as it was, so the command sees what the user pointed at when they asked. +using System; +using UnityEngine; + +namespace SceneAgent +{ + public class SceneAgentGaze : MonoBehaviour + { + public enum Source { MousePointer, CameraCentre } + + public SceneWorld world; + public SceneAgentSession session; + [Tooltip("Empty: Camera.main.")] + public Camera viewCamera; + [Tooltip("MousePointer falls back to the camera centre when the old Input Manager is off (Input System only).")] + public Source source = Source.MousePointer; + public float maxDistance = 30f; + + /// The id the user points at now, or null. + public string Current { get; private set; } + + void Update() + { + if (world == null) world = FindObjectOfType(); + var cam = viewCamera != null ? viewCamera : Camera.main; + if (world == null || cam == null || (session != null && session.Busy)) return; + Ray ray = source == Source.MousePointer && TryMouse(out var p) + ? cam.ScreenPointToRay(p) : cam.ViewportPointToRay(new Vector3(0.5f, 0.5f, 0f)); + string id = null; + if (Physics.Raycast(ray, out var hit, maxDistance)) + { + var it = hit.collider.GetComponentInParent(); + if (it != null && it.room == world.playerRoom) id = it.itemId; + } + Current = id; + world.gaze = id; + } + + static bool TryMouse(out Vector3 p) + { + try { p = Input.mousePosition; return true; } + catch (InvalidOperationException) { p = default; return false; } // Input System only + } + } +} diff --git a/Runtime/SceneAgentGaze.cs.meta b/Runtime/SceneAgentGaze.cs.meta new file mode 100644 index 0000000000000000000000000000000000000000..1cf0d1b8edbac6ac22d3adf37858c9a56652148c --- /dev/null +++ b/Runtime/SceneAgentGaze.cs.meta @@ -0,0 +1,11 @@ +fileFormatVersion: 2 +guid: 624864f83fffffc46adb473c4cb15787 +MonoImporter: + externalObjects: {} + serializedVersion: 2 + defaultReferences: [] + executionOrder: 0 + icon: {instanceID: 0} + userData: + assetBundleName: + assetBundleVariant: diff --git a/Runtime/SceneAgentScriptRunner.cs b/Runtime/SceneAgentScriptRunner.cs new file mode 100644 index 0000000000000000000000000000000000000000..07debb52d759f1a630d26149809161585d38048d --- /dev/null +++ b/Runtime/SceneAgentScriptRunner.cs @@ -0,0 +1,114 @@ +// Plays a list of commands through the agent and writes a report: for QA, regression tests and demos. +// one command per line; "// ..." comments; "#answer " answers the pending question; "#look |none" +// sets the gaze; "#wait " pauses (not counted in any step's time). +// In the Editor: assign a TextAsset and press Play. In a built game or batch run: +// Game.exe -sceneAgentScript commands.txt [-sceneAgentOut ] [-sceneAgentShots] (quits when done) +// Report: /report.json with every step's calls, results, answer, error, seconds and the scene signature. +using System; +using System.Collections; +using System.Collections.Generic; +using System.Globalization; +using System.IO; +using System.Linq; +using UnityEngine; + +namespace SceneAgent +{ + [RequireComponent(typeof(SceneAgentSession))] + public class SceneAgentScriptRunner : MonoBehaviour + { + public TextAsset script; + [Tooltip("Output folder (relative = under Application.persistentDataPath). Empty: SceneAgentRuns/