# VisualLearningIdea: v1 spec Supersedes `visuallearning-handover.html`, which drifted over several HuggingChat sessions. `cat-sun-scene.html` stays as the UX reference for the serial reading. ## Idea 3D is a language. The Space **translates English into 3D**, the way you would translate English into Japanese. A picture is easier to take in than text, because the reader doesn't have to guess the exact picture. Once 3D exists it can be materialized (papercraft, origami, …). The **simplification ladder** shows how far a mesh can be reduced and still mean its word, and that is a main output. One Gradio Space serves people (UI) and models (`demo.launch(mcp_server=True)`). HuggingChat models can't run tools, which is why this Space exists. **The Space visualises; discussion happens in the chat** (e.g. HuggingChat over MCP). The calling model already sees the picture, so the Space runs no vision model of its own and stores no token. Because HTML is the final destination, every picture comes back with a **mini HTML page** the model can read, edit and build on. ## Translation terms | Translation | Here | |---|---| | Source → target language | English → 3D | | Dictionary | Ready assets: SimReady → Objaverse | | Grammar words | `the / a / and`: no geometry; a determiner opens a "which?" question | | Verbs | Relations between meshes | | Word alignment | `tokens[].commits`: reading order differs from build order | | Forced choices | `unresolved`: which asset, how big | | Back-translation | The chat model reading the picture: done in the discussion, not by the Space | ## Dictionary - Tiers, each taken **on rejection**: `simready → objaverse → primitive → generation`. - **Available** = a SimReady or Objaverse asset exists (no construction). No asset means not available, with no exceptions. The UI offers only available words, so the user knows before submitting. - A missing word can be used only by **explicitly choosing a primitive** for it (`box, sphere, cylinder, cone, plane`). In the UI the user picks one. Over MCP, `check_words` returns `missing` and the user's model (e.g. in HuggingChat) decides: rephrase, or pass `primitives_json` such as `{"sun": "sphere"}`. - Assets that **can't be used** are skipped and recorded in `decision.skipped`: no license, NC (non-commercial) or ND (no derivatives; simplification is a derivative), or no geometry. - SimReady: package list from `NVIDIA-Omniverse/simready-central/submissions/*`. Only geometry USD + thumbnail are downloaded, because textures are ~98% of each ~1 GB package. Assets are metric, so height comes from the asset. - Objaverse: LVIS categories (`lvis-annotations.json.gz`) are the word index; the license comes from `metadata/.json.gz`. No real scale, so height is a nominal default (an `unresolved` decision the user can set). - Light comes from the verb, not from a special word: whatever "shone" becomes the light source and sits in the sky along the light direction. - When nothing shines, the picture still needs light: seeing needs light as sitting needs a ground. The scene gets soft daylight from no object (`sun.daylight: true`, seeded direction, shadows on) and `unresolved` asks "what light?" (default: daylight; rephrase with " shone/glowed" to make an object the light). - SimReady geometry comes without textures, so the colour is the mean of the asset's 256 px thumbnail (backdrop and transparent pixels left out), stored as `item.color`. `item.thumbnail` is a public dataset URL, so the chat can show and compare assets. Uncoloured meshes render neutral light grey. - A "which ?" question carries `on_reject`: what rejecting the tier would offer (e.g. `{"tier": "objaverse", "candidates": 101, "offered": 9}`), since those alternatives are invisible until the tier is rejected. - The ground implied by "sat" (with no stated object) is a support plane created by the relation. It is not a typed word, so it needs no asset. To exist, a thing needs a reality to exist in: reading skips this, but visualising exposes it. So the implied ground is also surfaced in `unresolved` ("cat sat on what?", default: ground plane), for the discussion to accept or rephrase. ## Serial side vs configuration side - **Serial (ledger)**: tokens in reading order, items each word commits, relations, unresolved questions. **Freezing lives here.** Frozen items can't change; the others can be swapped, rejected, resized or rerolled. - **Configuration**: once the serial side is resolved, any placement satisfying the relations is valid. `seed` picks one (positions, yaw, sun direction). ### Ledger shape ```jsonc { "sentence": "the cat sat and the sun shone", "tokens": [{"index": 2, "text": "sat", "kind": "verb", "commits": ["ground", "r2"], "opens": []}], "items": [ {"id": "n1", "kind": "object", "word": "cat", "token": 1, "frozen": false, "decision": {"tier": "objaverse", "chosen": {…}, "alternatives": […], "rejected_tiers": [], "skipped": [], "label": "cat"}, "height_m": {"value": 0.3, "source": "default"}, "license": "by"}, {"id": "ground", "kind": "environment", "role": "plane", "implied_by": 2}, {"id": "r2", "kind": "relation", "relation": "on", "subject": "n1", "object": "ground"}, {"id": "n5", "kind": "object", "word": "sun", "emits_light": true, "decision": {"tier": "primitive", "chosen": {"source": "primitive", "shape": "sphere"}, …}}, {"id": "r6", "kind": "relation", "relation": "lights", "subject": "n5", "object": "scene"} ], "unresolved": [{"id": "q0", "about": "n1", "question": "which cat?", "default_chosen": "cat", "alternatives": […]}], "configuration": {"seed": 0, "placements": {"n1": {"position": [0, 0, 0], "yaw_deg": 12}}, "sun": {…}} } ``` Relations: `on, under, beside, near, behind, in_front_of, above, faces, lights`. Verbs: `sat/stood/lay/rested` → `on` (ground implied), `shone/glowed` → `lights` (shadows only turn on with this coupling), `faced` → `faces`. ## Paragraphs: sentences + bindings `extend_scene` adds the next sentence to an existing ledger (token and item ids carry on, `sentences` lists them). Within a sentence or across sentences: - **"the "** when that noun is already in the ledger refers to it (the most recent one) instead of adding an item. Surfaced as an implied question: "which cat? the one already mentioned" (default: the same cat), alternatives include "say 'another cat' for a new one". - **"a/an/another "** always adds a new item. - **Pronouns** (it, he, she, they, him, her, them) refer back by a rule, `pronouns` on `translate_to_scene`/`extend_scene`, kept in the ledger: - `"recent"` (default): the most recent object. - `"subject"`: the subject of the most recent clause, else the most recent object. Closer to how readers take "it": in "the cat sat on the table. the apple lay beside it", `recent` gives the table, `subject` the cat. Either way the clause's own subject is left out when the pronoun is its object ("the apple lay beside it" can't mean the apple), and the choice is surfaced as "'it' = the cat?" with the other objects as alternatives, most recent first. A pronoun with nothing before it is an error. - Each reference question carries `binds: {token, to}`; relations record which token named each end (`tokens: {subject, object}`), so a reference can be rebound. - **`bind`** (a `refine_scene` change): `{"id": "q10", "bind": "n1"}` rebinds a reference question; `{"id": "n12", "bind": "n1"}` merges a duplicate object into another (relations move, its "which?" question goes, a light carries over). Both refuse to relate an object to itself and add a note to `resolutions`. - "what light?" is asked once per ledger and dropped when a later sentence lights the scene. ## MCP tools | Tool | In → out | |---|---| | `check_words` | sentence, primitives → per-word status, missing words, completions | | `translate_to_scene` | sentence, primitives, seed, html, pronouns → ledger, PNG, GLB, mini HTML file, mini HTML text | | `refine_scene` | ledger, changes, seed, html → ledger, PNG, GLB, mini HTML file, mini HTML text | | `extend_scene` | ledger, sentence, primitives, seed, html, pronouns → ledger, PNG, GLB, mini HTML file, mini HTML text | | `simplify_item` | ledger, item id, html, rings, samples → report, rung PNGs, rung GLBs, mini HTML file, mini HTML text | | `papercraft_rings` | ledger, item id, rings, samples, height_cm → pieces report, PNG per A4 page (to look at), SVG per A4 page (mm, to print) | `check_words` also returns `maps_to` per noun (the asset it would get), so a word that exists but maps to the wrong thing shows up before submitting. Ladder rungs: original → 20k / 5k / 1k / 250 / 60 faces → convex hull → oriented box. Where the object stops reading as its word is judged in the discussion. Decimation welds split vertices (UV and normal seams) between passes. When quadric decimation still stalls (CAD meshes with sliver triangles, e.g. a SimReady lamp stuck at ~21k faces), vertex clustering takes over: cruder, but it always reduces and keeps thin parts. Rungs are labelled with the faces they actually have, and the ladder stops descending when a rung doesn't reduce, so a stalled mesh never shows as several identical rungs. **Rings rung (second path, for papercraft).** `rings` horizontal sections, each outline sampled at `samples` evenly spaced angles from the section's centre (the farthest crossing per ray), joined into bands with flat caps. Every band is one paper strip and every vertex is (ring, angle); the report lists each ring's height, centre and radii, so it can be checked by hand. One stack per object: each ray takes the farthest crossing, so disjoint outlines in one section merge (two legs become one band bridging the gap), branching or overhanging parts (a lamp's arm and shade) merge into one envelope, and heights with no section are skipped. ### Papercraft (rings) `papercraft_rings` turns the rings rung into a printable net (SVG, millimetres, real size or scaled to `height_cm`). Each band between two rings unrolls exactly into one strip: its triangles share edges in sequence, so laying them flat keeps every length. A band that would curl past ~120° on paper overlaps itself, so it is split into parts (2a, 2b). Every joint has glue tabs on one side only, labelled with the piece they meet (band top edge → next band or top cap, seam end → own start or next part, bottom cap rim → band 1). Caps under 4 mm are dropped and the band closes to a point; tabs under 2 mm are dropped. Cut solid lines, fold dashed lines. The report lists, per piece, `glues_to` (the tabs it carries) and `receives_from` (pieces whose tabs land on it), so a tabless piece such as the top cap still shows its joint. Output is A4 pages (190 × 277 mm inside 10 mm margins), each as a 150 dpi PNG (chats, `` and fetchers show PNGs; SVG files come back without a MIME type) and as an SVG in millimetres for printing. Each piece is turned to its narrowest orientation and shelf-packed; strips longer than a page are split like curled ones. With `height_cm` blank the model is the tallest whose every piece fits a page, which is set by band width and cap size, not strip length. The report gives `real_height_cm`, `max_height_cm_for_a4` and `fits_a4`; a piece too big for A4 at a requested height gets a page of its own size. ### Mini HTML A self-contained page: Three.js (`three@0.160.0` from unpkg) plus one inline `D` data object holding the sentence, the reading (word → commits), relations, unresolved questions, sun, ground, and each object's position, yaw and colour. Each object also has either inline low-poly geometry (≤400 faces in a scene, rungs ≤1000 faces in a ladder) or a primitive spec. Vertices are rounded to cm. Each inline mesh states `faces`, `budget` and `floor`: `floor: true` means the mesh wouldn't decimate down to the budget, so the page carries more than asked. The geometry is inline, so a saved page works after Gradio's temporary file URLs expire. **Geometry is for the handoff, not the loop.** Even at 400 faces a page costs thousands of tokens, which a multi-turn chat can't afford on every refine. So `html` defaults to `"url"`: the page comes back as a file URL and the text field is empty. Models iterate on the ledger and the picture, and pass `html="inline"` only when they build on the page (`"skip"` omits it). An expired URL costs nothing: the ledger fully determines the scene, and the next refine re-creates the page. **No secrets.** The Space is public and stores no token; the datasets are public and download without one. Optional env: `VLI_CACHE`. ## Known gaps / open questions - **Materialization**: only the rings path exists (`papercraft_rings`). Nets from the decimated rungs (general mesh unfolding) are not built yet. - **Generation tier** is a stub: rejecting a primitive returns a clear error before changing the ledger (use `shape` or `height_m` instead). - SimReady names → words is a heuristic: the name after its category is indexed as a whole phrase plus its final part (`interiorPart_sun_visor_a01` → "sun visor", "visor"; never "sun" alone). `maps_to` makes the remaining misses visible. - Objaverse "front" direction is unknown, so `faces` assumes +Z forward. - `on` uses the bounding-box top, so objects on irregular tops may float or sink. - Renders are matplotlib (headless, CPU): approximate shading and a flat shadow on the ground plane only.