Spaces:
Running on Zero
Running on Zero
kwabs22
Claude Opus 5.5
Add a pronoun rule option: most recent object or last clause's subject
6287002 | # VisualLearningIdea: v1 spec | |
| Supersedes `visuallearning-handover.html`, which drifted over several HuggingChat | |
| sessions. `cat-sun-scene.html` stays as the UX reference for the serial reading. | |
| ## Idea | |
| 3D is a language. The Space **translates English into 3D**, the way you would | |
| translate English into Japanese. A picture is easier to take in than text, because | |
| the reader doesn't have to guess the exact picture. Once 3D exists it can be | |
| materialized (papercraft, origami, …). The **simplification ladder** shows how far | |
| a mesh can be reduced and still mean its word, and that is a main output. | |
| One Gradio Space serves people (UI) and models (`demo.launch(mcp_server=True)`). | |
| HuggingChat models can't run tools, which is why this Space exists. | |
| **The Space visualises; discussion happens in the chat** (e.g. HuggingChat over MCP). | |
| The calling model already sees the picture, so the Space runs no vision model of | |
| its own and stores no token. Because HTML is the final destination, every picture | |
| comes back with a **mini HTML page** the model can read, edit and build on. | |
| ## Translation terms | |
| | Translation | Here | | |
| |---|---| | |
| | Source → target language | English → 3D | | |
| | Dictionary | Ready assets: SimReady → Objaverse | | |
| | Grammar words | `the / a / and`: no geometry; a determiner opens a "which?" question | | |
| | Verbs | Relations between meshes | | |
| | Word alignment | `tokens[].commits`: reading order differs from build order | | |
| | Forced choices | `unresolved`: which asset, how big | | |
| | Back-translation | The chat model reading the picture: done in the discussion, not by the Space | | |
| ## Dictionary | |
| - Tiers, each taken **on rejection**: `simready → objaverse → primitive → generation`. | |
| - **Available** = a SimReady or Objaverse asset exists (no construction). No asset | |
| means not available, with no exceptions. The UI offers only available words, so | |
| the user knows before submitting. | |
| - A missing word can be used only by **explicitly choosing a primitive** for it | |
| (`box, sphere, cylinder, cone, plane`). In the UI the user picks one. Over MCP, | |
| `check_words` returns `missing` and the user's model (e.g. in HuggingChat) decides: | |
| rephrase, or pass `primitives_json` such as `{"sun": "sphere"}`. | |
| - Assets that **can't be used** are skipped and recorded in `decision.skipped`: | |
| no license, NC (non-commercial) or ND (no derivatives; simplification is a | |
| derivative), or no geometry. | |
| - SimReady: package list from `NVIDIA-Omniverse/simready-central/submissions/*`. | |
| Only geometry USD + thumbnail are downloaded, because textures are ~98% of each | |
| ~1 GB package. Assets are metric, so height comes from the asset. | |
| - Objaverse: LVIS categories (`lvis-annotations.json.gz`) are the word index; the | |
| license comes from `metadata/<shard>.json.gz`. No real scale, so height is a | |
| nominal default (an `unresolved` decision the user can set). | |
| - Light comes from the verb, not from a special word: whatever "shone" becomes the | |
| light source and sits in the sky along the light direction. | |
| - When nothing shines, the picture still needs light: seeing needs light as sitting | |
| needs a ground. The scene gets soft daylight from no object (`sun.daylight: true`, | |
| seeded direction, shadows on) and `unresolved` asks "what light?" (default: | |
| daylight; rephrase with "<noun> shone/glowed" to make an object the light). | |
| - SimReady geometry comes without textures, so the colour is the mean of the asset's | |
| 256 px thumbnail (backdrop and transparent pixels left out), stored as `item.color`. | |
| `item.thumbnail` is a public dataset URL, so the chat can show and compare assets. | |
| Uncoloured meshes render neutral light grey. | |
| - A "which <word>?" question carries `on_reject`: what rejecting the tier would offer | |
| (e.g. `{"tier": "objaverse", "candidates": 101, "offered": 9}`), since those | |
| alternatives are invisible until the tier is rejected. | |
| - The ground implied by "sat" (with no stated object) is a support plane created by | |
| the relation. It is not a typed word, so it needs no asset. To exist, a thing needs | |
| a reality to exist in: reading skips this, but visualising exposes it. So the | |
| implied ground is also surfaced in `unresolved` ("cat sat on what?", default: | |
| ground plane), for the discussion to accept or rephrase. | |
| ## Serial side vs configuration side | |
| - **Serial (ledger)**: tokens in reading order, items each word commits, relations, | |
| unresolved questions. **Freezing lives here.** Frozen items can't change; the | |
| others can be swapped, rejected, resized or rerolled. | |
| - **Configuration**: once the serial side is resolved, any placement satisfying the | |
| relations is valid. `seed` picks one (positions, yaw, sun direction). | |
| ### Ledger shape | |
| ```jsonc | |
| { | |
| "sentence": "the cat sat and the sun shone", | |
| "tokens": [{"index": 2, "text": "sat", "kind": "verb", "commits": ["ground", "r2"], "opens": []}], | |
| "items": [ | |
| {"id": "n1", "kind": "object", "word": "cat", "token": 1, "frozen": false, | |
| "decision": {"tier": "objaverse", "chosen": {…}, "alternatives": […], | |
| "rejected_tiers": [], "skipped": [], "label": "cat"}, | |
| "height_m": {"value": 0.3, "source": "default"}, "license": "by"}, | |
| {"id": "ground", "kind": "environment", "role": "plane", "implied_by": 2}, | |
| {"id": "r2", "kind": "relation", "relation": "on", "subject": "n1", "object": "ground"}, | |
| {"id": "n5", "kind": "object", "word": "sun", "emits_light": true, | |
| "decision": {"tier": "primitive", "chosen": {"source": "primitive", "shape": "sphere"}, …}}, | |
| {"id": "r6", "kind": "relation", "relation": "lights", "subject": "n5", "object": "scene"} | |
| ], | |
| "unresolved": [{"id": "q0", "about": "n1", "question": "which cat?", "default_chosen": "cat", "alternatives": […]}], | |
| "configuration": {"seed": 0, "placements": {"n1": {"position": [0, 0, 0], "yaw_deg": 12}}, "sun": {…}} | |
| } | |
| ``` | |
| Relations: `on, under, beside, near, behind, in_front_of, above, faces, lights`. | |
| Verbs: `sat/stood/lay/rested` → `on` (ground implied), `shone/glowed` → `lights` | |
| (shadows only turn on with this coupling), `faced` → `faces`. | |
| ## Paragraphs: sentences + bindings | |
| `extend_scene` adds the next sentence to an existing ledger (token and item ids carry | |
| on, `sentences` lists them). Within a sentence or across sentences: | |
| - **"the <noun>"** when that noun is already in the ledger refers to it (the most | |
| recent one) instead of adding an item. Surfaced as an implied question: "which cat? | |
| the one already mentioned" (default: the same cat), alternatives include "say | |
| 'another cat' for a new one". | |
| - **"a/an/another <noun>"** always adds a new item. | |
| - **Pronouns** (it, he, she, they, him, her, them) refer back by a rule, `pronouns` | |
| on `translate_to_scene`/`extend_scene`, kept in the ledger: | |
| - `"recent"` (default): the most recent object. | |
| - `"subject"`: the subject of the most recent clause, else the most recent object. | |
| Closer to how readers take "it": in "the cat sat on the table. the apple lay | |
| beside it", `recent` gives the table, `subject` the cat. | |
| Either way the clause's own subject is left out when the pronoun is its object ("the | |
| apple lay beside it" can't mean the apple), and the choice is surfaced as "'it' = the | |
| cat?" with the other objects as alternatives, most recent first. A pronoun with | |
| nothing before it is an error. | |
| - Each reference question carries `binds: {token, to}`; relations record which token | |
| named each end (`tokens: {subject, object}`), so a reference can be rebound. | |
| - **`bind`** (a `refine_scene` change): `{"id": "q10", "bind": "n1"}` rebinds a | |
| reference question; `{"id": "n12", "bind": "n1"}` merges a duplicate object into | |
| another (relations move, its "which?" question goes, a light carries over). Both | |
| refuse to relate an object to itself and add a note to `resolutions`. | |
| - "what light?" is asked once per ledger and dropped when a later sentence lights the scene. | |
| ## MCP tools | |
| | Tool | In → out | | |
| |---|---| | |
| | `check_words` | sentence, primitives → per-word status, missing words, completions | | |
| | `translate_to_scene` | sentence, primitives, seed, html, pronouns → ledger, PNG, GLB, mini HTML file, mini HTML text | | |
| | `refine_scene` | ledger, changes, seed, html → ledger, PNG, GLB, mini HTML file, mini HTML text | | |
| | `extend_scene` | ledger, sentence, primitives, seed, html, pronouns → ledger, PNG, GLB, mini HTML file, mini HTML text | | |
| | `simplify_item` | ledger, item id, html, rings, samples → report, rung PNGs, rung GLBs, mini HTML file, mini HTML text | | |
| | `papercraft_rings` | ledger, item id, rings, samples, height_cm → pieces report, PNG per A4 page (to look at), SVG per A4 page (mm, to print) | | |
| `check_words` also returns `maps_to` per noun (the asset it would get), so a word | |
| that exists but maps to the wrong thing shows up before submitting. | |
| Ladder rungs: original → 20k / 5k / 1k / 250 / 60 faces → convex hull → oriented | |
| box. Where the object stops reading as its word is judged in the discussion. | |
| Decimation welds split vertices (UV and normal seams) between passes. When quadric | |
| decimation still stalls (CAD meshes with sliver triangles, e.g. a SimReady lamp stuck | |
| at ~21k faces), vertex clustering takes over: cruder, but it always reduces and keeps | |
| thin parts. Rungs are | |
| labelled with the faces they actually have, and the ladder stops descending when | |
| a rung doesn't reduce, so a stalled mesh never shows as several identical rungs. | |
| **Rings rung (second path, for papercraft).** `rings` horizontal sections, each | |
| outline sampled at `samples` evenly spaced angles from the section's centre (the | |
| farthest crossing per ray), joined into bands with flat caps. Every band is one | |
| paper strip and every vertex is (ring, angle); the report lists each ring's height, | |
| centre and radii, so it can be checked by hand. One stack per object: each ray takes | |
| the farthest crossing, so disjoint outlines in one section merge (two legs become one | |
| band bridging the gap), branching or overhanging parts (a lamp's arm and shade) merge | |
| into one envelope, and heights with no section are skipped. | |
| ### Papercraft (rings) | |
| `papercraft_rings` turns the rings rung into a printable net (SVG, millimetres, real size | |
| or scaled to `height_cm`). Each band between two rings unrolls exactly into one strip: | |
| its triangles share edges in sequence, so laying them flat keeps every length. A band | |
| that would curl past ~120° on paper overlaps itself, so it is split into parts (2a, 2b). | |
| Every joint has glue tabs on one side only, labelled with the piece they meet (band top | |
| edge → next band or top cap, seam end → own start or next part, bottom cap rim → band 1). | |
| Caps under 4 mm are dropped and the band closes to a point; tabs under 2 mm are dropped. | |
| Cut solid lines, fold dashed lines. The report lists, per piece, `glues_to` (the tabs it | |
| carries) and `receives_from` (pieces whose tabs land on it), so a tabless piece such as | |
| the top cap still shows its joint. | |
| Output is A4 pages (190 × 277 mm inside 10 mm margins), each as a 150 dpi PNG (chats, | |
| `<img>` and fetchers show PNGs; SVG files come back without a MIME type) and as an SVG | |
| in millimetres for printing. Each piece is turned to its | |
| narrowest orientation and shelf-packed; strips longer than a page are split like curled | |
| ones. With `height_cm` blank the model is the tallest whose every piece fits a page, | |
| which is set by band width and cap size, not strip length. The report gives | |
| `real_height_cm`, `max_height_cm_for_a4` and `fits_a4`; a piece too big for A4 at a | |
| requested height gets a page of its own size. | |
| ### Mini HTML | |
| A self-contained page: Three.js (`three@0.160.0` from unpkg) plus one inline `D` | |
| data object holding the sentence, the reading (word → commits), relations, | |
| unresolved questions, sun, ground, and each object's position, yaw and colour. | |
| Each object also has either inline low-poly geometry (≤400 faces in a scene, | |
| rungs ≤1000 faces in a ladder) or a primitive spec. Vertices are rounded to cm. | |
| Each inline mesh states `faces`, `budget` and `floor`: `floor: true` means the mesh | |
| wouldn't decimate down to the budget, so the page carries more than asked. | |
| The geometry is inline, so a saved page works after Gradio's temporary file URLs expire. | |
| **Geometry is for the handoff, not the loop.** Even at 400 faces a page costs | |
| thousands of tokens, which a multi-turn chat can't afford on every refine. So | |
| `html` defaults to `"url"`: the page comes back as a file URL and the text field is | |
| empty. Models iterate on the ledger and the picture, and pass `html="inline"` only | |
| when they build on the page (`"skip"` omits it). An expired URL costs nothing: the | |
| ledger fully determines the scene, and the next refine re-creates the page. | |
| **No secrets.** The Space is public and stores no token; the datasets are public | |
| and download without one. Optional env: `VLI_CACHE`. | |
| ## Known gaps / open questions | |
| - **Materialization**: only the rings path exists (`papercraft_rings`). Nets from the | |
| decimated rungs (general mesh unfolding) are not built yet. | |
| - **Generation tier** is a stub: rejecting a primitive returns a clear error before | |
| changing the ledger (use `shape` or `height_m` instead). | |
| - SimReady names → words is a heuristic: the name after its category is indexed | |
| as a whole phrase plus its final part (`interiorPart_sun_visor_a01` → "sun visor", | |
| "visor"; never "sun" alone). `maps_to` makes the remaining misses visible. | |
| - Objaverse "front" direction is unknown, so `faces` assumes +Z forward. | |
| - `on` uses the bounding-box top, so objects on irregular tops may float or sink. | |
| - Renders are matplotlib (headless, CPU): approximate shading and a flat shadow on | |
| the ground plane only. | |