Spaces:
Running on Zero
Running on Zero
kwabs22 Claude Opus 5.5 commited on
Commit ·
7bb2fda
1
Parent(s): d7aa77a
Return a mini HTML page with every picture; drop the Space's vision model
Browse filesThe Space visualises and the chat discusses: translate, refine and simplify
now also return a self-contained Three.js page with inline low-poly geometry
for the calling model to build on. Removes describe_scene, the recognition
check, the login button (which leaked an MCP tool) and hf_oauth.
Also from the HuggingChat test: SimReady names no longer index lone
modifiers ("sun" found a sun visor), check_words shows what each noun maps
to, renders frame on the objects, light both faces, and the light glows.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- README.md +0 -3
- SPEC.md +28 -13
- app.py +65 -95
- vli/grammar.py +1 -0
- vli/html.py +199 -0
- vli/lexicon.py +14 -12
- vli/pipeline.py +2 -2
- vli/scene.py +15 -9
- vli/vlm.py +0 -35
README.md
CHANGED
|
@@ -8,9 +8,6 @@ sdk_version: 6.27.0
|
|
| 8 |
python_version: '3.12'
|
| 9 |
app_file: app.py
|
| 10 |
pinned: false
|
| 11 |
-
hf_oauth: true
|
| 12 |
-
hf_oauth_scopes:
|
| 13 |
-
- inference-api
|
| 14 |
short_description: Visualisation and speed of input attempt
|
| 15 |
---
|
| 16 |
|
|
|
|
| 8 |
python_version: '3.12'
|
| 9 |
app_file: app.py
|
| 10 |
pinned: false
|
|
|
|
|
|
|
|
|
|
| 11 |
short_description: Visualisation and speed of input attempt
|
| 12 |
---
|
| 13 |
|
SPEC.md
CHANGED
|
@@ -14,6 +14,11 @@ a mesh can be reduced and still mean its word, and that is a main output.
|
|
| 14 |
One Gradio Space serves people (UI) and models (`demo.launch(mcp_server=True)`).
|
| 15 |
HuggingChat models can't run tools, which is why this Space exists.
|
| 16 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 17 |
## Translation terms
|
| 18 |
|
| 19 |
| Translation | Here |
|
|
@@ -24,7 +29,7 @@ HuggingChat models can't run tools, which is why this Space exists.
|
|
| 24 |
| Verbs | Relations between meshes |
|
| 25 |
| Word alignment | `tokens[].commits`: reading order differs from build order |
|
| 26 |
| Forced choices | `unresolved`: which asset, how big |
|
| 27 |
-
| Back-translation |
|
| 28 |
|
| 29 |
## Dictionary
|
| 30 |
|
|
@@ -92,26 +97,36 @@ Verbs: `sat/stood/lay/rested` → `on` (ground implied), `shone/glowed` → `lig
|
|
| 92 |
| Tool | In → out |
|
| 93 |
|---|---|
|
| 94 |
| `check_words` | sentence, primitives → per-word status, missing words, completions |
|
| 95 |
-
| `translate_to_scene` | sentence, primitives, seed → ledger, PNG, GLB |
|
| 96 |
-
| `refine_scene` | ledger, changes, seed → ledger, PNG, GLB |
|
| 97 |
-
| `simplify_item` | ledger, item id
|
| 98 |
-
|
|
|
|
|
|
|
| 99 |
|
| 100 |
Ladder rungs: original → 20k / 5k / 1k / 250 / 60 faces → convex hull → oriented
|
| 101 |
-
box.
|
| 102 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 103 |
|
| 104 |
-
**No secrets.** The Space is public
|
| 105 |
-
and download without one.
|
| 106 |
-
the **caller's** token: HF sign-in in the UI (`hf_oauth`, `inference-api` scope),
|
| 107 |
-
or a Bearer `Authorization` header from MCP clients. Optional env: `VLM_MODEL`, `VLI_CACHE`.
|
| 108 |
|
| 109 |
## Known gaps / open questions
|
| 110 |
|
| 111 |
- **Materialization** (papercraft from a low rung) is not built yet.
|
| 112 |
- **Generation tier** is a stub: rejecting a primitive errors out.
|
| 113 |
-
- SimReady names → words is a heuristic: the
|
| 114 |
-
(`
|
|
|
|
| 115 |
- Objaverse "front" direction is unknown, so `faces` assumes +Z forward.
|
| 116 |
- `on` uses the bounding-box top, so objects on irregular tops may float or sink.
|
| 117 |
- Renders are matplotlib (headless, CPU): approximate shading and a flat shadow on
|
|
|
|
| 14 |
One Gradio Space serves people (UI) and models (`demo.launch(mcp_server=True)`).
|
| 15 |
HuggingChat models can't run tools, which is why this Space exists.
|
| 16 |
|
| 17 |
+
**The Space visualises; discussion happens in the chat** (e.g. HuggingChat over MCP).
|
| 18 |
+
The calling model already sees the picture, so the Space runs no vision model of
|
| 19 |
+
its own and stores no token. Because HTML is the final destination, every picture
|
| 20 |
+
comes back with a **mini HTML page** the model can read, edit and build on.
|
| 21 |
+
|
| 22 |
## Translation terms
|
| 23 |
|
| 24 |
| Translation | Here |
|
|
|
|
| 29 |
| Verbs | Relations between meshes |
|
| 30 |
| Word alignment | `tokens[].commits`: reading order differs from build order |
|
| 31 |
| Forced choices | `unresolved`: which asset, how big |
|
| 32 |
+
| Back-translation | The chat model reading the picture: done in the discussion, not by the Space |
|
| 33 |
|
| 34 |
## Dictionary
|
| 35 |
|
|
|
|
| 97 |
| Tool | In → out |
|
| 98 |
|---|---|
|
| 99 |
| `check_words` | sentence, primitives → per-word status, missing words, completions |
|
| 100 |
+
| `translate_to_scene` | sentence, primitives, seed → ledger, PNG, GLB, mini HTML |
|
| 101 |
+
| `refine_scene` | ledger, changes, seed → ledger, PNG, GLB, mini HTML |
|
| 102 |
+
| `simplify_item` | ledger, item id → report, rung PNGs, rung GLBs, mini HTML |
|
| 103 |
+
|
| 104 |
+
`check_words` also returns `maps_to` per noun (the asset it would get), so a word
|
| 105 |
+
that exists but maps to the wrong thing shows up before submitting.
|
| 106 |
|
| 107 |
Ladder rungs: original → 20k / 5k / 1k / 250 / 60 faces → convex hull → oriented
|
| 108 |
+
box. Where the object stops reading as its word is judged in the discussion.
|
| 109 |
+
|
| 110 |
+
### Mini HTML
|
| 111 |
+
|
| 112 |
+
A self-contained page: Three.js (`three@0.160.0` from unpkg) plus one inline `D`
|
| 113 |
+
data object holding the sentence, the reading (word → commits), relations,
|
| 114 |
+
unresolved questions, sun, ground, and each object's position, yaw and colour.
|
| 115 |
+
Each object also has either inline low-poly geometry (≤400 faces in a scene,
|
| 116 |
+
rungs ≤1000 faces in a ladder) or a primitive spec. The geometry is inline, so the
|
| 117 |
+
page works after Gradio's temporary file URLs expire, and it stays small enough
|
| 118 |
+
for a model's context.
|
| 119 |
|
| 120 |
+
**No secrets.** The Space is public and stores no token; the datasets are public
|
| 121 |
+
and download without one. Optional env: `VLI_CACHE`.
|
|
|
|
|
|
|
| 122 |
|
| 123 |
## Known gaps / open questions
|
| 124 |
|
| 125 |
- **Materialization** (papercraft from a low rung) is not built yet.
|
| 126 |
- **Generation tier** is a stub: rejecting a primitive errors out.
|
| 127 |
+
- SimReady names → words is a heuristic: the name after its category is indexed
|
| 128 |
+
as a whole phrase plus its final part (`interiorPart_sun_visor_a01` → "sun visor",
|
| 129 |
+
"visor"; never "sun" alone). `maps_to` makes the remaining misses visible.
|
| 130 |
- Objaverse "front" direction is unknown, so `faces` assumes +Z forward.
|
| 131 |
- `on` uses the bounding-box top, so objects on irregular tops may float or sink.
|
| 132 |
- Renders are matplotlib (headless, CPU): approximate shading and a flat shadow on
|
app.py
CHANGED
|
@@ -2,7 +2,10 @@
|
|
| 2 |
|
| 3 |
One Space for people (this UI) and for models (every public event below is an
|
| 4 |
MCP tool: demo.launch(mcp_server=True)). Helper events are api_visibility="private".
|
|
|
|
|
|
|
| 5 |
"""
|
|
|
|
| 6 |
import json
|
| 7 |
import os
|
| 8 |
import re
|
|
@@ -12,7 +15,7 @@ import threading
|
|
| 12 |
import spaces # ZeroGPU: import before any CUDA-touching library
|
| 13 |
import gradio as gr
|
| 14 |
|
| 15 |
-
from vli import assets, pipeline, scene, simplify
|
| 16 |
from vli.grammar import check_words as _check_words
|
| 17 |
from vli.lexicon import LEXICON, PRIMITIVE_SHAPES
|
| 18 |
|
|
@@ -35,14 +38,6 @@ def _json(text, name, default=None):
|
|
| 35 |
raise ValueError(f"{name} is not valid JSON")
|
| 36 |
|
| 37 |
|
| 38 |
-
def _user_token(oauth_token, request):
|
| 39 |
-
"""The caller's own token: from HF sign-in (UI) or a Bearer header (MCP). Never the Space's."""
|
| 40 |
-
if oauth_token is not None and oauth_token.token:
|
| 41 |
-
return oauth_token.token
|
| 42 |
-
auth = (request.headers.get("authorization", "") if request is not None else "")
|
| 43 |
-
return auth[7:].strip() if auth.lower().startswith("bearer ") else None
|
| 44 |
-
|
| 45 |
-
|
| 46 |
def _seed(value):
|
| 47 |
"""Seeds arrive as strings (some MCP clients can't send numbers)."""
|
| 48 |
try:
|
|
@@ -66,11 +61,11 @@ def generate_mesh(prompt):
|
|
| 66 |
def check_words(sentence: str, primitives_json: str = "{}") -> str:
|
| 67 |
"""Check which words of a sentence exist in the 3D vocabulary, and complete the word being typed.
|
| 68 |
|
| 69 |
-
A word is available only when a ready asset (SimReady or Objaverse) exists for it
|
| 70 |
-
Words with no asset are listed in "missing":
|
| 71 |
-
primitive shape in primitives_json. Verbs (sat,
|
| 72 |
-
prepositions (on, under, beside, near, behind, in front
|
| 73 |
-
between meshes; whatever "shone" becomes the light source.
|
| 74 |
|
| 75 |
Args:
|
| 76 |
sentence: The English sentence, complete or partly typed.
|
|
@@ -78,7 +73,7 @@ def check_words(sentence: str, primitives_json: str = "{}") -> str:
|
|
| 78 |
Shapes: box, sphere, cylinder, cone, plane.
|
| 79 |
|
| 80 |
Returns:
|
| 81 |
-
JSON string: {"ok", "words": [{text, kind, available_in | primitive}], "missing",
|
| 82 |
"primitive_shapes", "completions"}.
|
| 83 |
"""
|
| 84 |
try:
|
|
@@ -87,13 +82,14 @@ def check_words(sentence: str, primitives_json: str = "{}") -> str:
|
|
| 87 |
_fail(e)
|
| 88 |
|
| 89 |
|
| 90 |
-
def translate_to_scene(sentence: str, primitives_json: str = "{}", seed: str = "0") -> tuple[str, str, str]:
|
| 91 |
"""Translate an English sentence into a 3D scene.
|
| 92 |
|
| 93 |
Produces the serial ledger (tokens in reading order, the items each word commits,
|
| 94 |
-
and the unresolved
|
| 95 |
-
|
| 96 |
-
|
|
|
|
| 97 |
|
| 98 |
Args:
|
| 99 |
sentence: A sentence using available words, e.g. "the cat sat and the sun shone".
|
|
@@ -101,7 +97,7 @@ def translate_to_scene(sentence: str, primitives_json: str = "{}", seed: str = "
|
|
| 101 |
seed: Picks one of the configurations that satisfy the ledger (positions, facing, sun).
|
| 102 |
|
| 103 |
Returns:
|
| 104 |
-
(ledger JSON string, rendered PNG, GLB scene file).
|
| 105 |
"""
|
| 106 |
try:
|
| 107 |
return pipeline.translate(sentence, _json(primitives_json, "primitives_json", {}), _seed(seed))
|
|
@@ -109,17 +105,19 @@ def translate_to_scene(sentence: str, primitives_json: str = "{}", seed: str = "
|
|
| 109 |
_fail(e)
|
| 110 |
|
| 111 |
|
| 112 |
-
def refine_scene(ledger_json: str, changes_json: str = "[]", seed: str = "0") -> tuple[str, str, str]:
|
| 113 |
"""Refine a ledger: freeze some decisions and change the others, then re-render.
|
| 114 |
|
| 115 |
Changes are a JSON list of:
|
| 116 |
{"id": "n1", "freeze": true} lock an item (false unlocks)
|
| 117 |
-
{"id": "n1", "use": 2}
|
|
|
|
| 118 |
{"id": "n1", "reject": true} reject the tier: simready -> objaverse -> primitive -> generation
|
| 119 |
{"id": "n1", "height_m": 0.4} set the object's height in metres
|
| 120 |
{"id": "n1", "shape": "sphere"} use a primitive (box, sphere, cylinder, cone, plane)
|
| 121 |
{"reroll": true} every unfrozen object takes a random alternative
|
| 122 |
-
Frozen items cannot be changed. The configuration (placement) is always
|
|
|
|
| 123 |
|
| 124 |
Args:
|
| 125 |
ledger_json: Ledger JSON from translate_to_scene or a previous refine_scene.
|
|
@@ -127,7 +125,7 @@ def refine_scene(ledger_json: str, changes_json: str = "[]", seed: str = "0") ->
|
|
| 127 |
seed: Picks the configuration, and the alternatives for a reroll.
|
| 128 |
|
| 129 |
Returns:
|
| 130 |
-
(ledger JSON string, rendered PNG, GLB scene file).
|
| 131 |
"""
|
| 132 |
try:
|
| 133 |
ledger = _json(ledger_json, "ledger_json")
|
|
@@ -137,80 +135,39 @@ def refine_scene(ledger_json: str, changes_json: str = "[]", seed: str = "0") ->
|
|
| 137 |
_fail(e)
|
| 138 |
|
| 139 |
|
| 140 |
-
def simplify_item(ledger_json: str, item_id: str
|
| 141 |
-
oauth_token: gr.OAuthToken | None = None, request: gr.Request = None) -> tuple[str, list, list]:
|
| 142 |
"""Build the simplification ladder for one object: full asset -> fewer faces -> convex hull -> box.
|
| 143 |
|
| 144 |
-
|
| 145 |
-
still
|
|
|
|
| 146 |
|
| 147 |
Args:
|
| 148 |
ledger_json: Ledger JSON from translate_to_scene or refine_scene.
|
| 149 |
item_id: The object item to simplify, e.g. "n1".
|
| 150 |
-
check_recognition: Ask a vision model to name each rung (slower; runs on the caller's
|
| 151 |
-
own HF account, so it needs sign-in or a Bearer token).
|
| 152 |
|
| 153 |
Returns:
|
| 154 |
-
(report JSON string, gallery of rung pictures, GLB file per rung
|
|
|
|
| 155 |
"""
|
| 156 |
try:
|
| 157 |
-
token = _user_token(oauth_token, request) if check_recognition else None
|
| 158 |
-
if check_recognition and not token:
|
| 159 |
-
raise ValueError(vlm.SIGN_IN) # before any heavy work
|
| 160 |
ledger = _json(ledger_json, "ledger_json")
|
| 161 |
item = next((i for i in ledger["items"] if i["id"] == item_id), None)
|
| 162 |
if item is None or item["kind"] != "object":
|
| 163 |
raise ValueError(f"{item_id!r} is not an object item in this ledger")
|
| 164 |
mesh = assets.settle(item)
|
| 165 |
work = tempfile.mkdtemp(prefix="vli_ladder_")
|
|
|
|
| 166 |
report, gallery, files = {"item": item_id, "word": item["word"], "rungs": []}, [], []
|
| 167 |
-
for k, (label, rung) in enumerate(
|
| 168 |
png = scene.render({item_id: rung}, os.path.join(work, f"{k}.png"), title=label)
|
| 169 |
slug = re.sub(r"\W+", "_", label)
|
| 170 |
glb = os.path.join(work, f"{k}_{slug}.glb")
|
| 171 |
rung.export(glb)
|
| 172 |
-
|
| 173 |
-
if check_recognition:
|
| 174 |
-
answer = vlm.name_object(png, token)
|
| 175 |
-
entry["named_as"] = answer
|
| 176 |
-
entry["recognized"] = item["word"].lower() in answer.lower()
|
| 177 |
-
report["rungs"].append(entry)
|
| 178 |
gallery.append((png, label))
|
| 179 |
files.append(glb)
|
| 180 |
-
|
| 181 |
-
kept = [r for r in report["rungs"] if r["recognized"]]
|
| 182 |
-
report["simplest_recognized"] = kept[-1]["rung"] if kept else None
|
| 183 |
-
return json.dumps(report, indent=1), gallery, files
|
| 184 |
-
except Exception as e:
|
| 185 |
-
_fail(e)
|
| 186 |
-
|
| 187 |
-
|
| 188 |
-
def describe_scene(ledger_json: str, seed: str = "0",
|
| 189 |
-
oauth_token: gr.OAuthToken | None = None, request: gr.Request = None) -> str:
|
| 190 |
-
"""Back-translate: render the ledger's scene and have a vision model describe it in English.
|
| 191 |
-
|
| 192 |
-
Comparing the description with the original sentence checks the translation.
|
| 193 |
-
The vision model runs on the caller's own HF account (sign-in or Bearer token).
|
| 194 |
-
|
| 195 |
-
Args:
|
| 196 |
-
ledger_json: Ledger JSON from translate_to_scene or refine_scene.
|
| 197 |
-
seed: Which configuration to render.
|
| 198 |
-
|
| 199 |
-
Returns:
|
| 200 |
-
JSON string: {"original", "back_translation", "nouns_recovered": {word: bool}}.
|
| 201 |
-
"""
|
| 202 |
-
try:
|
| 203 |
-
token = _user_token(oauth_token, request)
|
| 204 |
-
if not token:
|
| 205 |
-
raise ValueError(vlm.SIGN_IN) # before any heavy work
|
| 206 |
-
ledger = _json(ledger_json, "ledger_json")
|
| 207 |
-
meshes = pipeline.settle_all(ledger)
|
| 208 |
-
config = scene.configure(ledger, meshes, _seed(seed))
|
| 209 |
-
png = scene.render_scene(ledger, meshes, config, os.path.join(tempfile.mkdtemp(), "scene.png"))
|
| 210 |
-
text = vlm.describe(png, token)
|
| 211 |
-
nouns = [i["word"] for i in ledger["items"] if i["kind"] == "object"]
|
| 212 |
-
return json.dumps({"original": ledger["sentence"], "back_translation": text,
|
| 213 |
-
"nouns_recovered": {n: n in text.lower() for n in nouns}}, indent=1)
|
| 214 |
except Exception as e:
|
| 215 |
_fail(e)
|
| 216 |
|
|
@@ -224,12 +181,12 @@ def _suggestions(words_json):
|
|
| 224 |
if w["kind"] == "unknown":
|
| 225 |
marks.append(f"~~{w['text']}~~")
|
| 226 |
elif w["kind"] == "noun":
|
| 227 |
-
marks.append(f"**{w['text']}**")
|
| 228 |
elif w["kind"] == "primitive":
|
| 229 |
marks.append(f"_{w['text']}_ ({w['primitive']})")
|
| 230 |
else:
|
| 231 |
marks.append(w["text"])
|
| 232 |
-
status = " ".join(marks) or "_start typing_"
|
| 233 |
if data.get("missing"):
|
| 234 |
status += " \n_Struck-through words have no asset: pick a suggestion, rephrase, or choose a primitive._"
|
| 235 |
missing = data.get("missing", [])
|
|
@@ -250,10 +207,22 @@ def _use_primitive(primitives_json, word, shape):
|
|
| 250 |
return json.dumps(prims)
|
| 251 |
|
| 252 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 253 |
EXAMPLE_CHANGES = '[\n {"id": "n1", "freeze": true},\n {"reroll": true}\n]'
|
| 254 |
|
| 255 |
DATASETS_MD = """
|
| 256 |
-
**Dictionary (public datasets, read-only):**
|
| 257 |
1. [NVIDIA SimReady](https://github.com/NVIDIA-Omniverse/simready-central): metric OpenUSD assets,
|
| 258 |
e.g. [nvidia/simready-assets](https://huggingface.co/datasets/nvidia/simready-assets). License per package
|
| 259 |
(e.g. CC-BY-4.0). Only geometry and the thumbnail are downloaded, not the ~1 GB of textures.
|
|
@@ -261,17 +230,14 @@ DATASETS_MD = """
|
|
| 261 |
License per object; non-commercial (NC) and no-derivatives (ND) objects are skipped.
|
| 262 |
3. Missing words: a primitive you choose (box, sphere, cylinder, cone, plane). Generation comes later.
|
| 263 |
|
| 264 |
-
|
| 265 |
-
|
| 266 |
-
via HF Inference), and those calls run on **your own** Hugging Face account. This Space stores no token,
|
| 267 |
-
so nobody's credits are spent but the caller's.
|
| 268 |
"""
|
| 269 |
|
| 270 |
with gr.Blocks(title="VisualLearningIdea") as demo:
|
| 271 |
gr.Markdown("# English → 3D\nType with the words the 3D dictionary has. **Bold** = an object asset; "
|
| 272 |
"verbs and prepositions are relations between meshes.")
|
| 273 |
gr.Markdown(DATASETS_MD)
|
| 274 |
-
gr.LoginButton()
|
| 275 |
with gr.Row():
|
| 276 |
sentence = gr.Textbox(label="Sentence", placeholder="the cat sat and the sun shone", scale=4)
|
| 277 |
seed = gr.Textbox(label="Configuration seed", value="0", scale=1)
|
|
@@ -287,33 +253,37 @@ with gr.Blocks(title="VisualLearningIdea") as demo:
|
|
| 287 |
with gr.Row():
|
| 288 |
picture = gr.Image(label="Picture", type="filepath")
|
| 289 |
model = gr.Model3D(label="3D")
|
|
|
|
|
|
|
|
|
|
|
|
|
| 290 |
with gr.Accordion("Ledger: freeze and change (serial side)", open=False):
|
| 291 |
ledger = gr.Code(label="Ledger", language="json", interactive=True)
|
| 292 |
changes = gr.Code(label="Changes", language="json", value=EXAMPLE_CHANGES, interactive=True)
|
| 293 |
refine = gr.Button("Refine")
|
| 294 |
with gr.Accordion("Simplification ladder", open=False):
|
| 295 |
-
|
| 296 |
-
item_id = gr.Textbox(label="Object item id", value="n1")
|
| 297 |
-
recog = gr.Checkbox(label="Check recognition with a vision model", value=False)
|
| 298 |
simp = gr.Button("Simplify")
|
| 299 |
ladder_pics = gr.Gallery(label="Rungs", columns=4)
|
|
|
|
|
|
|
| 300 |
ladder_report = gr.Code(label="Report", language="json")
|
| 301 |
ladder_files = gr.File(label="GLB per rung", file_count="multiple")
|
| 302 |
-
with gr.Accordion("Back-translation", open=False):
|
| 303 |
-
back = gr.Button("Describe the scene")
|
| 304 |
-
back_out = gr.Code(label="Back-translation", language="json")
|
| 305 |
|
| 306 |
gr.on([sentence.change, primitives.change], check_words, [sentence, primitives], words,
|
| 307 |
api_name="check_words", trigger_mode="always_last") \
|
| 308 |
.then(_suggestions, words, [status, suggest, missing], api_visibility="private")
|
| 309 |
suggest.select(_complete, [sentence, suggest], sentence, api_visibility="private")
|
| 310 |
use_prim.click(_use_primitive, [primitives, missing, shape], primitives, api_visibility="private")
|
| 311 |
-
go.click(translate_to_scene, [sentence, primitives, seed], [ledger, picture, model],
|
| 312 |
-
api_name="translate_to_scene")
|
| 313 |
-
|
| 314 |
-
|
| 315 |
-
|
| 316 |
-
|
|
|
|
|
|
|
|
|
|
| 317 |
|
| 318 |
if __name__ == "__main__":
|
| 319 |
demo.launch(mcp_server=True)
|
|
|
|
| 2 |
|
| 3 |
One Space for people (this UI) and for models (every public event below is an
|
| 4 |
MCP tool: demo.launch(mcp_server=True)). Helper events are api_visibility="private".
|
| 5 |
+
The Space only visualises; discussion happens in the chat that calls it. Every
|
| 6 |
+
picture comes back with a mini HTML page, because HTML is the final destination.
|
| 7 |
"""
|
| 8 |
+
import html as html_lib
|
| 9 |
import json
|
| 10 |
import os
|
| 11 |
import re
|
|
|
|
| 15 |
import spaces # ZeroGPU: import before any CUDA-touching library
|
| 16 |
import gradio as gr
|
| 17 |
|
| 18 |
+
from vli import assets, html, pipeline, scene, simplify
|
| 19 |
from vli.grammar import check_words as _check_words
|
| 20 |
from vli.lexicon import LEXICON, PRIMITIVE_SHAPES
|
| 21 |
|
|
|
|
| 38 |
raise ValueError(f"{name} is not valid JSON")
|
| 39 |
|
| 40 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 41 |
def _seed(value):
|
| 42 |
"""Seeds arrive as strings (some MCP clients can't send numbers)."""
|
| 43 |
try:
|
|
|
|
| 61 |
def check_words(sentence: str, primitives_json: str = "{}") -> str:
|
| 62 |
"""Check which words of a sentence exist in the 3D vocabulary, and complete the word being typed.
|
| 63 |
|
| 64 |
+
A word is available only when a ready asset (SimReady or Objaverse) exists for it;
|
| 65 |
+
"maps_to" names the asset it would get. Words with no asset are listed in "missing":
|
| 66 |
+
rephrase them, or map each one to a primitive shape in primitives_json. Verbs (sat,
|
| 67 |
+
stood, lay, shone, faced) and prepositions (on, under, beside, near, behind, in front
|
| 68 |
+
of, above) are relations between meshes; whatever "shone" becomes the light source.
|
| 69 |
|
| 70 |
Args:
|
| 71 |
sentence: The English sentence, complete or partly typed.
|
|
|
|
| 73 |
Shapes: box, sphere, cylinder, cone, plane.
|
| 74 |
|
| 75 |
Returns:
|
| 76 |
+
JSON string: {"ok", "words": [{text, kind, available_in, maps_to | primitive}], "missing",
|
| 77 |
"primitive_shapes", "completions"}.
|
| 78 |
"""
|
| 79 |
try:
|
|
|
|
| 82 |
_fail(e)
|
| 83 |
|
| 84 |
|
| 85 |
+
def translate_to_scene(sentence: str, primitives_json: str = "{}", seed: str = "0") -> tuple[str, str, str, str]:
|
| 86 |
"""Translate an English sentence into a 3D scene.
|
| 87 |
|
| 88 |
Produces the serial ledger (tokens in reading order, the items each word commits,
|
| 89 |
+
and the unresolved questions), one configuration of that ledger chosen by `seed`,
|
| 90 |
+
a rendered picture, a GLB model and a mini HTML page. The HTML is a self-contained
|
| 91 |
+
Three.js scene with inline low-poly geometry: read it, edit it, build on it. Words
|
| 92 |
+
with no asset must be mapped to a primitive in primitives_json (see check_words).
|
| 93 |
|
| 94 |
Args:
|
| 95 |
sentence: A sentence using available words, e.g. "the cat sat and the sun shone".
|
|
|
|
| 97 |
seed: Picks one of the configurations that satisfy the ledger (positions, facing, sun).
|
| 98 |
|
| 99 |
Returns:
|
| 100 |
+
(ledger JSON string, rendered PNG, GLB scene file, mini HTML page).
|
| 101 |
"""
|
| 102 |
try:
|
| 103 |
return pipeline.translate(sentence, _json(primitives_json, "primitives_json", {}), _seed(seed))
|
|
|
|
| 105 |
_fail(e)
|
| 106 |
|
| 107 |
|
| 108 |
+
def refine_scene(ledger_json: str, changes_json: str = "[]", seed: str = "0") -> tuple[str, str, str, str]:
|
| 109 |
"""Refine a ledger: freeze some decisions and change the others, then re-render.
|
| 110 |
|
| 111 |
Changes are a JSON list of:
|
| 112 |
{"id": "n1", "freeze": true} lock an item (false unlocks)
|
| 113 |
+
{"id": "n1", "use": 2} swap in alternatives[2]; the old choice takes slot 2,
|
| 114 |
+
so using 2 again swaps back (other slots don't move)
|
| 115 |
{"id": "n1", "reject": true} reject the tier: simready -> objaverse -> primitive -> generation
|
| 116 |
{"id": "n1", "height_m": 0.4} set the object's height in metres
|
| 117 |
{"id": "n1", "shape": "sphere"} use a primitive (box, sphere, cylinder, cone, plane)
|
| 118 |
{"reroll": true} every unfrozen object takes a random alternative
|
| 119 |
+
Frozen items cannot be changed. The configuration (placement, sun angle) is always
|
| 120 |
+
re-chosen from `seed`: change the seed for another valid configuration.
|
| 121 |
|
| 122 |
Args:
|
| 123 |
ledger_json: Ledger JSON from translate_to_scene or a previous refine_scene.
|
|
|
|
| 125 |
seed: Picks the configuration, and the alternatives for a reroll.
|
| 126 |
|
| 127 |
Returns:
|
| 128 |
+
(ledger JSON string, rendered PNG, GLB scene file, mini HTML page).
|
| 129 |
"""
|
| 130 |
try:
|
| 131 |
ledger = _json(ledger_json, "ledger_json")
|
|
|
|
| 135 |
_fail(e)
|
| 136 |
|
| 137 |
|
| 138 |
+
def simplify_item(ledger_json: str, item_id: str) -> tuple[str, list, list, str]:
|
|
|
|
| 139 |
"""Build the simplification ladder for one object: full asset -> fewer faces -> convex hull -> box.
|
| 140 |
|
| 141 |
+
Look at where the object stops reading as its word: that rung is the least geometry
|
| 142 |
+
that still carries the meaning, and the low rungs are the ones that can be made
|
| 143 |
+
physically (fewer faces = fewer papercraft pieces).
|
| 144 |
|
| 145 |
Args:
|
| 146 |
ledger_json: Ledger JSON from translate_to_scene or refine_scene.
|
| 147 |
item_id: The object item to simplify, e.g. "n1".
|
|
|
|
|
|
|
| 148 |
|
| 149 |
Returns:
|
| 150 |
+
(report JSON string, gallery of rung pictures, GLB file per rung,
|
| 151 |
+
mini HTML page with the low rungs side by side).
|
| 152 |
"""
|
| 153 |
try:
|
|
|
|
|
|
|
|
|
|
| 154 |
ledger = _json(ledger_json, "ledger_json")
|
| 155 |
item = next((i for i in ledger["items"] if i["id"] == item_id), None)
|
| 156 |
if item is None or item["kind"] != "object":
|
| 157 |
raise ValueError(f"{item_id!r} is not an object item in this ledger")
|
| 158 |
mesh = assets.settle(item)
|
| 159 |
work = tempfile.mkdtemp(prefix="vli_ladder_")
|
| 160 |
+
rungs = simplify.ladder(mesh)
|
| 161 |
report, gallery, files = {"item": item_id, "word": item["word"], "rungs": []}, [], []
|
| 162 |
+
for k, (label, rung) in enumerate(rungs):
|
| 163 |
png = scene.render({item_id: rung}, os.path.join(work, f"{k}.png"), title=label)
|
| 164 |
slug = re.sub(r"\W+", "_", label)
|
| 165 |
glb = os.path.join(work, f"{k}_{slug}.glb")
|
| 166 |
rung.export(glb)
|
| 167 |
+
report["rungs"].append({"rung": label, **simplify.stats(rung)})
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 168 |
gallery.append((png, label))
|
| 169 |
files.append(glb)
|
| 170 |
+
return json.dumps(report, indent=1), gallery, files, html.ladder_page(item, rungs)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 171 |
except Exception as e:
|
| 172 |
_fail(e)
|
| 173 |
|
|
|
|
| 181 |
if w["kind"] == "unknown":
|
| 182 |
marks.append(f"~~{w['text']}~~")
|
| 183 |
elif w["kind"] == "noun":
|
| 184 |
+
marks.append(f"**{w['text']}** → {w.get('maps_to', '?')}")
|
| 185 |
elif w["kind"] == "primitive":
|
| 186 |
marks.append(f"_{w['text']}_ ({w['primitive']})")
|
| 187 |
else:
|
| 188 |
marks.append(w["text"])
|
| 189 |
+
status = " · ".join(marks) or "_start typing_"
|
| 190 |
if data.get("missing"):
|
| 191 |
status += " \n_Struck-through words have no asset: pick a suggestion, rephrase, or choose a primitive._"
|
| 192 |
missing = data.get("missing", [])
|
|
|
|
| 207 |
return json.dumps(prims)
|
| 208 |
|
| 209 |
|
| 210 |
+
def _html_views(page):
|
| 211 |
+
"""Live preview (sandboxed iframe) and a downloadable .html file."""
|
| 212 |
+
if not page:
|
| 213 |
+
return "", None
|
| 214 |
+
path = os.path.join(tempfile.mkdtemp(prefix="vli_html_"), "scene.html")
|
| 215 |
+
with open(path, "w", encoding="utf-8") as f:
|
| 216 |
+
f.write(page)
|
| 217 |
+
frame = (f'<iframe sandbox="allow-scripts" srcdoc="{html_lib.escape(page, quote=True)}" '
|
| 218 |
+
'style="width:100%;height:460px;border:0;border-radius:8px"></iframe>')
|
| 219 |
+
return frame, path
|
| 220 |
+
|
| 221 |
+
|
| 222 |
EXAMPLE_CHANGES = '[\n {"id": "n1", "freeze": true},\n {"reroll": true}\n]'
|
| 223 |
|
| 224 |
DATASETS_MD = """
|
| 225 |
+
**Dictionary (public datasets, read-only, no account needed):**
|
| 226 |
1. [NVIDIA SimReady](https://github.com/NVIDIA-Omniverse/simready-central): metric OpenUSD assets,
|
| 227 |
e.g. [nvidia/simready-assets](https://huggingface.co/datasets/nvidia/simready-assets). License per package
|
| 228 |
(e.g. CC-BY-4.0). Only geometry and the thumbnail are downloaded, not the ~1 GB of textures.
|
|
|
|
| 230 |
License per object; non-commercial (NC) and no-derivatives (ND) objects are skipped.
|
| 231 |
3. Missing words: a primitive you choose (box, sphere, cylinder, cone, plane). Generation comes later.
|
| 232 |
|
| 233 |
+
This Space only visualises and stores no token. Discuss the results in your chat
|
| 234 |
+
(e.g. HuggingChat via MCP): every picture comes with a mini HTML page to build on.
|
|
|
|
|
|
|
| 235 |
"""
|
| 236 |
|
| 237 |
with gr.Blocks(title="VisualLearningIdea") as demo:
|
| 238 |
gr.Markdown("# English → 3D\nType with the words the 3D dictionary has. **Bold** = an object asset; "
|
| 239 |
"verbs and prepositions are relations between meshes.")
|
| 240 |
gr.Markdown(DATASETS_MD)
|
|
|
|
| 241 |
with gr.Row():
|
| 242 |
sentence = gr.Textbox(label="Sentence", placeholder="the cat sat and the sun shone", scale=4)
|
| 243 |
seed = gr.Textbox(label="Configuration seed", value="0", scale=1)
|
|
|
|
| 253 |
with gr.Row():
|
| 254 |
picture = gr.Image(label="Picture", type="filepath")
|
| 255 |
model = gr.Model3D(label="3D")
|
| 256 |
+
with gr.Accordion("Mini HTML", open=False):
|
| 257 |
+
scene_preview = gr.HTML()
|
| 258 |
+
scene_file = gr.File(label="Download .html")
|
| 259 |
+
scene_html = gr.Code(label="HTML", language="html")
|
| 260 |
with gr.Accordion("Ledger: freeze and change (serial side)", open=False):
|
| 261 |
ledger = gr.Code(label="Ledger", language="json", interactive=True)
|
| 262 |
changes = gr.Code(label="Changes", language="json", value=EXAMPLE_CHANGES, interactive=True)
|
| 263 |
refine = gr.Button("Refine")
|
| 264 |
with gr.Accordion("Simplification ladder", open=False):
|
| 265 |
+
item_id = gr.Textbox(label="Object item id", value="n1")
|
|
|
|
|
|
|
| 266 |
simp = gr.Button("Simplify")
|
| 267 |
ladder_pics = gr.Gallery(label="Rungs", columns=4)
|
| 268 |
+
ladder_preview = gr.HTML()
|
| 269 |
+
ladder_html = gr.Code(label="HTML", language="html")
|
| 270 |
ladder_report = gr.Code(label="Report", language="json")
|
| 271 |
ladder_files = gr.File(label="GLB per rung", file_count="multiple")
|
|
|
|
|
|
|
|
|
|
| 272 |
|
| 273 |
gr.on([sentence.change, primitives.change], check_words, [sentence, primitives], words,
|
| 274 |
api_name="check_words", trigger_mode="always_last") \
|
| 275 |
.then(_suggestions, words, [status, suggest, missing], api_visibility="private")
|
| 276 |
suggest.select(_complete, [sentence, suggest], sentence, api_visibility="private")
|
| 277 |
use_prim.click(_use_primitive, [primitives, missing, shape], primitives, api_visibility="private")
|
| 278 |
+
go.click(translate_to_scene, [sentence, primitives, seed], [ledger, picture, model, scene_html],
|
| 279 |
+
api_name="translate_to_scene") \
|
| 280 |
+
.then(_html_views, scene_html, [scene_preview, scene_file], api_visibility="private")
|
| 281 |
+
refine.click(refine_scene, [ledger, changes, seed], [ledger, picture, model, scene_html],
|
| 282 |
+
api_name="refine_scene") \
|
| 283 |
+
.then(_html_views, scene_html, [scene_preview, scene_file], api_visibility="private")
|
| 284 |
+
simp.click(simplify_item, [ledger, item_id], [ladder_report, ladder_pics, ladder_files, ladder_html],
|
| 285 |
+
api_name="simplify_item") \
|
| 286 |
+
.then(lambda page: _html_views(page)[0], ladder_html, ladder_preview, api_visibility="private")
|
| 287 |
|
| 288 |
if __name__ == "__main__":
|
| 289 |
demo.launch(mcp_server=True)
|
vli/grammar.py
CHANGED
|
@@ -237,6 +237,7 @@ def check_words(sentence, primitives=None):
|
|
| 237 |
entry = {"text": t["text"], "kind": t["kind"]}
|
| 238 |
if t["kind"] == "noun":
|
| 239 |
entry["available_in"] = LEXICON.counts(t["text"])
|
|
|
|
| 240 |
elif t["kind"] == "primitive":
|
| 241 |
entry["primitive"] = primitives[t["text"]]
|
| 242 |
words.append(entry)
|
|
|
|
| 237 |
entry = {"text": t["text"], "kind": t["kind"]}
|
| 238 |
if t["kind"] == "noun":
|
| 239 |
entry["available_in"] = LEXICON.counts(t["text"])
|
| 240 |
+
entry["maps_to"] = candidate_label(_object_decision(t["text"])["chosen"]) # what you'd get
|
| 241 |
elif t["kind"] == "primitive":
|
| 242 |
entry["primitive"] = primitives[t["text"]]
|
| 243 |
words.append(entry)
|
vli/html.py
ADDED
|
@@ -0,0 +1,199 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Mini HTML returned next to every picture.
|
| 2 |
+
|
| 3 |
+
HTML is the final destination: a chat model reads this page, edits it and
|
| 4 |
+
builds its own on it. The page is self-contained Three.js. Geometry is inlined
|
| 5 |
+
at low poly (a rung of the simplification ladder), so it outlives Gradio's
|
| 6 |
+
temporary file URLs and stays small enough for a model's context.
|
| 7 |
+
"""
|
| 8 |
+
import json
|
| 9 |
+
|
| 10 |
+
import numpy as np
|
| 11 |
+
|
| 12 |
+
from .scene import GLOW, _sky
|
| 13 |
+
from .simplify import base_color, decimate
|
| 14 |
+
|
| 15 |
+
THREE = "https://unpkg.com/three@0.160.0/"
|
| 16 |
+
SCENE_FACE_BUDGET = 400 # per object in a scene page
|
| 17 |
+
LADDER_MAX_FACES = 1000 # rungs above this are left to the GLB files
|
| 18 |
+
|
| 19 |
+
|
| 20 |
+
def _hex(rgb):
|
| 21 |
+
return "#%02x%02x%02x" % tuple(int(c) for c in rgb[:3])
|
| 22 |
+
|
| 23 |
+
|
| 24 |
+
def _sky_hex(sun):
|
| 25 |
+
sky = _sky(sun)
|
| 26 |
+
return sky if isinstance(sky, str) else _hex([c * 255 for c in sky])
|
| 27 |
+
|
| 28 |
+
|
| 29 |
+
def _geometry(mesh, budget):
|
| 30 |
+
m = decimate(mesh, budget) if len(mesh.faces) > budget else mesh
|
| 31 |
+
return {"faces": int(len(m.faces)),
|
| 32 |
+
"p": np.round(m.vertices, 3).ravel().tolist(),
|
| 33 |
+
"i": m.faces.ravel().astype(int).tolist()}
|
| 34 |
+
|
| 35 |
+
|
| 36 |
+
def _object(oid, word, mesh, position, yaw_deg, cand=None, glow=False, budget=SCENE_FACE_BUDGET, label=None):
|
| 37 |
+
o = {"id": oid, "word": word, "label": label, "position": [round(float(c), 3) for c in position],
|
| 38 |
+
"yaw_deg": round(float(yaw_deg), 1), "color": _hex(GLOW if glow else base_color(mesh)),
|
| 39 |
+
"emits_light": glow}
|
| 40 |
+
if cand and cand.get("source") == "primitive":
|
| 41 |
+
o["primitive"] = {"shape": cand["shape"], "size": [round(float(e), 3) for e in mesh.extents]}
|
| 42 |
+
else:
|
| 43 |
+
o["mesh"] = _geometry(mesh, budget)
|
| 44 |
+
return o
|
| 45 |
+
|
| 46 |
+
|
| 47 |
+
def scene_page(ledger, meshes, config):
|
| 48 |
+
sun = config["sun"]
|
| 49 |
+
objects = []
|
| 50 |
+
for it in ledger["items"]:
|
| 51 |
+
p = config["placements"].get(it["id"])
|
| 52 |
+
if it["kind"] != "object" or p is None:
|
| 53 |
+
continue
|
| 54 |
+
dec = it["decision"]
|
| 55 |
+
objects.append(_object(it["id"], it["word"], meshes[it["id"]], p["position"], p["yaw_deg"],
|
| 56 |
+
cand=dec["chosen"], glow=bool(p.get("light")), label=dec.get("label")))
|
| 57 |
+
ground = None
|
| 58 |
+
if any(i["kind"] == "environment" and i["role"] == "plane" for i in ledger["items"]):
|
| 59 |
+
reach = max([abs(c) for o in objects if not o["emits_light"] for c in (o["position"][0], o["position"][2])]
|
| 60 |
+
+ [0.3])
|
| 61 |
+
ground = {"radius": round(1.6 * reach + 0.5, 3)}
|
| 62 |
+
data = {
|
| 63 |
+
"title": ledger["sentence"],
|
| 64 |
+
"reading": [{"word": t["text"], "kind": t["kind"], "commits": t["commits"]} for t in ledger["tokens"]],
|
| 65 |
+
"relations": [{"subject": i["subject"], "relation": i["relation"], "object": i["object"]}
|
| 66 |
+
for i in ledger["items"] if i["kind"] == "relation"],
|
| 67 |
+
"unresolved": [{"question": q.get("question"), "default": q.get("default_chosen")}
|
| 68 |
+
for q in ledger["unresolved"]],
|
| 69 |
+
"sky": _sky_hex(sun), "sun": sun, "ground": ground, "objects": objects,
|
| 70 |
+
}
|
| 71 |
+
return page(data)
|
| 72 |
+
|
| 73 |
+
|
| 74 |
+
def ladder_page(item, rungs):
|
| 75 |
+
"""Rungs left to right, from the most detailed embeddable one down to the box."""
|
| 76 |
+
objects, x = [], 0.0
|
| 77 |
+
for label, rung in rungs:
|
| 78 |
+
if len(rung.faces) > LADDER_MAX_FACES:
|
| 79 |
+
continue
|
| 80 |
+
hw = 0.5 * max(rung.extents[0], rung.extents[2])
|
| 81 |
+
objects.append(_object(f"rung{len(objects)}", item["word"], rung, [x + hw, 0, 0], 0,
|
| 82 |
+
budget=LADDER_MAX_FACES, label=label))
|
| 83 |
+
x += 2 * hw + 0.3 * max(rung.extents[1], 0.1)
|
| 84 |
+
for o in objects:
|
| 85 |
+
o["position"][0] = round(o["position"][0] - x / 2, 3)
|
| 86 |
+
data = {"title": f"{item['word']}: simplification ladder",
|
| 87 |
+
"caption": "left → right: " + " · ".join(o["label"] for o in objects),
|
| 88 |
+
"sky": "#2a3346", "sun": None, "ground": None, "objects": objects}
|
| 89 |
+
return page(data)
|
| 90 |
+
|
| 91 |
+
|
| 92 |
+
def page(data):
|
| 93 |
+
payload = json.dumps(data, separators=(",", ":")).replace("</", "<\\/")
|
| 94 |
+
return TEMPLATE.replace("__THREE__", THREE).replace("__TITLE__", _escape(data["title"])) \
|
| 95 |
+
.replace("__DATA__", payload)
|
| 96 |
+
|
| 97 |
+
|
| 98 |
+
def _escape(s):
|
| 99 |
+
return s.replace("&", "&").replace("<", "<").replace(">", ">")
|
| 100 |
+
|
| 101 |
+
|
| 102 |
+
TEMPLATE = """<!doctype html>
|
| 103 |
+
<html><head><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
|
| 104 |
+
<title>__TITLE__</title>
|
| 105 |
+
<style>html,body{margin:0;height:100%;overflow:hidden;font:13px system-ui,sans-serif}
|
| 106 |
+
#cap{position:fixed;left:12px;bottom:12px;color:#fff;background:#0008;padding:6px 10px;border-radius:6px}</style>
|
| 107 |
+
<script type="importmap">{"imports":{"three":"__THREE__build/three.module.js","three/addons/":"__THREE__examples/jsm/"}}</script>
|
| 108 |
+
</head><body><div id="cap"></div>
|
| 109 |
+
<script type="module">
|
| 110 |
+
import * as THREE from "three";
|
| 111 |
+
import {OrbitControls} from "three/addons/controls/OrbitControls.js";
|
| 112 |
+
|
| 113 |
+
// Scene data from VisualLearningIdea: positions in metres, Y up, yaw about Y.
|
| 114 |
+
// Objects carry inline low-poly geometry ("mesh") or a primitive spec.
|
| 115 |
+
const D = __DATA__;
|
| 116 |
+
|
| 117 |
+
const renderer = new THREE.WebGLRenderer({antialias: true});
|
| 118 |
+
renderer.setPixelRatio(Math.min(devicePixelRatio, 2));
|
| 119 |
+
renderer.setSize(innerWidth, innerHeight);
|
| 120 |
+
renderer.shadowMap.enabled = true;
|
| 121 |
+
document.body.appendChild(renderer.domElement);
|
| 122 |
+
const scene = new THREE.Scene();
|
| 123 |
+
scene.background = new THREE.Color(D.sky);
|
| 124 |
+
scene.add(new THREE.HemisphereLight(0xffffff, 0x445533, 0.9));
|
| 125 |
+
const camera = new THREE.PerspectiveCamera(45, innerWidth / innerHeight, 0.01, 1000);
|
| 126 |
+
|
| 127 |
+
function geometryOf(o) {
|
| 128 |
+
if (o.primitive) {
|
| 129 |
+
const [w, h, d] = o.primitive.size;
|
| 130 |
+
const g = {sphere: () => new THREE.SphereGeometry(h / 2, 24, 16),
|
| 131 |
+
cylinder: () => new THREE.CylinderGeometry(w / 2, w / 2, h, 24),
|
| 132 |
+
cone: () => new THREE.ConeGeometry(w / 2, h, 24),
|
| 133 |
+
plane: () => new THREE.BoxGeometry(w, h, d),
|
| 134 |
+
box: () => new THREE.BoxGeometry(w, h, d)}[o.primitive.shape]();
|
| 135 |
+
g.translate(0, h / 2, 0); // rest on y = 0 like every asset
|
| 136 |
+
return g;
|
| 137 |
+
}
|
| 138 |
+
const g = new THREE.BufferGeometry();
|
| 139 |
+
g.setAttribute("position", new THREE.Float32BufferAttribute(o.mesh.p, 3));
|
| 140 |
+
g.setIndex(o.mesh.i);
|
| 141 |
+
g.computeVertexNormals();
|
| 142 |
+
return g;
|
| 143 |
+
}
|
| 144 |
+
|
| 145 |
+
const frame = new THREE.Box3();
|
| 146 |
+
for (const o of D.objects) {
|
| 147 |
+
const mat = o.emits_light
|
| 148 |
+
? new THREE.MeshBasicMaterial({color: o.color})
|
| 149 |
+
: new THREE.MeshStandardMaterial({color: o.color, roughness: 0.85, flatShading: true, side: THREE.DoubleSide});
|
| 150 |
+
const m = new THREE.Mesh(geometryOf(o), mat);
|
| 151 |
+
m.name = o.id;
|
| 152 |
+
m.position.fromArray(o.position);
|
| 153 |
+
m.rotation.y = THREE.MathUtils.degToRad(o.yaw_deg);
|
| 154 |
+
m.castShadow = !o.emits_light;
|
| 155 |
+
m.receiveShadow = true;
|
| 156 |
+
scene.add(m);
|
| 157 |
+
if (!o.emits_light) frame.expandByObject(m);
|
| 158 |
+
}
|
| 159 |
+
if (frame.isEmpty()) frame.setFromCenterAndSize(new THREE.Vector3(), new THREE.Vector3(1, 1, 1));
|
| 160 |
+
const center = frame.getCenter(new THREE.Vector3());
|
| 161 |
+
const radius = Math.max(frame.getSize(new THREE.Vector3()).length(), 0.3);
|
| 162 |
+
|
| 163 |
+
if (D.ground) {
|
| 164 |
+
const g = new THREE.Mesh(new THREE.CircleGeometry(D.ground.radius, 64),
|
| 165 |
+
new THREE.MeshStandardMaterial({color: 0x4d7c3a, roughness: 1}));
|
| 166 |
+
g.rotation.x = -Math.PI / 2;
|
| 167 |
+
g.receiveShadow = true;
|
| 168 |
+
scene.add(g);
|
| 169 |
+
}
|
| 170 |
+
|
| 171 |
+
const sun = new THREE.DirectionalLight(0xfff1d0, 2.2);
|
| 172 |
+
if (D.sun) {
|
| 173 |
+
const el = THREE.MathUtils.degToRad(D.sun.elevation_deg), az = THREE.MathUtils.degToRad(D.sun.azimuth_deg);
|
| 174 |
+
const dir = new THREE.Vector3(Math.cos(el) * Math.sin(az), Math.sin(el), Math.cos(el) * Math.cos(az));
|
| 175 |
+
sun.position.copy(center).addScaledVector(dir, radius * 3);
|
| 176 |
+
sun.castShadow = !!D.sun.shadows;
|
| 177 |
+
} else {
|
| 178 |
+
sun.position.copy(center).add(new THREE.Vector3(radius, radius * 2, radius * 1.2));
|
| 179 |
+
}
|
| 180 |
+
sun.target.position.copy(center);
|
| 181 |
+
Object.assign(sun.shadow.camera, {left: -radius * 1.5, right: radius * 1.5, top: radius * 1.5,
|
| 182 |
+
bottom: -radius * 1.5, far: radius * 8});
|
| 183 |
+
sun.shadow.mapSize.set(1024, 1024);
|
| 184 |
+
scene.add(sun, sun.target);
|
| 185 |
+
|
| 186 |
+
camera.position.set(center.x + radius * 0.9, center.y + radius * 0.6, center.z + radius * 1.2);
|
| 187 |
+
const controls = new OrbitControls(camera, renderer.domElement);
|
| 188 |
+
controls.target.copy(center);
|
| 189 |
+
controls.update();
|
| 190 |
+
|
| 191 |
+
document.getElementById("cap").textContent = D.caption || D.title;
|
| 192 |
+
addEventListener("resize", () => {
|
| 193 |
+
camera.aspect = innerWidth / innerHeight;
|
| 194 |
+
camera.updateProjectionMatrix();
|
| 195 |
+
renderer.setSize(innerWidth, innerHeight);
|
| 196 |
+
});
|
| 197 |
+
renderer.setAnimationLoop(() => { controls.update(); renderer.render(scene, camera); });
|
| 198 |
+
</script></body></html>
|
| 199 |
+
"""
|
vli/lexicon.py
CHANGED
|
@@ -37,22 +37,24 @@ def _split_part(part):
|
|
| 37 |
|
| 38 |
|
| 39 |
def simready_phrases(name):
|
| 40 |
-
"""Words an asset name like '
|
| 41 |
|
| 42 |
-
|
| 43 |
-
|
|
|
|
| 44 |
"""
|
| 45 |
parts = [p for p in name.split("_") if p and not VARIANT.fullmatch(p)]
|
|
|
|
| 46 |
out = []
|
| 47 |
-
for
|
| 48 |
-
|
| 49 |
-
|
| 50 |
-
|
| 51 |
-
|
| 52 |
-
out.append((" ".join(
|
| 53 |
-
if len(
|
| 54 |
-
out.append((
|
| 55 |
-
return out
|
| 56 |
|
| 57 |
|
| 58 |
def objaverse_word(category):
|
|
|
|
| 37 |
|
| 38 |
|
| 39 |
def simready_phrases(name):
|
| 40 |
+
"""Words an asset name like 'interiorPart_sun_visor_a01' can be looked up by.
|
| 41 |
|
| 42 |
+
Names read category_object..., where the object can span several parts
|
| 43 |
+
('sun_visor'). Only whole phrases and the final part are indexed, never a
|
| 44 |
+
lone modifier: 'sun' must not find a sun visor.
|
| 45 |
"""
|
| 46 |
parts = [p for p in name.split("_") if p and not VARIANT.fullmatch(p)]
|
| 47 |
+
body = parts[1:] or parts # drop the category prefix
|
| 48 |
out = []
|
| 49 |
+
phrase = " ".join(w for p in body for w in _split_part(p))
|
| 50 |
+
if phrase:
|
| 51 |
+
out.append((phrase, 0)) # 'sun visor', 'hand dryer automatic'
|
| 52 |
+
final = _split_part(body[-1]) if body else []
|
| 53 |
+
if final:
|
| 54 |
+
out.append((" ".join(final), 0)) # 'visor', 'bucky table'
|
| 55 |
+
if len(final) > 1:
|
| 56 |
+
out.append((final[-1], 1)) # 'table' from 'buckyTable': weaker
|
| 57 |
+
return list(dict.fromkeys(out))
|
| 58 |
|
| 59 |
|
| 60 |
def objaverse_word(category):
|
vli/pipeline.py
CHANGED
|
@@ -9,7 +9,7 @@ import tempfile
|
|
| 9 |
|
| 10 |
import numpy as np
|
| 11 |
|
| 12 |
-
from . import assets, scene
|
| 13 |
from .grammar import build_ledger, candidate_label, next_tier, parse_primitives, primitive_candidate
|
| 14 |
|
| 15 |
|
|
@@ -40,7 +40,7 @@ def outputs(ledger, seed):
|
|
| 40 |
work = _workdir()
|
| 41 |
png = scene.render_scene(ledger, meshes, config, os.path.join(work, "scene.png"))
|
| 42 |
glb = scene.export_glb(ledger, meshes, config, os.path.join(work, "scene.glb"))
|
| 43 |
-
return json.dumps(ledger, indent=1), png, glb
|
| 44 |
|
| 45 |
|
| 46 |
def translate(sentence, primitives=None, seed=0):
|
|
|
|
| 9 |
|
| 10 |
import numpy as np
|
| 11 |
|
| 12 |
+
from . import assets, html, scene
|
| 13 |
from .grammar import build_ledger, candidate_label, next_tier, parse_primitives, primitive_candidate
|
| 14 |
|
| 15 |
|
|
|
|
| 40 |
work = _workdir()
|
| 41 |
png = scene.render_scene(ledger, meshes, config, os.path.join(work, "scene.png"))
|
| 42 |
glb = scene.export_glb(ledger, meshes, config, os.path.join(work, "scene.glb"))
|
| 43 |
+
return json.dumps(ledger, indent=1), png, glb, html.scene_page(ledger, meshes, config)
|
| 44 |
|
| 45 |
|
| 46 |
def translate(sentence, primitives=None, seed=0):
|
vli/scene.py
CHANGED
|
@@ -15,6 +15,7 @@ from scipy.spatial import ConvexHull # noqa: E402
|
|
| 15 |
from .simplify import base_color, decimate # noqa: E402
|
| 16 |
|
| 17 |
GAP = 0.1
|
|
|
|
| 18 |
RENDER_FACE_BUDGET = 3000
|
| 19 |
|
| 20 |
|
|
@@ -46,7 +47,7 @@ def configure(ledger, meshes, seed=0):
|
|
| 46 |
|
| 47 |
sun = None
|
| 48 |
if lights:
|
| 49 |
-
sun = {"id": lights[0]["id"], "elevation_deg": float(rng.uniform(
|
| 50 |
"azimuth_deg": float(rng.uniform(-150, 150)),
|
| 51 |
"shadows": any(r["relation"] == "lights" for r in relations)}
|
| 52 |
|
|
@@ -89,9 +90,10 @@ def configure(ledger, meshes, seed=0):
|
|
| 89 |
|
| 90 |
if sun: # whatever shone sits in the sky along the light direction
|
| 91 |
reach = max([abs(c) + _half_width(meshes[o]) for o, p in place.items()
|
| 92 |
-
for c in (p["position"][0], p["position"][2])] + [
|
| 93 |
-
pos = sun_direction(sun) * 1.3 * reach # shadows use the direction; distance is only for the view
|
| 94 |
for it in lights:
|
|
|
|
|
|
|
| 95 |
place[it["id"]] = {"position": [float(c) for c in pos], "yaw_deg": 0.0, "light": True}
|
| 96 |
return {"seed": seed, "placements": place, "sun": sun}
|
| 97 |
|
|
@@ -106,7 +108,7 @@ def _ground(ledger, placed):
|
|
| 106 |
if not any(i["kind"] == "environment" and i["role"] == "plane" for i in ledger["items"]):
|
| 107 |
return None
|
| 108 |
lights = {i["id"] for i in ledger["items"] if i.get("emits_light")}
|
| 109 |
-
reach = max([np.abs(m.bounds[:, [0, 2]]).max() for oid, m in placed.items() if oid not in lights] + [
|
| 110 |
g = trimesh.creation.cylinder(radius=reach * 1.6, height=0.01, sections=48)
|
| 111 |
g.apply_transform(trimesh.transformations.rotation_matrix(math.pi / 2, [1, 0, 0]))
|
| 112 |
g.apply_translation([0, -0.005, 0])
|
|
@@ -119,6 +121,8 @@ def placed_meshes(ledger, meshes, config):
|
|
| 119 |
for oid, p in config["placements"].items():
|
| 120 |
m = meshes[oid].copy()
|
| 121 |
m.apply_transform(transform(p))
|
|
|
|
|
|
|
| 122 |
out[oid] = m
|
| 123 |
return out
|
| 124 |
|
|
@@ -150,9 +154,11 @@ def _to_axes(v):
|
|
| 150 |
return np.stack([v[..., 0], -v[..., 2], v[..., 1]], axis=-1)
|
| 151 |
|
| 152 |
|
| 153 |
-
def _shade(mesh, light, color):
|
| 154 |
-
|
| 155 |
-
|
|
|
|
|
|
|
| 156 |
rgb = np.clip(color[:3] / 255.0 * k[:, None], 0, 1)
|
| 157 |
return np.c_[rgb, np.ones(len(rgb))]
|
| 158 |
|
|
@@ -187,11 +193,11 @@ def render(meshes_by_id, path, sun=None, ground=None, title=None):
|
|
| 187 |
ax.add_collection3d(Poly3DCollection([_to_axes(poly)], facecolors=[(0, 0, 0, 0.35)], zorder=1))
|
| 188 |
|
| 189 |
tris, colors = [], []
|
| 190 |
-
for m in meshes_by_id.
|
| 191 |
color = base_color(m)
|
| 192 |
r = decimate(m, RENDER_FACE_BUDGET) if len(m.faces) > RENDER_FACE_BUDGET else m
|
| 193 |
tris.append(r.triangles)
|
| 194 |
-
colors.append(_shade(r, light, color))
|
| 195 |
pts.append(r.vertices)
|
| 196 |
if tris:
|
| 197 |
ax.add_collection3d(Poly3DCollection(_to_axes(np.concatenate(tris)), facecolors=np.concatenate(colors),
|
|
|
|
| 15 |
from .simplify import base_color, decimate # noqa: E402
|
| 16 |
|
| 17 |
GAP = 0.1
|
| 18 |
+
GLOW = [255, 214, 107, 255]
|
| 19 |
RENDER_FACE_BUDGET = 3000
|
| 20 |
|
| 21 |
|
|
|
|
| 47 |
|
| 48 |
sun = None
|
| 49 |
if lights:
|
| 50 |
+
sun = {"id": lights[0]["id"], "elevation_deg": float(rng.uniform(25, 65)),
|
| 51 |
"azimuth_deg": float(rng.uniform(-150, 150)),
|
| 52 |
"shadows": any(r["relation"] == "lights" for r in relations)}
|
| 53 |
|
|
|
|
| 90 |
|
| 91 |
if sun: # whatever shone sits in the sky along the light direction
|
| 92 |
reach = max([abs(c) + _half_width(meshes[o]) for o, p in place.items()
|
| 93 |
+
for c in (p["position"][0], p["position"][2])] + [0.3])
|
|
|
|
| 94 |
for it in lights:
|
| 95 |
+
# shadows use the direction; distance only frames the view
|
| 96 |
+
pos = sun_direction(sun) * (1.5 * reach + 2 * _half_width(meshes[it["id"]]))
|
| 97 |
place[it["id"]] = {"position": [float(c) for c in pos], "yaw_deg": 0.0, "light": True}
|
| 98 |
return {"seed": seed, "placements": place, "sun": sun}
|
| 99 |
|
|
|
|
| 108 |
if not any(i["kind"] == "environment" and i["role"] == "plane" for i in ledger["items"]):
|
| 109 |
return None
|
| 110 |
lights = {i["id"] for i in ledger["items"] if i.get("emits_light")}
|
| 111 |
+
reach = max([np.abs(m.bounds[:, [0, 2]]).max() for oid, m in placed.items() if oid not in lights] + [0.3])
|
| 112 |
g = trimesh.creation.cylinder(radius=reach * 1.6, height=0.01, sections=48)
|
| 113 |
g.apply_transform(trimesh.transformations.rotation_matrix(math.pi / 2, [1, 0, 0]))
|
| 114 |
g.apply_translation([0, -0.005, 0])
|
|
|
|
| 121 |
for oid, p in config["placements"].items():
|
| 122 |
m = meshes[oid].copy()
|
| 123 |
m.apply_transform(transform(p))
|
| 124 |
+
if p.get("light"): # a light source glows: warm, flat colour
|
| 125 |
+
m.visual = trimesh.visual.ColorVisuals(m, face_colors=np.tile(GLOW, (len(m.faces), 1)))
|
| 126 |
out[oid] = m
|
| 127 |
return out
|
| 128 |
|
|
|
|
| 154 |
return np.stack([v[..., 0], -v[..., 2], v[..., 1]], axis=-1)
|
| 155 |
|
| 156 |
|
| 157 |
+
def _shade(mesh, light, color, glow=False):
|
| 158 |
+
if glow:
|
| 159 |
+
k = np.ones(len(mesh.faces))
|
| 160 |
+
else: # two-sided: Objaverse meshes often have flipped faces
|
| 161 |
+
k = 0.45 + 0.55 * np.abs(mesh.face_normals @ light)
|
| 162 |
rgb = np.clip(color[:3] / 255.0 * k[:, None], 0, 1)
|
| 163 |
return np.c_[rgb, np.ones(len(rgb))]
|
| 164 |
|
|
|
|
| 193 |
ax.add_collection3d(Poly3DCollection([_to_axes(poly)], facecolors=[(0, 0, 0, 0.35)], zorder=1))
|
| 194 |
|
| 195 |
tris, colors = [], []
|
| 196 |
+
for oid, m in meshes_by_id.items():
|
| 197 |
color = base_color(m)
|
| 198 |
r = decimate(m, RENDER_FACE_BUDGET) if len(m.faces) > RENDER_FACE_BUDGET else m
|
| 199 |
tris.append(r.triangles)
|
| 200 |
+
colors.append(_shade(r, light, color, glow=bool(sun) and oid == sun["id"]))
|
| 201 |
pts.append(r.vertices)
|
| 202 |
if tris:
|
| 203 |
ax.add_collection3d(Poly3DCollection(_to_axes(np.concatenate(tris)), facecolors=np.concatenate(colors),
|
vli/vlm.py
DELETED
|
@@ -1,35 +0,0 @@
|
|
| 1 |
-
"""Back-translation: a vision model reads the picture back into English.
|
| 2 |
-
|
| 3 |
-
Calls always use the *caller's* token, never one stored on the Space, so a
|
| 4 |
-
public Space can't spend its owner's inference credits.
|
| 5 |
-
"""
|
| 6 |
-
import base64
|
| 7 |
-
import os
|
| 8 |
-
|
| 9 |
-
from huggingface_hub import InferenceClient
|
| 10 |
-
|
| 11 |
-
VLM_MODEL = os.environ.get("VLM_MODEL", "Qwen/Qwen2.5-VL-7B-Instruct")
|
| 12 |
-
SIGN_IN = ("The vision model runs on your own Hugging Face account: sign in first "
|
| 13 |
-
"(MCP callers: send your HF token as a Bearer Authorization header).")
|
| 14 |
-
|
| 15 |
-
|
| 16 |
-
def ask(image_path, prompt, token, max_tokens=80):
|
| 17 |
-
if not token:
|
| 18 |
-
raise ValueError(SIGN_IN)
|
| 19 |
-
with open(image_path, "rb") as f:
|
| 20 |
-
data = base64.b64encode(f.read()).decode()
|
| 21 |
-
client = InferenceClient(token=token)
|
| 22 |
-
resp = client.chat_completion(
|
| 23 |
-
model=VLM_MODEL, max_tokens=max_tokens,
|
| 24 |
-
messages=[{"role": "user", "content": [
|
| 25 |
-
{"type": "image_url", "image_url": {"url": f"data:image/png;base64,{data}"}},
|
| 26 |
-
{"type": "text", "text": prompt}]}])
|
| 27 |
-
return resp.choices[0].message.content.strip()
|
| 28 |
-
|
| 29 |
-
|
| 30 |
-
def describe(image_path, token):
|
| 31 |
-
return ask(image_path, "Describe this 3D scene in one short plain sentence.", token)
|
| 32 |
-
|
| 33 |
-
|
| 34 |
-
def name_object(image_path, token):
|
| 35 |
-
return ask(image_path, "What single object is this? Answer with one or two words.", token, max_tokens=10)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|