kwabs22 Claude Opus 5.5 commited on
Commit
7bb2fda
·
1 Parent(s): d7aa77a

Return a mini HTML page with every picture; drop the Space's vision model

Browse files

The Space visualises and the chat discusses: translate, refine and simplify
now also return a self-contained Three.js page with inline low-poly geometry
for the calling model to build on. Removes describe_scene, the recognition
check, the login button (which leaked an MCP tool) and hf_oauth.

Also from the HuggingChat test: SimReady names no longer index lone
modifiers ("sun" found a sun visor), check_words shows what each noun maps
to, renders frame on the objects, light both faces, and the light glows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Files changed (9) hide show
  1. README.md +0 -3
  2. SPEC.md +28 -13
  3. app.py +65 -95
  4. vli/grammar.py +1 -0
  5. vli/html.py +199 -0
  6. vli/lexicon.py +14 -12
  7. vli/pipeline.py +2 -2
  8. vli/scene.py +15 -9
  9. vli/vlm.py +0 -35
README.md CHANGED
@@ -8,9 +8,6 @@ sdk_version: 6.27.0
8
  python_version: '3.12'
9
  app_file: app.py
10
  pinned: false
11
- hf_oauth: true
12
- hf_oauth_scopes:
13
- - inference-api
14
  short_description: Visualisation and speed of input attempt
15
  ---
16
 
 
8
  python_version: '3.12'
9
  app_file: app.py
10
  pinned: false
 
 
 
11
  short_description: Visualisation and speed of input attempt
12
  ---
13
 
SPEC.md CHANGED
@@ -14,6 +14,11 @@ a mesh can be reduced and still mean its word, and that is a main output.
14
  One Gradio Space serves people (UI) and models (`demo.launch(mcp_server=True)`).
15
  HuggingChat models can't run tools, which is why this Space exists.
16
 
 
 
 
 
 
17
  ## Translation terms
18
 
19
  | Translation | Here |
@@ -24,7 +29,7 @@ HuggingChat models can't run tools, which is why this Space exists.
24
  | Verbs | Relations between meshes |
25
  | Word alignment | `tokens[].commits`: reading order differs from build order |
26
  | Forced choices | `unresolved`: which asset, how big |
27
- | Back-translation | `describe_scene`: a vision model reads the picture back |
28
 
29
  ## Dictionary
30
 
@@ -92,26 +97,36 @@ Verbs: `sat/stood/lay/rested` → `on` (ground implied), `shone/glowed` → `lig
92
  | Tool | In → out |
93
  |---|---|
94
  | `check_words` | sentence, primitives → per-word status, missing words, completions |
95
- | `translate_to_scene` | sentence, primitives, seed → ledger, PNG, GLB |
96
- | `refine_scene` | ledger, changes, seed → ledger, PNG, GLB |
97
- | `simplify_item` | ledger, item id, check_recognition → report, rung PNGs, rung GLBs |
98
- | `describe_scene` | ledger, seed → back-translation JSON |
 
 
99
 
100
  Ladder rungs: original → 20k / 5k / 1k / 250 / 60 faces → convex hull → oriented
101
- box. With `check_recognition`, a vision model names each rung, and
102
- `simplest_recognized` is the lowest rung that still reads as the word.
 
 
 
 
 
 
 
 
 
103
 
104
- **No secrets.** The Space is public, so it stores no token. Datasets are public
105
- and download without one. Vision-model calls (back-translation, recognition) use
106
- the **caller's** token: HF sign-in in the UI (`hf_oauth`, `inference-api` scope),
107
- or a Bearer `Authorization` header from MCP clients. Optional env: `VLM_MODEL`, `VLI_CACHE`.
108
 
109
  ## Known gaps / open questions
110
 
111
  - **Materialization** (papercraft from a low rung) is not built yet.
112
  - **Generation tier** is a stub: rejecting a primitive errors out.
113
- - SimReady names → words is a heuristic: the second `_` part is the head word
114
- (`appliance_refrigerator_a01` → refrigerator). Some names will mis-index.
 
115
  - Objaverse "front" direction is unknown, so `faces` assumes +Z forward.
116
  - `on` uses the bounding-box top, so objects on irregular tops may float or sink.
117
  - Renders are matplotlib (headless, CPU): approximate shading and a flat shadow on
 
14
  One Gradio Space serves people (UI) and models (`demo.launch(mcp_server=True)`).
15
  HuggingChat models can't run tools, which is why this Space exists.
16
 
17
+ **The Space visualises; discussion happens in the chat** (e.g. HuggingChat over MCP).
18
+ The calling model already sees the picture, so the Space runs no vision model of
19
+ its own and stores no token. Because HTML is the final destination, every picture
20
+ comes back with a **mini HTML page** the model can read, edit and build on.
21
+
22
  ## Translation terms
23
 
24
  | Translation | Here |
 
29
  | Verbs | Relations between meshes |
30
  | Word alignment | `tokens[].commits`: reading order differs from build order |
31
  | Forced choices | `unresolved`: which asset, how big |
32
+ | Back-translation | The chat model reading the picture: done in the discussion, not by the Space |
33
 
34
  ## Dictionary
35
 
 
97
  | Tool | In → out |
98
  |---|---|
99
  | `check_words` | sentence, primitives → per-word status, missing words, completions |
100
+ | `translate_to_scene` | sentence, primitives, seed → ledger, PNG, GLB, mini HTML |
101
+ | `refine_scene` | ledger, changes, seed → ledger, PNG, GLB, mini HTML |
102
+ | `simplify_item` | ledger, item id → report, rung PNGs, rung GLBs, mini HTML |
103
+
104
+ `check_words` also returns `maps_to` per noun (the asset it would get), so a word
105
+ that exists but maps to the wrong thing shows up before submitting.
106
 
107
  Ladder rungs: original → 20k / 5k / 1k / 250 / 60 faces → convex hull → oriented
108
+ box. Where the object stops reading as its word is judged in the discussion.
109
+
110
+ ### Mini HTML
111
+
112
+ A self-contained page: Three.js (`three@0.160.0` from unpkg) plus one inline `D`
113
+ data object holding the sentence, the reading (word → commits), relations,
114
+ unresolved questions, sun, ground, and each object's position, yaw and colour.
115
+ Each object also has either inline low-poly geometry (≤400 faces in a scene,
116
+ rungs ≤1000 faces in a ladder) or a primitive spec. The geometry is inline, so the
117
+ page works after Gradio's temporary file URLs expire, and it stays small enough
118
+ for a model's context.
119
 
120
+ **No secrets.** The Space is public and stores no token; the datasets are public
121
+ and download without one. Optional env: `VLI_CACHE`.
 
 
122
 
123
  ## Known gaps / open questions
124
 
125
  - **Materialization** (papercraft from a low rung) is not built yet.
126
  - **Generation tier** is a stub: rejecting a primitive errors out.
127
+ - SimReady names → words is a heuristic: the name after its category is indexed
128
+ as a whole phrase plus its final part (`interiorPart_sun_visor_a01` → "sun visor",
129
+ "visor"; never "sun" alone). `maps_to` makes the remaining misses visible.
130
  - Objaverse "front" direction is unknown, so `faces` assumes +Z forward.
131
  - `on` uses the bounding-box top, so objects on irregular tops may float or sink.
132
  - Renders are matplotlib (headless, CPU): approximate shading and a flat shadow on
app.py CHANGED
@@ -2,7 +2,10 @@
2
 
3
  One Space for people (this UI) and for models (every public event below is an
4
  MCP tool: demo.launch(mcp_server=True)). Helper events are api_visibility="private".
 
 
5
  """
 
6
  import json
7
  import os
8
  import re
@@ -12,7 +15,7 @@ import threading
12
  import spaces # ZeroGPU: import before any CUDA-touching library
13
  import gradio as gr
14
 
15
- from vli import assets, pipeline, scene, simplify, vlm
16
  from vli.grammar import check_words as _check_words
17
  from vli.lexicon import LEXICON, PRIMITIVE_SHAPES
18
 
@@ -35,14 +38,6 @@ def _json(text, name, default=None):
35
  raise ValueError(f"{name} is not valid JSON")
36
 
37
 
38
- def _user_token(oauth_token, request):
39
- """The caller's own token: from HF sign-in (UI) or a Bearer header (MCP). Never the Space's."""
40
- if oauth_token is not None and oauth_token.token:
41
- return oauth_token.token
42
- auth = (request.headers.get("authorization", "") if request is not None else "")
43
- return auth[7:].strip() if auth.lower().startswith("bearer ") else None
44
-
45
-
46
  def _seed(value):
47
  """Seeds arrive as strings (some MCP clients can't send numbers)."""
48
  try:
@@ -66,11 +61,11 @@ def generate_mesh(prompt):
66
  def check_words(sentence: str, primitives_json: str = "{}") -> str:
67
  """Check which words of a sentence exist in the 3D vocabulary, and complete the word being typed.
68
 
69
- A word is available only when a ready asset (SimReady or Objaverse) exists for it.
70
- Words with no asset are listed in "missing": rephrase them, or map each one to a
71
- primitive shape in primitives_json. Verbs (sat, stood, lay, shone, faced) and
72
- prepositions (on, under, beside, near, behind, in front of, above) are relations
73
- between meshes; whatever "shone" becomes the light source.
74
 
75
  Args:
76
  sentence: The English sentence, complete or partly typed.
@@ -78,7 +73,7 @@ def check_words(sentence: str, primitives_json: str = "{}") -> str:
78
  Shapes: box, sphere, cylinder, cone, plane.
79
 
80
  Returns:
81
- JSON string: {"ok", "words": [{text, kind, available_in | primitive}], "missing",
82
  "primitive_shapes", "completions"}.
83
  """
84
  try:
@@ -87,13 +82,14 @@ def check_words(sentence: str, primitives_json: str = "{}") -> str:
87
  _fail(e)
88
 
89
 
90
- def translate_to_scene(sentence: str, primitives_json: str = "{}", seed: str = "0") -> tuple[str, str, str]:
91
  """Translate an English sentence into a 3D scene.
92
 
93
  Produces the serial ledger (tokens in reading order, the items each word commits,
94
- and the unresolved "which?" questions), one configuration of that ledger chosen by
95
- `seed`, a rendered picture and a GLB model. Words with no asset must be mapped to a
96
- primitive in primitives_json (call check_words first to see which are missing).
 
97
 
98
  Args:
99
  sentence: A sentence using available words, e.g. "the cat sat and the sun shone".
@@ -101,7 +97,7 @@ def translate_to_scene(sentence: str, primitives_json: str = "{}", seed: str = "
101
  seed: Picks one of the configurations that satisfy the ledger (positions, facing, sun).
102
 
103
  Returns:
104
- (ledger JSON string, rendered PNG, GLB scene file).
105
  """
106
  try:
107
  return pipeline.translate(sentence, _json(primitives_json, "primitives_json", {}), _seed(seed))
@@ -109,17 +105,19 @@ def translate_to_scene(sentence: str, primitives_json: str = "{}", seed: str = "
109
  _fail(e)
110
 
111
 
112
- def refine_scene(ledger_json: str, changes_json: str = "[]", seed: str = "0") -> tuple[str, str, str]:
113
  """Refine a ledger: freeze some decisions and change the others, then re-render.
114
 
115
  Changes are a JSON list of:
116
  {"id": "n1", "freeze": true} lock an item (false unlocks)
117
- {"id": "n1", "use": 2} pick alternatives[2] as the asset
 
118
  {"id": "n1", "reject": true} reject the tier: simready -> objaverse -> primitive -> generation
119
  {"id": "n1", "height_m": 0.4} set the object's height in metres
120
  {"id": "n1", "shape": "sphere"} use a primitive (box, sphere, cylinder, cone, plane)
121
  {"reroll": true} every unfrozen object takes a random alternative
122
- Frozen items cannot be changed. The configuration (placement) is always re-chosen from `seed`.
 
123
 
124
  Args:
125
  ledger_json: Ledger JSON from translate_to_scene or a previous refine_scene.
@@ -127,7 +125,7 @@ def refine_scene(ledger_json: str, changes_json: str = "[]", seed: str = "0") ->
127
  seed: Picks the configuration, and the alternatives for a reroll.
128
 
129
  Returns:
130
- (ledger JSON string, rendered PNG, GLB scene file).
131
  """
132
  try:
133
  ledger = _json(ledger_json, "ledger_json")
@@ -137,80 +135,39 @@ def refine_scene(ledger_json: str, changes_json: str = "[]", seed: str = "0") ->
137
  _fail(e)
138
 
139
 
140
- def simplify_item(ledger_json: str, item_id: str, check_recognition: bool = False,
141
- oauth_token: gr.OAuthToken | None = None, request: gr.Request = None) -> tuple[str, list, list]:
142
  """Build the simplification ladder for one object: full asset -> fewer faces -> convex hull -> box.
143
 
144
- With check_recognition, a vision model names each rung's picture; the lowest rung
145
- still named as the word is the simplest form that keeps its meaning.
 
146
 
147
  Args:
148
  ledger_json: Ledger JSON from translate_to_scene or refine_scene.
149
  item_id: The object item to simplify, e.g. "n1".
150
- check_recognition: Ask a vision model to name each rung (slower; runs on the caller's
151
- own HF account, so it needs sign-in or a Bearer token).
152
 
153
  Returns:
154
- (report JSON string, gallery of rung pictures, GLB file per rung).
 
155
  """
156
  try:
157
- token = _user_token(oauth_token, request) if check_recognition else None
158
- if check_recognition and not token:
159
- raise ValueError(vlm.SIGN_IN) # before any heavy work
160
  ledger = _json(ledger_json, "ledger_json")
161
  item = next((i for i in ledger["items"] if i["id"] == item_id), None)
162
  if item is None or item["kind"] != "object":
163
  raise ValueError(f"{item_id!r} is not an object item in this ledger")
164
  mesh = assets.settle(item)
165
  work = tempfile.mkdtemp(prefix="vli_ladder_")
 
166
  report, gallery, files = {"item": item_id, "word": item["word"], "rungs": []}, [], []
167
- for k, (label, rung) in enumerate(simplify.ladder(mesh)):
168
  png = scene.render({item_id: rung}, os.path.join(work, f"{k}.png"), title=label)
169
  slug = re.sub(r"\W+", "_", label)
170
  glb = os.path.join(work, f"{k}_{slug}.glb")
171
  rung.export(glb)
172
- entry = {"rung": label, **simplify.stats(rung)}
173
- if check_recognition:
174
- answer = vlm.name_object(png, token)
175
- entry["named_as"] = answer
176
- entry["recognized"] = item["word"].lower() in answer.lower()
177
- report["rungs"].append(entry)
178
  gallery.append((png, label))
179
  files.append(glb)
180
- if check_recognition:
181
- kept = [r for r in report["rungs"] if r["recognized"]]
182
- report["simplest_recognized"] = kept[-1]["rung"] if kept else None
183
- return json.dumps(report, indent=1), gallery, files
184
- except Exception as e:
185
- _fail(e)
186
-
187
-
188
- def describe_scene(ledger_json: str, seed: str = "0",
189
- oauth_token: gr.OAuthToken | None = None, request: gr.Request = None) -> str:
190
- """Back-translate: render the ledger's scene and have a vision model describe it in English.
191
-
192
- Comparing the description with the original sentence checks the translation.
193
- The vision model runs on the caller's own HF account (sign-in or Bearer token).
194
-
195
- Args:
196
- ledger_json: Ledger JSON from translate_to_scene or refine_scene.
197
- seed: Which configuration to render.
198
-
199
- Returns:
200
- JSON string: {"original", "back_translation", "nouns_recovered": {word: bool}}.
201
- """
202
- try:
203
- token = _user_token(oauth_token, request)
204
- if not token:
205
- raise ValueError(vlm.SIGN_IN) # before any heavy work
206
- ledger = _json(ledger_json, "ledger_json")
207
- meshes = pipeline.settle_all(ledger)
208
- config = scene.configure(ledger, meshes, _seed(seed))
209
- png = scene.render_scene(ledger, meshes, config, os.path.join(tempfile.mkdtemp(), "scene.png"))
210
- text = vlm.describe(png, token)
211
- nouns = [i["word"] for i in ledger["items"] if i["kind"] == "object"]
212
- return json.dumps({"original": ledger["sentence"], "back_translation": text,
213
- "nouns_recovered": {n: n in text.lower() for n in nouns}}, indent=1)
214
  except Exception as e:
215
  _fail(e)
216
 
@@ -224,12 +181,12 @@ def _suggestions(words_json):
224
  if w["kind"] == "unknown":
225
  marks.append(f"~~{w['text']}~~")
226
  elif w["kind"] == "noun":
227
- marks.append(f"**{w['text']}**")
228
  elif w["kind"] == "primitive":
229
  marks.append(f"_{w['text']}_ ({w['primitive']})")
230
  else:
231
  marks.append(w["text"])
232
- status = " ".join(marks) or "_start typing_"
233
  if data.get("missing"):
234
  status += " \n_Struck-through words have no asset: pick a suggestion, rephrase, or choose a primitive._"
235
  missing = data.get("missing", [])
@@ -250,10 +207,22 @@ def _use_primitive(primitives_json, word, shape):
250
  return json.dumps(prims)
251
 
252
 
 
 
 
 
 
 
 
 
 
 
 
 
253
  EXAMPLE_CHANGES = '[\n {"id": "n1", "freeze": true},\n {"reroll": true}\n]'
254
 
255
  DATASETS_MD = """
256
- **Dictionary (public datasets, read-only):**
257
  1. [NVIDIA SimReady](https://github.com/NVIDIA-Omniverse/simready-central): metric OpenUSD assets,
258
  e.g. [nvidia/simready-assets](https://huggingface.co/datasets/nvidia/simready-assets). License per package
259
  (e.g. CC-BY-4.0). Only geometry and the thumbnail are downloaded, not the ~1 GB of textures.
@@ -261,17 +230,14 @@ DATASETS_MD = """
261
  License per object; non-commercial (NC) and no-derivatives (ND) objects are skipped.
262
  3. Missing words: a primitive you choose (box, sphere, cylinder, cone, plane). Generation comes later.
263
 
264
- **Why sign in?** Building scenes needs no account: the datasets are public. Only *Back-translation* and
265
- *Check recognition* call a vision model ([Qwen2.5-VL](https://huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct)
266
- via HF Inference), and those calls run on **your own** Hugging Face account. This Space stores no token,
267
- so nobody's credits are spent but the caller's.
268
  """
269
 
270
  with gr.Blocks(title="VisualLearningIdea") as demo:
271
  gr.Markdown("# English → 3D\nType with the words the 3D dictionary has. **Bold** = an object asset; "
272
  "verbs and prepositions are relations between meshes.")
273
  gr.Markdown(DATASETS_MD)
274
- gr.LoginButton()
275
  with gr.Row():
276
  sentence = gr.Textbox(label="Sentence", placeholder="the cat sat and the sun shone", scale=4)
277
  seed = gr.Textbox(label="Configuration seed", value="0", scale=1)
@@ -287,33 +253,37 @@ with gr.Blocks(title="VisualLearningIdea") as demo:
287
  with gr.Row():
288
  picture = gr.Image(label="Picture", type="filepath")
289
  model = gr.Model3D(label="3D")
 
 
 
 
290
  with gr.Accordion("Ledger: freeze and change (serial side)", open=False):
291
  ledger = gr.Code(label="Ledger", language="json", interactive=True)
292
  changes = gr.Code(label="Changes", language="json", value=EXAMPLE_CHANGES, interactive=True)
293
  refine = gr.Button("Refine")
294
  with gr.Accordion("Simplification ladder", open=False):
295
- with gr.Row():
296
- item_id = gr.Textbox(label="Object item id", value="n1")
297
- recog = gr.Checkbox(label="Check recognition with a vision model", value=False)
298
  simp = gr.Button("Simplify")
299
  ladder_pics = gr.Gallery(label="Rungs", columns=4)
 
 
300
  ladder_report = gr.Code(label="Report", language="json")
301
  ladder_files = gr.File(label="GLB per rung", file_count="multiple")
302
- with gr.Accordion("Back-translation", open=False):
303
- back = gr.Button("Describe the scene")
304
- back_out = gr.Code(label="Back-translation", language="json")
305
 
306
  gr.on([sentence.change, primitives.change], check_words, [sentence, primitives], words,
307
  api_name="check_words", trigger_mode="always_last") \
308
  .then(_suggestions, words, [status, suggest, missing], api_visibility="private")
309
  suggest.select(_complete, [sentence, suggest], sentence, api_visibility="private")
310
  use_prim.click(_use_primitive, [primitives, missing, shape], primitives, api_visibility="private")
311
- go.click(translate_to_scene, [sentence, primitives, seed], [ledger, picture, model],
312
- api_name="translate_to_scene")
313
- refine.click(refine_scene, [ledger, changes, seed], [ledger, picture, model], api_name="refine_scene")
314
- simp.click(simplify_item, [ledger, item_id, recog], [ladder_report, ladder_pics, ladder_files],
315
- api_name="simplify_item")
316
- back.click(describe_scene, [ledger, seed], back_out, api_name="describe_scene")
 
 
 
317
 
318
  if __name__ == "__main__":
319
  demo.launch(mcp_server=True)
 
2
 
3
  One Space for people (this UI) and for models (every public event below is an
4
  MCP tool: demo.launch(mcp_server=True)). Helper events are api_visibility="private".
5
+ The Space only visualises; discussion happens in the chat that calls it. Every
6
+ picture comes back with a mini HTML page, because HTML is the final destination.
7
  """
8
+ import html as html_lib
9
  import json
10
  import os
11
  import re
 
15
  import spaces # ZeroGPU: import before any CUDA-touching library
16
  import gradio as gr
17
 
18
+ from vli import assets, html, pipeline, scene, simplify
19
  from vli.grammar import check_words as _check_words
20
  from vli.lexicon import LEXICON, PRIMITIVE_SHAPES
21
 
 
38
  raise ValueError(f"{name} is not valid JSON")
39
 
40
 
 
 
 
 
 
 
 
 
41
  def _seed(value):
42
  """Seeds arrive as strings (some MCP clients can't send numbers)."""
43
  try:
 
61
  def check_words(sentence: str, primitives_json: str = "{}") -> str:
62
  """Check which words of a sentence exist in the 3D vocabulary, and complete the word being typed.
63
 
64
+ A word is available only when a ready asset (SimReady or Objaverse) exists for it;
65
+ "maps_to" names the asset it would get. Words with no asset are listed in "missing":
66
+ rephrase them, or map each one to a primitive shape in primitives_json. Verbs (sat,
67
+ stood, lay, shone, faced) and prepositions (on, under, beside, near, behind, in front
68
+ of, above) are relations between meshes; whatever "shone" becomes the light source.
69
 
70
  Args:
71
  sentence: The English sentence, complete or partly typed.
 
73
  Shapes: box, sphere, cylinder, cone, plane.
74
 
75
  Returns:
76
+ JSON string: {"ok", "words": [{text, kind, available_in, maps_to | primitive}], "missing",
77
  "primitive_shapes", "completions"}.
78
  """
79
  try:
 
82
  _fail(e)
83
 
84
 
85
+ def translate_to_scene(sentence: str, primitives_json: str = "{}", seed: str = "0") -> tuple[str, str, str, str]:
86
  """Translate an English sentence into a 3D scene.
87
 
88
  Produces the serial ledger (tokens in reading order, the items each word commits,
89
+ and the unresolved questions), one configuration of that ledger chosen by `seed`,
90
+ a rendered picture, a GLB model and a mini HTML page. The HTML is a self-contained
91
+ Three.js scene with inline low-poly geometry: read it, edit it, build on it. Words
92
+ with no asset must be mapped to a primitive in primitives_json (see check_words).
93
 
94
  Args:
95
  sentence: A sentence using available words, e.g. "the cat sat and the sun shone".
 
97
  seed: Picks one of the configurations that satisfy the ledger (positions, facing, sun).
98
 
99
  Returns:
100
+ (ledger JSON string, rendered PNG, GLB scene file, mini HTML page).
101
  """
102
  try:
103
  return pipeline.translate(sentence, _json(primitives_json, "primitives_json", {}), _seed(seed))
 
105
  _fail(e)
106
 
107
 
108
+ def refine_scene(ledger_json: str, changes_json: str = "[]", seed: str = "0") -> tuple[str, str, str, str]:
109
  """Refine a ledger: freeze some decisions and change the others, then re-render.
110
 
111
  Changes are a JSON list of:
112
  {"id": "n1", "freeze": true} lock an item (false unlocks)
113
+ {"id": "n1", "use": 2} swap in alternatives[2]; the old choice takes slot 2,
114
+ so using 2 again swaps back (other slots don't move)
115
  {"id": "n1", "reject": true} reject the tier: simready -> objaverse -> primitive -> generation
116
  {"id": "n1", "height_m": 0.4} set the object's height in metres
117
  {"id": "n1", "shape": "sphere"} use a primitive (box, sphere, cylinder, cone, plane)
118
  {"reroll": true} every unfrozen object takes a random alternative
119
+ Frozen items cannot be changed. The configuration (placement, sun angle) is always
120
+ re-chosen from `seed`: change the seed for another valid configuration.
121
 
122
  Args:
123
  ledger_json: Ledger JSON from translate_to_scene or a previous refine_scene.
 
125
  seed: Picks the configuration, and the alternatives for a reroll.
126
 
127
  Returns:
128
+ (ledger JSON string, rendered PNG, GLB scene file, mini HTML page).
129
  """
130
  try:
131
  ledger = _json(ledger_json, "ledger_json")
 
135
  _fail(e)
136
 
137
 
138
+ def simplify_item(ledger_json: str, item_id: str) -> tuple[str, list, list, str]:
 
139
  """Build the simplification ladder for one object: full asset -> fewer faces -> convex hull -> box.
140
 
141
+ Look at where the object stops reading as its word: that rung is the least geometry
142
+ that still carries the meaning, and the low rungs are the ones that can be made
143
+ physically (fewer faces = fewer papercraft pieces).
144
 
145
  Args:
146
  ledger_json: Ledger JSON from translate_to_scene or refine_scene.
147
  item_id: The object item to simplify, e.g. "n1".
 
 
148
 
149
  Returns:
150
+ (report JSON string, gallery of rung pictures, GLB file per rung,
151
+ mini HTML page with the low rungs side by side).
152
  """
153
  try:
 
 
 
154
  ledger = _json(ledger_json, "ledger_json")
155
  item = next((i for i in ledger["items"] if i["id"] == item_id), None)
156
  if item is None or item["kind"] != "object":
157
  raise ValueError(f"{item_id!r} is not an object item in this ledger")
158
  mesh = assets.settle(item)
159
  work = tempfile.mkdtemp(prefix="vli_ladder_")
160
+ rungs = simplify.ladder(mesh)
161
  report, gallery, files = {"item": item_id, "word": item["word"], "rungs": []}, [], []
162
+ for k, (label, rung) in enumerate(rungs):
163
  png = scene.render({item_id: rung}, os.path.join(work, f"{k}.png"), title=label)
164
  slug = re.sub(r"\W+", "_", label)
165
  glb = os.path.join(work, f"{k}_{slug}.glb")
166
  rung.export(glb)
167
+ report["rungs"].append({"rung": label, **simplify.stats(rung)})
 
 
 
 
 
168
  gallery.append((png, label))
169
  files.append(glb)
170
+ return json.dumps(report, indent=1), gallery, files, html.ladder_page(item, rungs)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
171
  except Exception as e:
172
  _fail(e)
173
 
 
181
  if w["kind"] == "unknown":
182
  marks.append(f"~~{w['text']}~~")
183
  elif w["kind"] == "noun":
184
+ marks.append(f"**{w['text']}** → {w.get('maps_to', '?')}")
185
  elif w["kind"] == "primitive":
186
  marks.append(f"_{w['text']}_ ({w['primitive']})")
187
  else:
188
  marks.append(w["text"])
189
+ status = " · ".join(marks) or "_start typing_"
190
  if data.get("missing"):
191
  status += " \n_Struck-through words have no asset: pick a suggestion, rephrase, or choose a primitive._"
192
  missing = data.get("missing", [])
 
207
  return json.dumps(prims)
208
 
209
 
210
+ def _html_views(page):
211
+ """Live preview (sandboxed iframe) and a downloadable .html file."""
212
+ if not page:
213
+ return "", None
214
+ path = os.path.join(tempfile.mkdtemp(prefix="vli_html_"), "scene.html")
215
+ with open(path, "w", encoding="utf-8") as f:
216
+ f.write(page)
217
+ frame = (f'<iframe sandbox="allow-scripts" srcdoc="{html_lib.escape(page, quote=True)}" '
218
+ 'style="width:100%;height:460px;border:0;border-radius:8px"></iframe>')
219
+ return frame, path
220
+
221
+
222
  EXAMPLE_CHANGES = '[\n {"id": "n1", "freeze": true},\n {"reroll": true}\n]'
223
 
224
  DATASETS_MD = """
225
+ **Dictionary (public datasets, read-only, no account needed):**
226
  1. [NVIDIA SimReady](https://github.com/NVIDIA-Omniverse/simready-central): metric OpenUSD assets,
227
  e.g. [nvidia/simready-assets](https://huggingface.co/datasets/nvidia/simready-assets). License per package
228
  (e.g. CC-BY-4.0). Only geometry and the thumbnail are downloaded, not the ~1 GB of textures.
 
230
  License per object; non-commercial (NC) and no-derivatives (ND) objects are skipped.
231
  3. Missing words: a primitive you choose (box, sphere, cylinder, cone, plane). Generation comes later.
232
 
233
+ This Space only visualises and stores no token. Discuss the results in your chat
234
+ (e.g. HuggingChat via MCP): every picture comes with a mini HTML page to build on.
 
 
235
  """
236
 
237
  with gr.Blocks(title="VisualLearningIdea") as demo:
238
  gr.Markdown("# English → 3D\nType with the words the 3D dictionary has. **Bold** = an object asset; "
239
  "verbs and prepositions are relations between meshes.")
240
  gr.Markdown(DATASETS_MD)
 
241
  with gr.Row():
242
  sentence = gr.Textbox(label="Sentence", placeholder="the cat sat and the sun shone", scale=4)
243
  seed = gr.Textbox(label="Configuration seed", value="0", scale=1)
 
253
  with gr.Row():
254
  picture = gr.Image(label="Picture", type="filepath")
255
  model = gr.Model3D(label="3D")
256
+ with gr.Accordion("Mini HTML", open=False):
257
+ scene_preview = gr.HTML()
258
+ scene_file = gr.File(label="Download .html")
259
+ scene_html = gr.Code(label="HTML", language="html")
260
  with gr.Accordion("Ledger: freeze and change (serial side)", open=False):
261
  ledger = gr.Code(label="Ledger", language="json", interactive=True)
262
  changes = gr.Code(label="Changes", language="json", value=EXAMPLE_CHANGES, interactive=True)
263
  refine = gr.Button("Refine")
264
  with gr.Accordion("Simplification ladder", open=False):
265
+ item_id = gr.Textbox(label="Object item id", value="n1")
 
 
266
  simp = gr.Button("Simplify")
267
  ladder_pics = gr.Gallery(label="Rungs", columns=4)
268
+ ladder_preview = gr.HTML()
269
+ ladder_html = gr.Code(label="HTML", language="html")
270
  ladder_report = gr.Code(label="Report", language="json")
271
  ladder_files = gr.File(label="GLB per rung", file_count="multiple")
 
 
 
272
 
273
  gr.on([sentence.change, primitives.change], check_words, [sentence, primitives], words,
274
  api_name="check_words", trigger_mode="always_last") \
275
  .then(_suggestions, words, [status, suggest, missing], api_visibility="private")
276
  suggest.select(_complete, [sentence, suggest], sentence, api_visibility="private")
277
  use_prim.click(_use_primitive, [primitives, missing, shape], primitives, api_visibility="private")
278
+ go.click(translate_to_scene, [sentence, primitives, seed], [ledger, picture, model, scene_html],
279
+ api_name="translate_to_scene") \
280
+ .then(_html_views, scene_html, [scene_preview, scene_file], api_visibility="private")
281
+ refine.click(refine_scene, [ledger, changes, seed], [ledger, picture, model, scene_html],
282
+ api_name="refine_scene") \
283
+ .then(_html_views, scene_html, [scene_preview, scene_file], api_visibility="private")
284
+ simp.click(simplify_item, [ledger, item_id], [ladder_report, ladder_pics, ladder_files, ladder_html],
285
+ api_name="simplify_item") \
286
+ .then(lambda page: _html_views(page)[0], ladder_html, ladder_preview, api_visibility="private")
287
 
288
  if __name__ == "__main__":
289
  demo.launch(mcp_server=True)
vli/grammar.py CHANGED
@@ -237,6 +237,7 @@ def check_words(sentence, primitives=None):
237
  entry = {"text": t["text"], "kind": t["kind"]}
238
  if t["kind"] == "noun":
239
  entry["available_in"] = LEXICON.counts(t["text"])
 
240
  elif t["kind"] == "primitive":
241
  entry["primitive"] = primitives[t["text"]]
242
  words.append(entry)
 
237
  entry = {"text": t["text"], "kind": t["kind"]}
238
  if t["kind"] == "noun":
239
  entry["available_in"] = LEXICON.counts(t["text"])
240
+ entry["maps_to"] = candidate_label(_object_decision(t["text"])["chosen"]) # what you'd get
241
  elif t["kind"] == "primitive":
242
  entry["primitive"] = primitives[t["text"]]
243
  words.append(entry)
vli/html.py ADDED
@@ -0,0 +1,199 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Mini HTML returned next to every picture.
2
+
3
+ HTML is the final destination: a chat model reads this page, edits it and
4
+ builds its own on it. The page is self-contained Three.js. Geometry is inlined
5
+ at low poly (a rung of the simplification ladder), so it outlives Gradio's
6
+ temporary file URLs and stays small enough for a model's context.
7
+ """
8
+ import json
9
+
10
+ import numpy as np
11
+
12
+ from .scene import GLOW, _sky
13
+ from .simplify import base_color, decimate
14
+
15
+ THREE = "https://unpkg.com/three@0.160.0/"
16
+ SCENE_FACE_BUDGET = 400 # per object in a scene page
17
+ LADDER_MAX_FACES = 1000 # rungs above this are left to the GLB files
18
+
19
+
20
+ def _hex(rgb):
21
+ return "#%02x%02x%02x" % tuple(int(c) for c in rgb[:3])
22
+
23
+
24
+ def _sky_hex(sun):
25
+ sky = _sky(sun)
26
+ return sky if isinstance(sky, str) else _hex([c * 255 for c in sky])
27
+
28
+
29
+ def _geometry(mesh, budget):
30
+ m = decimate(mesh, budget) if len(mesh.faces) > budget else mesh
31
+ return {"faces": int(len(m.faces)),
32
+ "p": np.round(m.vertices, 3).ravel().tolist(),
33
+ "i": m.faces.ravel().astype(int).tolist()}
34
+
35
+
36
+ def _object(oid, word, mesh, position, yaw_deg, cand=None, glow=False, budget=SCENE_FACE_BUDGET, label=None):
37
+ o = {"id": oid, "word": word, "label": label, "position": [round(float(c), 3) for c in position],
38
+ "yaw_deg": round(float(yaw_deg), 1), "color": _hex(GLOW if glow else base_color(mesh)),
39
+ "emits_light": glow}
40
+ if cand and cand.get("source") == "primitive":
41
+ o["primitive"] = {"shape": cand["shape"], "size": [round(float(e), 3) for e in mesh.extents]}
42
+ else:
43
+ o["mesh"] = _geometry(mesh, budget)
44
+ return o
45
+
46
+
47
+ def scene_page(ledger, meshes, config):
48
+ sun = config["sun"]
49
+ objects = []
50
+ for it in ledger["items"]:
51
+ p = config["placements"].get(it["id"])
52
+ if it["kind"] != "object" or p is None:
53
+ continue
54
+ dec = it["decision"]
55
+ objects.append(_object(it["id"], it["word"], meshes[it["id"]], p["position"], p["yaw_deg"],
56
+ cand=dec["chosen"], glow=bool(p.get("light")), label=dec.get("label")))
57
+ ground = None
58
+ if any(i["kind"] == "environment" and i["role"] == "plane" for i in ledger["items"]):
59
+ reach = max([abs(c) for o in objects if not o["emits_light"] for c in (o["position"][0], o["position"][2])]
60
+ + [0.3])
61
+ ground = {"radius": round(1.6 * reach + 0.5, 3)}
62
+ data = {
63
+ "title": ledger["sentence"],
64
+ "reading": [{"word": t["text"], "kind": t["kind"], "commits": t["commits"]} for t in ledger["tokens"]],
65
+ "relations": [{"subject": i["subject"], "relation": i["relation"], "object": i["object"]}
66
+ for i in ledger["items"] if i["kind"] == "relation"],
67
+ "unresolved": [{"question": q.get("question"), "default": q.get("default_chosen")}
68
+ for q in ledger["unresolved"]],
69
+ "sky": _sky_hex(sun), "sun": sun, "ground": ground, "objects": objects,
70
+ }
71
+ return page(data)
72
+
73
+
74
+ def ladder_page(item, rungs):
75
+ """Rungs left to right, from the most detailed embeddable one down to the box."""
76
+ objects, x = [], 0.0
77
+ for label, rung in rungs:
78
+ if len(rung.faces) > LADDER_MAX_FACES:
79
+ continue
80
+ hw = 0.5 * max(rung.extents[0], rung.extents[2])
81
+ objects.append(_object(f"rung{len(objects)}", item["word"], rung, [x + hw, 0, 0], 0,
82
+ budget=LADDER_MAX_FACES, label=label))
83
+ x += 2 * hw + 0.3 * max(rung.extents[1], 0.1)
84
+ for o in objects:
85
+ o["position"][0] = round(o["position"][0] - x / 2, 3)
86
+ data = {"title": f"{item['word']}: simplification ladder",
87
+ "caption": "left → right: " + " · ".join(o["label"] for o in objects),
88
+ "sky": "#2a3346", "sun": None, "ground": None, "objects": objects}
89
+ return page(data)
90
+
91
+
92
+ def page(data):
93
+ payload = json.dumps(data, separators=(",", ":")).replace("</", "<\\/")
94
+ return TEMPLATE.replace("__THREE__", THREE).replace("__TITLE__", _escape(data["title"])) \
95
+ .replace("__DATA__", payload)
96
+
97
+
98
+ def _escape(s):
99
+ return s.replace("&", "&amp;").replace("<", "&lt;").replace(">", "&gt;")
100
+
101
+
102
+ TEMPLATE = """<!doctype html>
103
+ <html><head><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
104
+ <title>__TITLE__</title>
105
+ <style>html,body{margin:0;height:100%;overflow:hidden;font:13px system-ui,sans-serif}
106
+ #cap{position:fixed;left:12px;bottom:12px;color:#fff;background:#0008;padding:6px 10px;border-radius:6px}</style>
107
+ <script type="importmap">{"imports":{"three":"__THREE__build/three.module.js","three/addons/":"__THREE__examples/jsm/"}}</script>
108
+ </head><body><div id="cap"></div>
109
+ <script type="module">
110
+ import * as THREE from "three";
111
+ import {OrbitControls} from "three/addons/controls/OrbitControls.js";
112
+
113
+ // Scene data from VisualLearningIdea: positions in metres, Y up, yaw about Y.
114
+ // Objects carry inline low-poly geometry ("mesh") or a primitive spec.
115
+ const D = __DATA__;
116
+
117
+ const renderer = new THREE.WebGLRenderer({antialias: true});
118
+ renderer.setPixelRatio(Math.min(devicePixelRatio, 2));
119
+ renderer.setSize(innerWidth, innerHeight);
120
+ renderer.shadowMap.enabled = true;
121
+ document.body.appendChild(renderer.domElement);
122
+ const scene = new THREE.Scene();
123
+ scene.background = new THREE.Color(D.sky);
124
+ scene.add(new THREE.HemisphereLight(0xffffff, 0x445533, 0.9));
125
+ const camera = new THREE.PerspectiveCamera(45, innerWidth / innerHeight, 0.01, 1000);
126
+
127
+ function geometryOf(o) {
128
+ if (o.primitive) {
129
+ const [w, h, d] = o.primitive.size;
130
+ const g = {sphere: () => new THREE.SphereGeometry(h / 2, 24, 16),
131
+ cylinder: () => new THREE.CylinderGeometry(w / 2, w / 2, h, 24),
132
+ cone: () => new THREE.ConeGeometry(w / 2, h, 24),
133
+ plane: () => new THREE.BoxGeometry(w, h, d),
134
+ box: () => new THREE.BoxGeometry(w, h, d)}[o.primitive.shape]();
135
+ g.translate(0, h / 2, 0); // rest on y = 0 like every asset
136
+ return g;
137
+ }
138
+ const g = new THREE.BufferGeometry();
139
+ g.setAttribute("position", new THREE.Float32BufferAttribute(o.mesh.p, 3));
140
+ g.setIndex(o.mesh.i);
141
+ g.computeVertexNormals();
142
+ return g;
143
+ }
144
+
145
+ const frame = new THREE.Box3();
146
+ for (const o of D.objects) {
147
+ const mat = o.emits_light
148
+ ? new THREE.MeshBasicMaterial({color: o.color})
149
+ : new THREE.MeshStandardMaterial({color: o.color, roughness: 0.85, flatShading: true, side: THREE.DoubleSide});
150
+ const m = new THREE.Mesh(geometryOf(o), mat);
151
+ m.name = o.id;
152
+ m.position.fromArray(o.position);
153
+ m.rotation.y = THREE.MathUtils.degToRad(o.yaw_deg);
154
+ m.castShadow = !o.emits_light;
155
+ m.receiveShadow = true;
156
+ scene.add(m);
157
+ if (!o.emits_light) frame.expandByObject(m);
158
+ }
159
+ if (frame.isEmpty()) frame.setFromCenterAndSize(new THREE.Vector3(), new THREE.Vector3(1, 1, 1));
160
+ const center = frame.getCenter(new THREE.Vector3());
161
+ const radius = Math.max(frame.getSize(new THREE.Vector3()).length(), 0.3);
162
+
163
+ if (D.ground) {
164
+ const g = new THREE.Mesh(new THREE.CircleGeometry(D.ground.radius, 64),
165
+ new THREE.MeshStandardMaterial({color: 0x4d7c3a, roughness: 1}));
166
+ g.rotation.x = -Math.PI / 2;
167
+ g.receiveShadow = true;
168
+ scene.add(g);
169
+ }
170
+
171
+ const sun = new THREE.DirectionalLight(0xfff1d0, 2.2);
172
+ if (D.sun) {
173
+ const el = THREE.MathUtils.degToRad(D.sun.elevation_deg), az = THREE.MathUtils.degToRad(D.sun.azimuth_deg);
174
+ const dir = new THREE.Vector3(Math.cos(el) * Math.sin(az), Math.sin(el), Math.cos(el) * Math.cos(az));
175
+ sun.position.copy(center).addScaledVector(dir, radius * 3);
176
+ sun.castShadow = !!D.sun.shadows;
177
+ } else {
178
+ sun.position.copy(center).add(new THREE.Vector3(radius, radius * 2, radius * 1.2));
179
+ }
180
+ sun.target.position.copy(center);
181
+ Object.assign(sun.shadow.camera, {left: -radius * 1.5, right: radius * 1.5, top: radius * 1.5,
182
+ bottom: -radius * 1.5, far: radius * 8});
183
+ sun.shadow.mapSize.set(1024, 1024);
184
+ scene.add(sun, sun.target);
185
+
186
+ camera.position.set(center.x + radius * 0.9, center.y + radius * 0.6, center.z + radius * 1.2);
187
+ const controls = new OrbitControls(camera, renderer.domElement);
188
+ controls.target.copy(center);
189
+ controls.update();
190
+
191
+ document.getElementById("cap").textContent = D.caption || D.title;
192
+ addEventListener("resize", () => {
193
+ camera.aspect = innerWidth / innerHeight;
194
+ camera.updateProjectionMatrix();
195
+ renderer.setSize(innerWidth, innerHeight);
196
+ });
197
+ renderer.setAnimationLoop(() => { controls.update(); renderer.render(scene, camera); });
198
+ </script></body></html>
199
+ """
vli/lexicon.py CHANGED
@@ -37,22 +37,24 @@ def _split_part(part):
37
 
38
 
39
  def simready_phrases(name):
40
- """Words an asset name like 'appliance_handDryer_automatic_a01' can be looked up by.
41
 
42
- SimReady names read category_object_modifiers, so the second part is the
43
- best match (rank 0); every other part is a weaker match (rank 1).
 
44
  """
45
  parts = [p for p in name.split("_") if p and not VARIANT.fullmatch(p)]
 
46
  out = []
47
- for i, part in enumerate(parts):
48
- words = _split_part(part)
49
- if not words:
50
- continue
51
- rank = 0 if (i == 1 or len(parts) == 1) else 1
52
- out.append((" ".join(words), rank))
53
- if len(words) > 1:
54
- out.append((words[-1], rank + 1))
55
- return out
56
 
57
 
58
  def objaverse_word(category):
 
37
 
38
 
39
  def simready_phrases(name):
40
+ """Words an asset name like 'interiorPart_sun_visor_a01' can be looked up by.
41
 
42
+ Names read category_object..., where the object can span several parts
43
+ ('sun_visor'). Only whole phrases and the final part are indexed, never a
44
+ lone modifier: 'sun' must not find a sun visor.
45
  """
46
  parts = [p for p in name.split("_") if p and not VARIANT.fullmatch(p)]
47
+ body = parts[1:] or parts # drop the category prefix
48
  out = []
49
+ phrase = " ".join(w for p in body for w in _split_part(p))
50
+ if phrase:
51
+ out.append((phrase, 0)) # 'sun visor', 'hand dryer automatic'
52
+ final = _split_part(body[-1]) if body else []
53
+ if final:
54
+ out.append((" ".join(final), 0)) # 'visor', 'bucky table'
55
+ if len(final) > 1:
56
+ out.append((final[-1], 1)) # 'table' from 'buckyTable': weaker
57
+ return list(dict.fromkeys(out))
58
 
59
 
60
  def objaverse_word(category):
vli/pipeline.py CHANGED
@@ -9,7 +9,7 @@ import tempfile
9
 
10
  import numpy as np
11
 
12
- from . import assets, scene
13
  from .grammar import build_ledger, candidate_label, next_tier, parse_primitives, primitive_candidate
14
 
15
 
@@ -40,7 +40,7 @@ def outputs(ledger, seed):
40
  work = _workdir()
41
  png = scene.render_scene(ledger, meshes, config, os.path.join(work, "scene.png"))
42
  glb = scene.export_glb(ledger, meshes, config, os.path.join(work, "scene.glb"))
43
- return json.dumps(ledger, indent=1), png, glb
44
 
45
 
46
  def translate(sentence, primitives=None, seed=0):
 
9
 
10
  import numpy as np
11
 
12
+ from . import assets, html, scene
13
  from .grammar import build_ledger, candidate_label, next_tier, parse_primitives, primitive_candidate
14
 
15
 
 
40
  work = _workdir()
41
  png = scene.render_scene(ledger, meshes, config, os.path.join(work, "scene.png"))
42
  glb = scene.export_glb(ledger, meshes, config, os.path.join(work, "scene.glb"))
43
+ return json.dumps(ledger, indent=1), png, glb, html.scene_page(ledger, meshes, config)
44
 
45
 
46
  def translate(sentence, primitives=None, seed=0):
vli/scene.py CHANGED
@@ -15,6 +15,7 @@ from scipy.spatial import ConvexHull # noqa: E402
15
  from .simplify import base_color, decimate # noqa: E402
16
 
17
  GAP = 0.1
 
18
  RENDER_FACE_BUDGET = 3000
19
 
20
 
@@ -46,7 +47,7 @@ def configure(ledger, meshes, seed=0):
46
 
47
  sun = None
48
  if lights:
49
- sun = {"id": lights[0]["id"], "elevation_deg": float(rng.uniform(20, 70)),
50
  "azimuth_deg": float(rng.uniform(-150, 150)),
51
  "shadows": any(r["relation"] == "lights" for r in relations)}
52
 
@@ -89,9 +90,10 @@ def configure(ledger, meshes, seed=0):
89
 
90
  if sun: # whatever shone sits in the sky along the light direction
91
  reach = max([abs(c) + _half_width(meshes[o]) for o, p in place.items()
92
- for c in (p["position"][0], p["position"][2])] + [1.0])
93
- pos = sun_direction(sun) * 1.3 * reach # shadows use the direction; distance is only for the view
94
  for it in lights:
 
 
95
  place[it["id"]] = {"position": [float(c) for c in pos], "yaw_deg": 0.0, "light": True}
96
  return {"seed": seed, "placements": place, "sun": sun}
97
 
@@ -106,7 +108,7 @@ def _ground(ledger, placed):
106
  if not any(i["kind"] == "environment" and i["role"] == "plane" for i in ledger["items"]):
107
  return None
108
  lights = {i["id"] for i in ledger["items"] if i.get("emits_light")}
109
- reach = max([np.abs(m.bounds[:, [0, 2]]).max() for oid, m in placed.items() if oid not in lights] + [1.0])
110
  g = trimesh.creation.cylinder(radius=reach * 1.6, height=0.01, sections=48)
111
  g.apply_transform(trimesh.transformations.rotation_matrix(math.pi / 2, [1, 0, 0]))
112
  g.apply_translation([0, -0.005, 0])
@@ -119,6 +121,8 @@ def placed_meshes(ledger, meshes, config):
119
  for oid, p in config["placements"].items():
120
  m = meshes[oid].copy()
121
  m.apply_transform(transform(p))
 
 
122
  out[oid] = m
123
  return out
124
 
@@ -150,9 +154,11 @@ def _to_axes(v):
150
  return np.stack([v[..., 0], -v[..., 2], v[..., 1]], axis=-1)
151
 
152
 
153
- def _shade(mesh, light, color):
154
- n = mesh.face_normals
155
- k = 0.35 + 0.65 * np.clip(n @ light, 0, 1)
 
 
156
  rgb = np.clip(color[:3] / 255.0 * k[:, None], 0, 1)
157
  return np.c_[rgb, np.ones(len(rgb))]
158
 
@@ -187,11 +193,11 @@ def render(meshes_by_id, path, sun=None, ground=None, title=None):
187
  ax.add_collection3d(Poly3DCollection([_to_axes(poly)], facecolors=[(0, 0, 0, 0.35)], zorder=1))
188
 
189
  tris, colors = [], []
190
- for m in meshes_by_id.values():
191
  color = base_color(m)
192
  r = decimate(m, RENDER_FACE_BUDGET) if len(m.faces) > RENDER_FACE_BUDGET else m
193
  tris.append(r.triangles)
194
- colors.append(_shade(r, light, color))
195
  pts.append(r.vertices)
196
  if tris:
197
  ax.add_collection3d(Poly3DCollection(_to_axes(np.concatenate(tris)), facecolors=np.concatenate(colors),
 
15
  from .simplify import base_color, decimate # noqa: E402
16
 
17
  GAP = 0.1
18
+ GLOW = [255, 214, 107, 255]
19
  RENDER_FACE_BUDGET = 3000
20
 
21
 
 
47
 
48
  sun = None
49
  if lights:
50
+ sun = {"id": lights[0]["id"], "elevation_deg": float(rng.uniform(25, 65)),
51
  "azimuth_deg": float(rng.uniform(-150, 150)),
52
  "shadows": any(r["relation"] == "lights" for r in relations)}
53
 
 
90
 
91
  if sun: # whatever shone sits in the sky along the light direction
92
  reach = max([abs(c) + _half_width(meshes[o]) for o, p in place.items()
93
+ for c in (p["position"][0], p["position"][2])] + [0.3])
 
94
  for it in lights:
95
+ # shadows use the direction; distance only frames the view
96
+ pos = sun_direction(sun) * (1.5 * reach + 2 * _half_width(meshes[it["id"]]))
97
  place[it["id"]] = {"position": [float(c) for c in pos], "yaw_deg": 0.0, "light": True}
98
  return {"seed": seed, "placements": place, "sun": sun}
99
 
 
108
  if not any(i["kind"] == "environment" and i["role"] == "plane" for i in ledger["items"]):
109
  return None
110
  lights = {i["id"] for i in ledger["items"] if i.get("emits_light")}
111
+ reach = max([np.abs(m.bounds[:, [0, 2]]).max() for oid, m in placed.items() if oid not in lights] + [0.3])
112
  g = trimesh.creation.cylinder(radius=reach * 1.6, height=0.01, sections=48)
113
  g.apply_transform(trimesh.transformations.rotation_matrix(math.pi / 2, [1, 0, 0]))
114
  g.apply_translation([0, -0.005, 0])
 
121
  for oid, p in config["placements"].items():
122
  m = meshes[oid].copy()
123
  m.apply_transform(transform(p))
124
+ if p.get("light"): # a light source glows: warm, flat colour
125
+ m.visual = trimesh.visual.ColorVisuals(m, face_colors=np.tile(GLOW, (len(m.faces), 1)))
126
  out[oid] = m
127
  return out
128
 
 
154
  return np.stack([v[..., 0], -v[..., 2], v[..., 1]], axis=-1)
155
 
156
 
157
+ def _shade(mesh, light, color, glow=False):
158
+ if glow:
159
+ k = np.ones(len(mesh.faces))
160
+ else: # two-sided: Objaverse meshes often have flipped faces
161
+ k = 0.45 + 0.55 * np.abs(mesh.face_normals @ light)
162
  rgb = np.clip(color[:3] / 255.0 * k[:, None], 0, 1)
163
  return np.c_[rgb, np.ones(len(rgb))]
164
 
 
193
  ax.add_collection3d(Poly3DCollection([_to_axes(poly)], facecolors=[(0, 0, 0, 0.35)], zorder=1))
194
 
195
  tris, colors = [], []
196
+ for oid, m in meshes_by_id.items():
197
  color = base_color(m)
198
  r = decimate(m, RENDER_FACE_BUDGET) if len(m.faces) > RENDER_FACE_BUDGET else m
199
  tris.append(r.triangles)
200
+ colors.append(_shade(r, light, color, glow=bool(sun) and oid == sun["id"]))
201
  pts.append(r.vertices)
202
  if tris:
203
  ax.add_collection3d(Poly3DCollection(_to_axes(np.concatenate(tris)), facecolors=np.concatenate(colors),
vli/vlm.py DELETED
@@ -1,35 +0,0 @@
1
- """Back-translation: a vision model reads the picture back into English.
2
-
3
- Calls always use the *caller's* token, never one stored on the Space, so a
4
- public Space can't spend its owner's inference credits.
5
- """
6
- import base64
7
- import os
8
-
9
- from huggingface_hub import InferenceClient
10
-
11
- VLM_MODEL = os.environ.get("VLM_MODEL", "Qwen/Qwen2.5-VL-7B-Instruct")
12
- SIGN_IN = ("The vision model runs on your own Hugging Face account: sign in first "
13
- "(MCP callers: send your HF token as a Bearer Authorization header).")
14
-
15
-
16
- def ask(image_path, prompt, token, max_tokens=80):
17
- if not token:
18
- raise ValueError(SIGN_IN)
19
- with open(image_path, "rb") as f:
20
- data = base64.b64encode(f.read()).decode()
21
- client = InferenceClient(token=token)
22
- resp = client.chat_completion(
23
- model=VLM_MODEL, max_tokens=max_tokens,
24
- messages=[{"role": "user", "content": [
25
- {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{data}"}},
26
- {"type": "text", "text": prompt}]}])
27
- return resp.choices[0].message.content.strip()
28
-
29
-
30
- def describe(image_path, token):
31
- return ask(image_path, "Describe this 3D scene in one short plain sentence.", token)
32
-
33
-
34
- def name_object(image_path, token):
35
- return ask(image_path, "What single object is this? Answer with one or two words.", token, max_tokens=10)