README.md

---
title: Snap2Sim Inside The Machine
sdk: gradio
app_file: app.py
license: mit
colorFrom: blue
colorTo: yellow
short_description: Photograph a part. Get a mechanical cutaway.
models:
- unsloth/NVIDIA-Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF
tags:
- build-small-hackathon
- backyard-ai
- llama-champion
- nvidia-nemotron-quest
- modal-award
- openai-codex
- off-brand
- best-demo
- sharing-is-caring
- llama-cpp
- gguf
- gradio
- three-js
- track:backyard
- sponsor:openai
- sponsor:nvidia
- sponsor:modal
- achievement:offbrand
- achievement:llama
- achievement:sharing
---

# Snap2Sim: Inside the Machine

You find a small metal cylinder at a flea market. What is it? How does it work
inside?

Snap2Sim turns a photo of a hardware component into a narrated mechanical
cutaway. Drop in a picture of a valve, lock, pump, gear train, fan, latch, or
other mystery part; the app identifies the likely mechanism, names the internal
pieces, and renders a moving 3D field-manual view that explains how the object
works.

The goal is not just recognition. It is the moment a curious maker gets to see
the hidden motion: the plunger sliding, the spring loading, the rotor spinning,
the pawl catching, the port opening.

## Demo

- Hugging Face Space: https://huggingface.co/spaces/build-small-hackathon/Snap2Sim
- App host: https://build-small-hackathon-snap2sim.hf.space
- Source of truth: https://github.com/Bigstonks1/Snap2Sim
- Demo video: https://youtu.be/nuisDKMyyF8
- X post: https://x.com/Ryno67114241/status/2066660199558152411

The Space is public for submission under the Build Small Hackathon organization.
The app is built as a Gradio Space with a custom `gradio.Server` shell; GitHub
remains the source of truth for code changes.

## Hackathon Fit

Snap2Sim is a Backyard AI project for people who learn by taking things apart:
garage tinkerers, repair hobbyists, thrift-store scavengers, robotics students,
and anyone holding an unknown part and wondering what is going on inside.

The README metadata tags request consideration for:

| Track or badge | Why it applies |
|---|---|
| Backyard AI | Explains real-world objects from a user's own photo. |
| Llama Champion | Runs the primary model through llama.cpp with GGUF assets. |
| NVIDIA Nemotron Quest | Uses NVIDIA Nemotron 3 Nano Omni as the vision-reasoning model. |
| Modal Award | GPU inference is deployed through Modal with cached model assets. |
| OpenAI Codex | Built with Codex assistance and Codex-attributed commits in the linked GitHub repo. |
| Off-Brand | The experience is a branded maker tool, not a generic chatbot or image demo. |
| Best Demo / Sharing is Caring | Public demo video and X post are linked in the Demo section. |
| Off the Grid | Not claimed while inference runs on Modal. |

## How It Works

1. The browser accepts a component photo and sends it to the Space's same-origin
   `/analyze_image` API.
2. The Space calls a secured Modal endpoint running
   `unsloth/NVIDIA-Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF` through
   llama.cpp and `llama-mtmd-cli`.
3. Nemotron returns a structured mechanism JSON payload: component name,
   confidence, internal parts, geometry hints, motion types, annotations, and a
   plain-language explanation.
4. `/generate_scene` validates the analysis and chooses the render mode.
5. The browser renders the result with trusted deterministic Three.js code. If a
   confident 3D reconstruction is not justified, it falls back to annotated
   callouts on the original photo instead of pretending.

The model describes the mechanism. The browser owns the rendering. That keeps
the demo reliable and avoids injecting model-authored HTML or JavaScript into
the page.

## Tech Stack

- Frontend: plain `index.html`, CSS, and JavaScript served directly by
  `gradio.Server`.
- Rendering: deterministic Three.js primitives, labels, OrbitControls, camera
  fitting, and an annotated-photo fallback.
- Model runtime: NVIDIA Nemotron 3 Nano Omni 30B-A3B GGUF, `UD-Q4_K_M`, with
  `mmproj-F16.gguf` through llama.cpp.
- GPU service: Modal, with weights cached in a Modal Volume and bearer-token
  protection between the Space and Modal.
- Space runtime: Hugging Face `cpu-basic`; expensive inference happens on Modal.
- Contract: JSON schema validation in `snap2sim/schema.py` before anything is
  rendered.

The primary model is approximately 30B total parameters with roughly 3B active
per MoE token, keeping the single-model pipeline under the hackathon's 32B model
budget.

## What Makes It Different

Most image demos stop at a label. Snap2Sim tries to answer the maker's next
question: "what would I see if I cut this open?"

- It produces a mechanism explanation, not just a class name.
- It maps internal parts into a moving technical cutaway.
- It shows uncertainty honestly with a confidence threshold and a photo
  annotation fallback.
- It uses a custom public-facing interface instead of a default chat or form UI.
- It keeps secrets and GPU endpoints server-side while the client calls only
  same-origin APIs.

## Current Verification

Recent local and deployed checks confirmed:

- The public Space serves the trusted `index.html` shell.
- `/analyze_image` returns validated mechanism payloads through the secured
  Modal bearer-token flow.
- A public smoke test on the org-owned Space on June 15, 2026 returned a real
  Modal analysis for a synthetic target-like image as `rotary indexing
  mechanism` at `0.85` confidence with 3 parts.
- `/generate_scene` returns a deterministic Three.js scene descriptor, not raw
  HTML; a public smoke test returned `renderer: three`.
- The renderer handles six-part scenes, axial orientation from geometry sizes,
  geometry-aware motion-axis fallback, fitted camera framing, orbit controls,
  reset view, mobile layout, and annotated-photo fallbacks.
- Unauthenticated Modal requests return `401 Unauthorized`.

## Run Locally

```powershell
pip install -r requirements.txt
python app.py
```

For UI-only development, set:

```powershell
$env:INFERENCE_BACKEND="local"
python app.py
```

For the real demo path, configure the public variables and private token as
described in `.env.example` and `SECURITY.md`. Do not commit tokens, Modal
endpoint URLs, uploaded user photos, or downloaded model weights.

## Runtime Preflight

Check the selected GGUF and projector metadata before spending GPU time:

```powershell
python scripts/verify_runtime_assets.py
```

Useful Modal checks:

```powershell
python -m modal run modal_app.py::check_remote_imports
python -m modal run modal_app.py::download_runtime_assets
python -m modal run modal_app.py::smoke_test_llamacpp_image
python -m modal run modal_app.py::run_analysis_endpoint_check
```

On Windows, use UTF-8 output when running Modal commands:

```powershell
$env:PYTHONIOENCODING="utf-8"; python -m modal run modal_app.py::run_runtime_preflight
```

## Repository Map

- `app.py` - Gradio Server app, same-origin APIs, image validation, rate limits.
- `index.html` - upload flow, analysis panel, Three.js renderer, fallback UI.
- `modal_app.py` - Modal deployment, llama.cpp runtime, diagnostics.
- `prompts/` - archived original build prompt, GitHub sync prompt, and deck prompt.
- `presentation/deck.html` - reveal.js judging deck for the hackathon pitch.
- `docs/` - implementation notes, review passes, and representative fixtures.
- `snap2sim/backend.py` - backend selection and authenticated Modal client.
- `snap2sim/prompts.py` - vision prompt contract.
- `snap2sim/schema.py` - mechanism JSON schema and sample payload.
- `scripts/verify_runtime_assets.py` - Hugging Face model metadata preflight.
- `SECURITY.md` - public deployment handling rules.

## Acknowledgements

Built by Jason Do with implementation assistance from OpenAI Codex.