README.md

metadata

title: Decide Live
emoji: 🎯
colorFrom: green
colorTo: gray
sdk: static
pinned: false
license: apache-2.0
short_description: On-device decisions from camera and microphone
models:
  - google/embeddinggemma-2
  - FINAL-Bench/EmbeddingGemma2-ZTC-ONNX

Decide Live: on-device decisions from camera and microphone

A static page that makes zero-token decisions in the browser. A multimodal embedding model, EmbeddingGemma 2 (Apache-2.0) fine-tuned for decisions, runs locally with transformers.js 4.3.1 on WebGPU, or on the CPU (WASM) when WebGPU is not available (much slower).

The ONNX files (from the private model repo FINAL-Bench/EmbeddingGemma2-ZTC-ONNX, same file layout as onnx-community/embeddinggemma-2-ONNX) are served from this Space under models/egemma2-decide/, so the page needs no access to any other repository. There is no server and no API call per decision. The weights are downloaded once and cached by the browser. Camera frames, microphone audio, images and text are processed on the device and are never uploaded.

What it does

• Live: embeds camera frames (or a demo feed made from the sample images) and scores up to 4 decision cards. By default each card embeds the frame together with its own question, in the trained format (one pass per card). The "frame alone" setting embeds the frame once for all cards (faster, not the trained format). Each card shows probability bars, the current decision, and a confidence gate.
• Microphone / sound file: the same cards for 2 second sound windows (16 kHz mono). Windows quieter than -55 dBFS are not sent to the model. The audio encoder is the unchanged base one.
• Single input: one image, text, audio clip or short video, a question, an optional context, and 2 to 8 options written as key: description (or a yes / no question).
• Egg check: 8 held-out candling photos (not used in training) and your own photo, decided with the trained egg prompt.
• Sample tests: 18 image tests with gold answers, run in the browser with an accuracy report, plus a comparison of prompt formats.

How a decision is computed (trained format)

1. Query: the media placeholders first, then task: classification | query: {question}, then \nContext: {context} when a context is given. The model embeds media and text together.
2. Each option: title: none | text: {description}, embedded once and cached.
3. Embedding: mean pooling over the attention mask, then L2 normalization (768 numbers).
4. P(option) = softmax(scale * cosine). The scale was learned in training: 19.96 (temperature 1 / scale, about 0.0501), stored in models/egemma2-decide/decision.json.

Yes / no questions keep the question in the query and score the options Yes and No. The older prompt formats of the first version of this page remain selectable for comparison.

Files

• index.html, style.css, app.js: the page.
• worker.js: loads the model in a Web Worker and returns embeddings.
• models/egemma2-decide/: ONNX model, config, processor and tokenizer files, and decision.json (scale and prompt formats). Text and vision graphs in q4f16 (as in the model repo) and in a portable q4 (the model repo's onnx_portable/ files: 4-bit weights, with plain 8-bit gathers for the embedding and position tables, so the same files also run on the WASM runtime, which lacks GatherBlockQuantized), plus the 4-bit audio encoder. fp16 and fp32 files are not copied here because Space storage is limited to 1 GB.
• examples/: sample images and samples.json (name, files, state, type, question, options, gold); examples/egg-test/: held-out egg candling photos.
• CREDITS.md: sources and licenses of the sample images.

Code: Apache-2.0. Model: derived from google/embeddinggemma-2 (Apache-2.0).