README.md

---
title: Decide Live
emoji: 🎯
colorFrom: green
colorTo: gray
sdk: static
pinned: false
license: apache-2.0
short_description: On-device decisions from camera and microphone
models:
  - google/embeddinggemma-2
  - FINAL-Bench/EmbeddingGemma2-ZTC-ONNX
---

# Decide Live: on-device decisions from camera and microphone

A static page that makes zero-token decisions in the browser. A multimodal embedding model,
[EmbeddingGemma 2](https://huggingface.co/google/embeddinggemma-2) (Apache-2.0) fine-tuned for decisions,
runs locally with [transformers.js](https://github.com/huggingface/transformers.js) 4.3.1 on WebGPU,
or on the CPU (WASM) when WebGPU is not available (much slower).

The ONNX files (from the private model repo `FINAL-Bench/EmbeddingGemma2-ZTC-ONNX`, same file layout as
[onnx-community/embeddinggemma-2-ONNX](https://huggingface.co/onnx-community/embeddinggemma-2-ONNX)) are served
from this Space under `models/egemma2-decide/`, so the page needs no access to any other repository.
There is no server and no API call per decision. The weights are downloaded once and cached by the browser.
Camera frames, microphone audio, images and text are processed on the device and are never uploaded.

## What it does

- **Live**: embeds camera frames (or a demo feed made from the sample images) and scores up to 4 decision
  cards. By default each card embeds the frame together with its own question, in the trained format
  (one pass per card). The "frame alone" setting embeds the frame once for all cards (faster, not the
  trained format). Each card shows probability bars, the current decision, and a confidence gate.
- **Microphone / sound file**: the same cards for 2 second sound windows (16 kHz mono). Windows
  quieter than -55 dBFS are not sent to the model. The audio encoder is the unchanged base one.
- **Single input**: one image, text, audio clip or short video, a question, an optional context,
  and 2 to 8 options written as `key: description` (or a yes / no question).
- **Egg check**: 8 held-out candling photos (not used in training) and your own photo, decided with
  the trained egg prompt.
- **Sample tests**: 18 image tests with gold answers, run in the browser with an accuracy report,
  plus a comparison of prompt formats.

## How a decision is computed (trained format)

1. Query: the media placeholders first, then `task: classification | query: {question}`, then
   `\nContext: {context}` when a context is given. The model embeds media and text together.
2. Each option: `title: none | text: {description}`, embedded once and cached.
3. Embedding: mean pooling over the attention mask, then L2 normalization (768 numbers).
4. P(option) = softmax(scale * cosine). The scale was learned in training: 19.96
   (temperature 1 / scale, about 0.0501), stored in `models/egemma2-decide/decision.json`.

Yes / no questions keep the question in the query and score the options `Yes` and `No`.
The older prompt formats of the first version of this page remain selectable for comparison.

## Files

- `index.html`, `style.css`, `app.js`: the page.
- `worker.js`: loads the model in a Web Worker and returns embeddings.
- `models/egemma2-decide/`: ONNX model, config, processor and tokenizer files, and `decision.json`
  (scale and prompt formats). Text and vision graphs in `q4f16` (as in the model repo) and in a portable
  `q4` (the model repo's `onnx_portable/` files: 4-bit weights, with plain 8-bit gathers for the embedding
  and position tables, so the same files also run on the WASM runtime, which lacks GatherBlockQuantized),
  plus the 4-bit audio encoder. fp16 and fp32 files are not copied here because Space storage is limited to 1 GB.
- `examples/`: sample images and `samples.json` (name, files, state, type, question, options, gold);
  `examples/egg-test/`: held-out egg candling photos.
- `CREDITS.md`: sources and licenses of the sample images.

Code: Apache-2.0. Model: derived from google/embeddinggemma-2 (Apache-2.0).