Trained Eye release product

README.md

@@ -3,14 +3,118 @@ base_model: meta-models/Muse-Glimmer-30B
 base_model_relation: adapter
 library_name: peft
 license: apache-2.0
+language: [en]
 pipeline_tag: image-text-to-text
-tags: [trained-eye, qlora, axolotl, smart-glasses]
+tags: [trained-eye, qlora, lora, axolotl, smart-glasses, egocentric-vision]
 ---
 
-# Trained Eye watch skills (LoRA adapter)
+# Trained Eye: watch skills for Muse Glimmer 30B (LoRA adapter)
 
-A QLoRA adapter (PEFT) for `meta-models/Muse-Glimmer-30B` at revision `a4e59da52a7bc87ae7251dd5545c0dd437c44b68`. It answers yes or no to one watch condition,
-given in the prompt, about a first-person camera frame. Run `product`; skills: burner_left_on, fridge_left_open.
+**What this is.** A QLoRA adapter (PEFT) for `meta-models/Muse-Glimmer-30B` (revision `a4e59da52a7bc87ae7251dd5545c0dd437c44b68`), from
+[Trained Eye](https://github.com/amaan784/trained-eye), which lets Meta glasses watch for something and say when it
+happens. Given one first-person camera frame and a condition in the prompt, it answers
+yes or no. It was trained on two skills at once (the training run is called
+`product`): `burner_left_on` ("Tell me when a burner has been left on"), `fridge_left_open` ("Tell me when the fridge door is left open"). It is not a model on its own: it needs the base model's weights.
+
+[Code](https://github.com/amaan784/trained-eye) | [Playground](https://aas2438--trained-eye-playground-web.us-east.modal.run) | [Training runs (W&B)](https://wandb.ai/amaan784-columbia-university/trained-eye) |
+[How it works](https://aas2438--trained-eye-playground-web.us-east.modal.run/how-it-works)
+
+## What is in this repository
+
+| File | What it is |
+|---|---|
+| `adapter_model.safetensors` | The LoRA weights: rank 32 on every attention and MLP projection of the language model's layers; the vision tower is unchanged. 767 MB. |
+| `adapter_config.json` | PEFT's settings for those weights: the base model, the rank, alpha and the target modules. |
+| `config.json` | The model configuration the training run saved with the adapter. |
+| `processor_config.json` | The image processor's settings, saved with the adapter. |
+| `tokenizer.json` | The tokenizer, saved with the adapter. |
+| `tokenizer_config.json` | The tokenizer's settings, saved with the adapter. |
+| `chat_template.jinja` | The chat template the prompts were built with. |
+| `manifests/<skill>.jsonl` | One line per generated training image (300 for burner_left_on, 300 for fridge_left_open): Muse Image's edit prompt, the answer it was made for (yes, or no for a near-miss), and the SHA-256 of the frame it edited and of the image. The images themselves are not here. |
+| `LICENSE` | The base model's license text (Apache-2.0). |
+| `release.json` | Every file here with its size and SHA-256. |
+| `README.md` | This card. |
+
+Not in this repository:
+
+- The base model's weights: get `meta-models/Muse-Glimmer-30B` at revision `a4e59da52a7bc87ae7251dd5545c0dd437c44b68` from Hugging Face, under its own terms.
+- The training images. The generated images are never published (they keep Muse Image's
+  provenance and stay with the author), and neither are the teaching frames they were
+  edited from. `manifests/` says how each generated image was made.
+- The test clips and their labels, and the training code, which is on [GitHub](https://github.com/amaan784/trained-eye).
+
+## How to use it
+
+Send each frame the way the training examples were built:
+
+- System message: `You watch first-person camera frames for one condition. Answer only yes or no.`
+- User message: the frame, padded (not cropped) to 512 x 512 pixels with black
+  bars, then this text: `Condition: <what to watch for>. Not: <a near-miss>; <another>.
+  Is the condition true in this frame?`
+
+The conditions it was trained with, word for word:
+
+| Skill | Asked as | Condition | Not |
+|---|---|---|---|
+| burner_left_on | Tell me when a burner has been left on | a burner is visibly on with a blue gas flame or glowing red hot and no pot or pan on it | a burner is off, dark and unlit with no flame or glow; a burner is on but has a pot or pan sitting on it; an unlit neighbouring burner on the same cooktop; shiny foil burner liner or metal grate with no flame or red glow |
+| fridge_left_open | Tell me when the fridge door is left open | the fridge door stands open with its brightly lit interior, food shelves, and door bins exposed | the fridge doors are fully closed, showing only the dark flat exterior with no interior light visible; the bottom freezer drawer is closed beneath the fridge compartment; a closed kitchen cabinet above the refrigerator; an open cabinet or pantry door that is not the refrigerator |
+
+Glimmer writes a short reasoning before it answers, so read the last `yes` or `no` token it
+generates, and take the chance of yes from that token's top logprobs (yes over yes plus
+no). An alert should wait until that chance has stayed over a threshold, calibrated on
+your own labelled clips, for the skill's hold (Trained Eye's watcher does this).
+
+What ran on Nov 4: the adapter merged into the bf16 base with PEFT's
+`merge_and_unload()` on one 80 GB H100, then served with vLLM's
+`vllm/vllm-openai:muse-glimmer` image, each request with `logprobs` on, `top_logprobs`
+10, up to 512 new tokens and `chat_template_kwargs` `{"reasoning_strength": "low"}`.
+Loading the adapter with vLLM's `--enable-lora` did not start for this model on that
+image. The merge, as the project's `te export` runs it:
+
+```python
+import torch
+from peft import PeftModel
+from transformers import AutoModelForImageTextToText, AutoProcessor
+
+base, revision = "meta-models/Muse-Glimmer-30B", "a4e59da52a7bc87ae7251dd5545c0dd437c44b68"
+model = AutoModelForImageTextToText.from_pretrained(
+    base, revision=revision, torch_dtype=torch.bfloat16, device_map="auto")
+model = PeftModel.from_pretrained(model, "amaan784/trained-eye").merge_and_unload()
+model.save_pretrained("trained-eye-merged")
+AutoProcessor.from_pretrained(base, revision=revision).save_pretrained("trained-eye-merged")
+```
+
+## Results
+
+In short: on other kitchens filmed first-person on phones, the fine-tuned model spotted the condition about as accurately as base Glimmer and less accurately than Muse Spark (frame F1 0.87, 0.88 and 0.94), answering in 1.64 s a frame against 2.53 s and 11.80 s, at $4.00 an hour against $16.53 for Spark.
+
+| Skill | Model | Frame precision | Frame recall | Frame F1 | Frame accuracy | Seconds per frame | Dollars per hour |
+|---|---|---|---|---|---|---|---|
+| burner_left_on | this adapter | 1.00 | 0.47 | 0.64 | 0.70 | 1.69 | $4.10 |
+| burner_left_on | base Muse Glimmer 30B | 1.00 | 0.65 | 0.79 | 0.80 | 3.22 | $4.26 |
+| burner_left_on | Muse Spark (API) | 0.98 | 0.96 | 0.97 | 0.97 | 11.93 | $16.53 |
+| fridge_left_open | this adapter | 0.93 | 0.96 | 0.95 | 0.95 | 1.58 | $3.95 |
+| fridge_left_open | base Muse Glimmer 30B | 0.86 | 0.97 | 0.91 | 0.91 | 2.34 | $3.95 |
+| fridge_left_open | Muse Spark (API) | 0.87 | 0.98 | 0.92 | 0.91 | 11.67 | $16.53 |
+
+Test footage: other kitchens filmed first-person on phones (handheld; four fridges and two stoves the model was not taught on; labels drafted by an AI assistant, see docs/evaluation.md). From `te eval`'s `results.json` (SHA-256 `6704865a78632b3e1d28aa5b50c1f56ca68537b9307949ead80969b32ecb1d8a`); evaluations scored: 6, pending: 0. n/a means there was nothing to compute it from. Each skill alerts only once its condition has stayed true for its hold: event_recall_in_window counts an alert within the skill's window after the hold ends, and time_to_alert_median_s is counted from the end of the hold. Holds and windows: burner_left_on, a 10 s hold and a 5 s window; fridge_left_open, a 20 s hold and a 10 s window. Thresholds were calibrated for frame F1 (this footage has too few events past the hold to judge alerts).
+
+<details><summary>Every scored evaluation</summary>
+
+| evaluation | experiment | skill | variant | hours | precision | recall | f1 | event_recall_in_window | false_alarms_per_hour | time_to_alert_median_s | latency_median_s | usd_per_hour |
+|---|---|---|---|---|---|---|---|---|---|---|---|---|
+| phone-exp4-finetuned-fridge_left_open | 4 | fridge_left_open | finetuned | 0.1322 | 0.9328 | 0.9633 | 0.9478 | 0.6667 | 7.563 | 0.0 | 1.5845 | 3.95 |
+| phone-exp4-base-fridge_left_open | 4 | fridge_left_open | base | 0.1322 | 0.8614 | 0.9746 | 0.9145 | 0.6667 | 7.563 | 0.0 | 2.341 | 3.95 |
+| phone-exp4-spark-fridge_left_open | 4 | fridge_left_open | spark | 0.1322 | 0.8723 | 0.9755 | 0.921 | 0.6667 | 7.563 | 0.0 | 11.6665 | 16.5276 |
+| phone-exp4-finetuned-burner_left_on | 4 | burner_left_on | finetuned | 0.0594 | 1.0 | 0.4667 | 0.6364 | 0.0 | 0.0 | 13.0 | 1.689 | 4.1045 |
+| phone-exp4-base-burner_left_on | 4 | burner_left_on | base | 0.0594 | 1.0 | 0.65 | 0.7879 | 0.5 | 0.0 | 0.0 | 3.2215 | 4.2589 |
+| phone-exp4-spark-burner_left_on | 4 | burner_left_on | spark | 0.0594 | 0.9829 | 0.9583 | 0.9705 | 0.5 | 16.8224 | 9.5 | 11.9315 | 16.5276 |
+
+</details>
+
+The full tables, the EPIC-KITCHENS-100 results and how it was measured are on GitHub:
+[results](https://github.com/amaan784/trained-eye/blob/main/eval/results.md), [EPIC-KITCHENS](https://github.com/amaan784/trained-eye/blob/main/eval/epic/results.md),
+[method](https://github.com/amaan784/trained-eye/blob/main/docs/evaluation.md).
 
 ## Training
 
@@ -53,19 +157,6 @@ Bundle `mixed`: 603 training and 65 validation examples (train.jsonl SHA-256 `c0
 
 The generated images are not included. `manifests/` lists every edit prompt with the SHA-256 of its keyframe and of its image. For burner_left_on, fridge_left_open, `te regenerate` needs the original keyframes, which are not published.
 
-## Evaluation
-
-Test footage: other kitchens filmed first-person on phones (handheld; four fridges and two stoves the model was not taught on; labels drafted by an AI assistant, see docs/evaluation.md). From `te eval`'s `results.json` (SHA-256 `6704865a78632b3e1d28aa5b50c1f56ca68537b9307949ead80969b32ecb1d8a`); evaluations scored: 6, pending: 0. n/a means there was nothing to compute it from. Each skill alerts only once its condition has stayed true for its hold: event_recall_in_window counts an alert within the skill's window after the hold ends, and time_to_alert_median_s is counted from the end of the hold. Holds and windows: burner_left_on, a 10 s hold and a 5 s window; fridge_left_open, a 20 s hold and a 10 s window. Thresholds were calibrated for frame F1 (this footage has too few events past the hold to judge alerts).
-
-| evaluation | experiment | skill | variant | hours | precision | recall | f1 | event_recall_in_window | false_alarms_per_hour | time_to_alert_median_s | latency_median_s | usd_per_hour |
-|---|---|---|---|---|---|---|---|---|---|---|---|---|
-| phone-exp4-finetuned-fridge_left_open | 4 | fridge_left_open | finetuned | 0.1322 | 0.9328 | 0.9633 | 0.9478 | 0.6667 | 7.563 | 0.0 | 1.5845 | 3.95 |
-| phone-exp4-base-fridge_left_open | 4 | fridge_left_open | base | 0.1322 | 0.8614 | 0.9746 | 0.9145 | 0.6667 | 7.563 | 0.0 | 2.341 | 3.95 |
-| phone-exp4-spark-fridge_left_open | 4 | fridge_left_open | spark | 0.1322 | 0.8723 | 0.9755 | 0.921 | 0.6667 | 7.563 | 0.0 | 11.6665 | 16.5276 |
-| phone-exp4-finetuned-burner_left_on | 4 | burner_left_on | finetuned | 0.0594 | 1.0 | 0.4667 | 0.6364 | 0.0 | 0.0 | 13.0 | 1.689 | 4.1045 |
-| phone-exp4-base-burner_left_on | 4 | burner_left_on | base | 0.0594 | 1.0 | 0.65 | 0.7879 | 0.5 | 0.0 | 0.0 | 3.2215 | 4.2589 |
-| phone-exp4-spark-burner_left_on | 4 | burner_left_on | spark | 0.0594 | 0.9829 | 0.9583 | 0.9705 | 0.5 | 16.8224 | 9.5 | 11.9315 | 16.5276 |
-
 ## Uses
 
 Intended: yes/no watch skills on first-person frames from smart glasses or a phone, each
@@ -96,3 +187,15 @@ or coaching, and anything where a missed or false alert could put someone at ris
 ## License
 
 `apache-2.0` as the base model's (meta-models/Muse-Glimmer-30B at revision `a4e59da52a7bc87ae7251dd5545c0dd437c44b68`; its license text is in `LICENSE`). These weights are a modification of that model: QLoRA fine-tuning on the data above (for a merged build, that adapter merged into the base). Any use is also subject to Meta's [Muse Glimmer Usage Policy](https://huggingface.co/meta-models/Muse-Glimmer-30B/blob/main/USAGE_POLICY.md).
+
+## Citation
+
+```bibtex
+@software{sheikh2026trainedeye,
+  author = {Sheikh, Amaan},
+  title  = {Trained Eye: Teach Your Glasses to Notice Anything},
+  year   = {2026},
+  month  = nov,
+  url    = {https://github.com/amaan784/trained-eye}
+}
+```

release.json

@@ -11,8 +11,8 @@
   },
   {
    "path": "README.md",
-   "bytes": 6421,
-   "sha256": "5a94223f6ffc7b35c084a0054460dec6d83d4044fe91e8e5f1b418fcb560e044"
+   "bytes": 13141,
+   "sha256": "3d64879065af36515d76bc616b8ee7262c5dc3c1dea5abd95998f910ea504bf8"
   },
   {
    "path": "adapter_config.json",