Blame for README.md

---
base_model: meta-models/Muse-Glimmer-30B
base_model_relation: adapter
library_name: peft
license: apache-2.0
language: [en]
pipeline_tag: image-text-to-text
tags: [trained-eye, qlora, lora, axolotl, smart-glasses, egocentric-vision]
---

# Trained Eye: watch skills for Muse Glimmer 30B (LoRA adapter)

**What this is.** A QLoRA adapter (PEFT) for `meta-models/Muse-Glimmer-30B` (revision `a4e59da52a7bc87ae7251dd5545c0dd437c44b68`), from
[Trained Eye](https://github.com/amaan784/trained-eye), which lets Meta glasses watch for something and say when it
happens. Given one first-person camera frame and a condition in the prompt, it answers
yes or no. It was trained on two skills at once (the training run is called
`product`): `burner_left_on` ("Tell me when a burner has been left on"), `fridge_left_open` ("Tell me when the fridge door is left open"). It is not a model on its own: it needs the base model's weights.

[Code](https://github.com/amaan784/trained-eye) | [Playground](https://aas2438--trained-eye-playground-web.us-east.modal.run) | [Training runs (W&B)](https://wandb.ai/amaan784-columbia-university/trained-eye) |
[How it works](https://aas2438--trained-eye-playground-web.us-east.modal.run/how-it-works)

## What is in this repository

| File | What it is |
|---|---|
| `adapter_model.safetensors` | The LoRA weights: rank 32 on every attention and MLP projection of the language model's layers; the vision tower is unchanged. 767 MB. |
| `adapter_config.json` | PEFT's settings for those weights: the base model, the rank, alpha and the target modules. |
| `config.json` | The model configuration the training run saved with the adapter. |
| `processor_config.json` | The image processor's settings, saved with the adapter. |
| `tokenizer.json` | The tokenizer, saved with the adapter. |
| `tokenizer_config.json` | The tokenizer's settings, saved with the adapter. |
| `chat_template.jinja` | The chat template the prompts were built with. |
| `manifests/<skill>.jsonl` | One line per generated training image (300 for burner_left_on, 300 for fridge_left_open): Muse Image's edit prompt, the answer it was made for (yes, or no for a near-miss), and the SHA-256 of the frame it edited and of the image. The images themselves are not here. |
| `LICENSE` | The base model's license text (Apache-2.0). |
| `release.json` | Every file here with its size and SHA-256. |
| `README.md` | This card. |

Not in this repository:

- The base model's weights: get `meta-models/Muse-Glimmer-30B` at revision `a4e59da52a7bc87ae7251dd5545c0dd437c44b68` from Hugging Face, under its own terms.
- The training images. The generated images are never published (they keep Muse Image's
  provenance and stay with the author), and neither are the teaching frames they were
  edited from. `manifests/` says how each generated image was made.
- The test clips and their labels, and the training code, which is on [GitHub](https://github.com/amaan784/trained-eye).

## How to use it

Send each frame the way the training examples were built:

- System message: `You watch first-person camera frames for one condition. Answer only yes or no.`
- User message: the frame, padded (not cropped) to 512 x 512 pixels with black
  bars, then this text: `Condition: <what to watch for>. Not: <a near-miss>; <another>.
  Is the condition true in this frame?`

The conditions it was trained with, word for word:

| Skill | Asked as | Condition | Not |
|---|---|---|---|
| burner_left_on | Tell me when a burner has been left on | a burner is visibly on with a blue gas flame or glowing red hot and no pot or pan on it | a burner is off, dark and unlit with no flame or glow; a burner is on but has a pot or pan sitting on it; an unlit neighbouring burner on the same cooktop; shiny foil burner liner or metal grate with no flame or red glow |
| fridge_left_open | Tell me when the fridge door is left open | the fridge door stands open with its brightly lit interior, food shelves, and door bins exposed | the fridge doors are fully closed, showing only the dark flat exterior with no interior light visible; the bottom freezer drawer is closed beneath the fridge compartment; a closed kitchen cabinet above the refrigerator; an open cabinet or pantry door that is not the refrigerator |

Glimmer writes a short reasoning before it answers, so read the last `yes` or `no` token it
generates, and take the chance of yes from that token's top logprobs (yes over yes plus
no). An alert should wait until that chance has stayed over a threshold, calibrated on
your own labelled clips, for the skill's hold (Trained Eye's watcher does this).

What ran on Nov 4: the adapter merged into the bf16 base with PEFT's
`merge_and_unload()` on one 80 GB H100, then served with vLLM's
`vllm/vllm-openai:muse-glimmer` image, each request with `logprobs` on, `top_logprobs`
10, up to 512 new tokens and `chat_template_kwargs` `{"reasoning_strength": "low"}`.
Loading the adapter with vLLM's `--enable-lora` did not start for this model on that
image. The merge, as the project's `te export` runs it:

```python
import torch
from peft import PeftModel
from transformers import AutoModelForImageTextToText, AutoProcessor

base, revision = "meta-models/Muse-Glimmer-30B", "a4e59da52a7bc87ae7251dd5545c0dd437c44b68"
model = AutoModelForImageTextToText.from_pretrained(
    base, revision=revision, torch_dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(model, "amaan784/trained-eye").merge_and_unload()
model.save_pretrained("trained-eye-merged")
AutoProcessor.from_pretrained(base, revision=revision).save_pretrained("trained-eye-merged")
```

## Results

In short: on other kitchens filmed first-person on phones, the fine-tuned model spotted the condition about as accurately as base Glimmer and less accurately than Muse Spark (frame F1 0.87, 0.88 and 0.94), answering in 1.64 s a frame against 2.53 s and 11.80 s, at $4.00 an hour against $16.53 for Spark.

By skill, the fine-tuned model was the least accurate of the three on burner_left_on (frame F1 0.64, against 0.79 for base Glimmer and 0.97 for Muse Spark) and the most accurate of the three on fridge_left_open (frame F1 0.95, against 0.91 and 0.92). burner_left_on has no calibration clip, so the fine-tuned model runs it at the default threshold, 0.50.

| Skill | Model | Frame precision | Frame recall | Frame F1 | Frame accuracy | Seconds per frame | Dollars per hour |
|---|---|---|---|---|---|---|---|
| burner_left_on | this adapter | 1.00 | 0.47 | 0.64 | 0.70 | 1.69 | $4.10 |
| burner_left_on | base Muse Glimmer 30B | 1.00 | 0.65 | 0.79 | 0.80 | 3.22 | $4.26 |
| burner_left_on | Muse Spark (API) | 0.98 | 0.96 | 0.97 | 0.97 | 11.93 | $16.53 |
| fridge_left_open | this adapter | 0.93 | 0.96 | 0.95 | 0.95 | 1.58 | $3.95 |
| fridge_left_open | base Muse Glimmer 30B | 0.86 | 0.97 | 0.91 | 0.91 | 2.34 | $3.95 |
| fridge_left_open | Muse Spark (API) | 0.87 | 0.98 | 0.92 | 0.91 | 11.67 | $16.53 |

Test footage: other kitchens filmed first-person on phones (handheld; four fridges and two stoves the model was not taught on; labels drafted by an AI assistant, see docs/evaluation.md). From `te eval`'s `results.json` (SHA-256 `6704865a78632b3e1d28aa5b50c1f56ca68537b9307949ead80969b32ecb1d8a`); evaluations scored: 6, pending: 0. n/a means there was nothing to compute it from. Each skill alerts only once its condition has stayed true for its hold: event_recall_in_window counts an alert within the skill's window after the hold ends, and time_to_alert_median_s is counted from the end of the hold. Holds and windows: burner_left_on, a 10 s hold and a 5 s window; fridge_left_open, a 20 s hold and a 10 s window. Thresholds were calibrated for frame F1 (this footage has too few events past the hold to judge alerts).

<details><summary>Every scored evaluation</summary>

| evaluation | experiment | skill | variant | hours | precision | recall | f1 | event_recall_in_window | false_alarms_per_hour | time_to_alert_median_s | latency_median_s | usd_per_hour |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| phone-exp4-finetuned-fridge_left_open | 4 | fridge_left_open | finetuned | 0.1322 | 0.9328 | 0.9633 | 0.9478 | 0.6667 | 7.563 | 0.0 | 1.5845 | 3.95 |
| phone-exp4-base-fridge_left_open | 4 | fridge_left_open | base | 0.1322 | 0.8614 | 0.9746 | 0.9145 | 0.6667 | 7.563 | 0.0 | 2.341 | 3.95 |
| phone-exp4-spark-fridge_left_open | 4 | fridge_left_open | spark | 0.1322 | 0.8723 | 0.9755 | 0.921 | 0.6667 | 7.563 | 0.0 | 11.6665 | 16.5276 |
| phone-exp4-finetuned-burner_left_on | 4 | burner_left_on | finetuned | 0.0594 | 1.0 | 0.4667 | 0.6364 | 0.0 | 0.0 | 13.0 | 1.689 | 4.1045 |
| phone-exp4-base-burner_left_on | 4 | burner_left_on | base | 0.0594 | 1.0 | 0.65 | 0.7879 | 0.5 | 0.0 | 0.0 | 3.2215 | 4.2589 |
| phone-exp4-spark-burner_left_on | 4 | burner_left_on | spark | 0.0594 | 0.9829 | 0.9583 | 0.9705 | 0.5 | 16.8224 | 9.5 | 11.9315 | 16.5276 |

</details>

The full tables, the EPIC-KITCHENS-100 results and how it was measured are on GitHub:
[results](https://github.com/amaan784/trained-eye/blob/main/eval/results.md), [EPIC-KITCHENS](https://github.com/amaan784/trained-eye/blob/main/eval/epic/results.md),
[method](https://github.com/amaan784/trained-eye/blob/main/docs/evaluation.md).

## Training

Meta's cookbook QLoRA recipe for Muse Glimmer (meta-oss-cookbook at commit b30ca2c,
muse-glimmer/post-training/sft/axolotl) with Axolotl 0.19.0, unchanged except the paths, the
W&B names and the gradient accumulation that keeps its global batch of 32 on one GPU.
The training data was made through the Meta Model API:
compiler `muse-wonder-1.0`, sam `sam-3.1`, image `muse-image-1.0`, labeler `muse-spark-1.3`.

| Setting | Value |
|---|---|
| `adapter` | `qlora` |
| `load_in_4bit` | `true` |
| `lora_r` | `32` |
| `lora_alpha` | `16` |
| `lora_dropout` | `0.05` |
| `lora_target_modules` | `model\.language_model\.layers\.\d+\.(self_attn\.(q\|k\|v\|o)_proj\|mlp\.(gate\|up\|down)_proj)` |
| `sequence_len` | `4096` |
| `micro_batch_size` | `1` |
| `num_epochs` | `1` |
| `optimizer` | `adamw_bnb_8bit` |
| `lr_scheduler` | `cosine` |
| `learning_rate` | `0.0002` |
| `warmup_ratio` | `0.1` |
| `weight_decay` | `0.0` |
| `bf16` | `true` |
| `attn_implementation` | `flash_attention_2` |
| `seed` | `42` |

## Training data

Bundle `mixed`: 603 training and 65 validation examples (train.jsonl SHA-256 `c056bbd61a66ece9bc9b4ec2af87f8b587a6cf952bf0d020fef29f4624473e49`). Real: keyframes of the user's own teaching clips, labelled by the user or else by Muse Spark. Generated: images Muse Image made by editing those same keyframes (the condition, near-misses, other light and angles), labelled by Muse Spark and kept only where Spark agreed with the label it was made for.

| Skill | Split | Real yes | Real no | Generated yes | Generated no |
|---|---|---|---|---|---|
| burner_left_on | train | 33 | 0 | 125 | 135 |
| burner_left_on | val | 7 | 0 | 12 | 15 |
| fridge_left_open | train | 45 | 0 | 134 | 131 |
| fridge_left_open | val | 1 | 0 | 16 | 14 |

The generated images are not included. `manifests/` lists every edit prompt with the SHA-256 of its keyframe and of its image. For burner_left_on, fridge_left_open, `te regenerate` needs the original keyframes, which are not published.

## Uses

Intended: yes/no watch skills on first-person frames from smart glasses or a phone, each
taught by its user from their own footage: the state of an object, a display, a sign or a
vehicle (has the fridge been left open? is the tap still running?), with an alert threshold
calibrated on the user's own validation clips and a hold before each alert. An alert is a
reminder: it never replaces a smoke alarm or turning a burner off.

Out of scope: people (identifying, recognising or tracking anyone), bodies, health,
medication and other sensitive traits, covert monitoring, sensitive places, sports analysis
or coaching, and anything where a missed or false alert could put someone at risk.

## Limitations

- Each skill comes from one user's scenes and camera; others are untested beyond the
  evaluation above. Most training images are Muse Image edits with Muse Spark's labels, which can be wrong.
- It answers per frame: alerts need the watcher's smoothing, each skill's hold, a
  calibrated threshold and a cooldown. The vision tower stayed frozen, so small, distant or glary targets stay hard.
- Scoring reads the server's top logprobs for the yes or no token, so the server must return them.
- In plain words: overall it is about as accurate as base Glimmer and faster; it is the best
  of the three on the fridge and the weakest on the burner, so do not read it as an accuracy
  gain from fine-tuning on every skill.
- It watches only what the glasses can see: look away and it cannot tell whether the
  condition is still true.
- If Muse Spark's check fails outright (an error, not a no), the alert is still spoken and
  logged as unconfirmed, on purpose, so a cloud outage never silences an alert; a no from
  Spark holds it back.
- The test labels were drafted by an AI assistant, too few events lasted past the hold to
  judge alerts well, and Experiments 2 and 3 did not run.
- Validation during training was split by example, not by source frame, so related images
  can sit on both sides; the separate test footage is not affected.

## Privacy and provenance

- Faces are blurred in everything published with this model. Watch mode does not store
  frames unless the user saves a clip, and never hides the glasses' capture LED.
- Muse Image outputs keep their watermark and provenance metadata, byte for byte, and are
  never published.

## License

`apache-2.0` as the base model's (meta-models/Muse-Glimmer-30B at revision `a4e59da52a7bc87ae7251dd5545c0dd437c44b68`; its license text is in `LICENSE`). These weights are a modification of that model: QLoRA fine-tuning on the data above (for a merged build, that adapter merged into the base). Any use is also subject to Meta's [Muse Glimmer Usage Policy](https://huggingface.co/meta-models/Muse-Glimmer-30B/blob/main/USAGE_POLICY.md).

## Citation

```bibtex
@software{sheikh2026trainedeye,
  author = {Sheikh, Amaan},
  title  = {Trained Eye: Teach Your Glasses to Notice Anything},
  year   = {2026},
  month  = nov,
  url    = {https://github.com/amaan784/trained-eye}
}
```