Trained Eye release product

da5da2e verified about 1 hour ago

14.3 kB

---
base_model: meta-models/Muse-Glimmer-30B
base_model_relation: adapter
library_name: peft
license: apache-2.0
language: [en]
pipeline_tag: image-text-to-text
tags: [trained-eye, qlora, lora, axolotl, smart-glasses, egocentric-vision]
---
# Trained Eye: watch skills for Muse Glimmer 30B (LoRA adapter)

**What this is.** A QLoRA adapter (PEFT) for `meta-models/Muse-Glimmer-30B` (revision `a4e59da52a7bc87ae7251dd5545c0dd437c44b68`), from
[Trained Eye](https://github.com/amaan784/trained-eye), which lets Meta glasses watch for something and say when it
happens. Given one first-person camera frame and a condition in the prompt, it answers
yes or no. It was trained on two skills at once (the training run is called
`product`): `burner_left_on` ("Tell me when a burner has been left on"), `fridge_left_open` ("Tell me when the fridge door is left open"). It is not a model on its own: it needs the base model's weights.

[Code](https://github.com/amaan784/trained-eye) | [Playground](https://aas2438--trained-eye-playground-web.us-east.modal.run) | [Training runs (W&B)](https://wandb.ai/amaan784-columbia-university/trained-eye) |
[How it works](https://aas2438--trained-eye-playground-web.us-east.modal.run/how-it-works)

## What is in this repository

| File | What it is |
|---|---|
| `adapter_model.safetensors` | The LoRA weights: rank 32 on every attention and MLP projection of the language model's layers; the vision tower is unchanged. 767 MB. |
| `adapter_config.json` | PEFT's settings for those weights: the base model, the rank, alpha and the target modules. |
| `config.json` | The model configuration the training run saved with the adapter. |
| `processor_config.json` | The image processor's settings, saved with the adapter. |
| `tokenizer.json` | The tokenizer, saved with the adapter. |
| `tokenizer_config.json` | The tokenizer's settings, saved with the adapter. |
| `chat_template.jinja` | The chat template the prompts were built with. |
| `manifests/<skill>.jsonl` | One line per generated training image (300 for burner_left_on, 300 for fridge_left_open): Muse Image's edit prompt, the answer it was made for (yes, or no for a near-miss), and the SHA-256 of the frame it edited and of the image. The images themselves are not here. |
| `LICENSE` | The base model's license text (Apache-2.0). |
| `release.json` | Every file here with its size and SHA-256. |
| `README.md` | This card. |

Not in this repository:

- The base model's weights: get `meta-models/Muse-Glimmer-30B` at revision `a4e59da52a7bc87ae7251dd5545c0dd437c44b68` from Hugging Face, under its own terms.
- The training images. The generated images are never published (they keep Muse Image's
  provenance and stay with the author), and neither are the teaching frames they were
  edited from. `manifests/` says how each generated image was made.
- The test clips and their labels, and the training code, which is on [GitHub](https://github.com/amaan784/trained-eye).

## How to use it

Send each frame the way the training examples were built:

- System message: `You watch first-person camera frames for one condition. Answer only yes or no.`
- User message: the frame, padded (not cropped) to 512 x 512 pixels with black
  bars, then this text: `Condition: <what to watch for>. Not: <a near-miss>; <another>.
  Is the condition true in this frame?`

The conditions it was trained with, word for word:

| Skill | Asked as | Condition | Not |
|---|---|---|---|
| burner_left_on | Tell me when a burner has been left on | a burner is visibly on with a blue gas flame or glowing red hot and no pot or pan on it | a burner is off, dark and unlit with no flame or glow; a burner is on but has a pot or pan sitting on it; an unlit neighbouring burner on the same cooktop; shiny foil burner liner or metal grate with no flame or red glow |
| fridge_left_open | Tell me when the fridge door is left open | the fridge door stands open with its brightly lit interior, food shelves, and door bins exposed | the fridge doors are fully closed, showing only the dark flat exterior with no interior light visible; the bottom freezer drawer is closed beneath the fridge compartment; a closed kitchen cabinet above the refrigerator; an open cabinet or pantry door that is not the refrigerator |

Glimmer writes a short reasoning before it answers, so read the last `yes` or `no` token it
generates, and take the chance of yes from that token's top logprobs (yes over yes plus
no). An alert should wait until that chance has stayed over a threshold, calibrated on
your own labelled clips, for the skill's hold (Trained Eye's watcher does this).

What ran on Nov 4: the adapter merged into the bf16 base with PEFT's
`merge_and_unload()` on one 80 GB H100, then served with vLLM's
`vllm/vllm-openai:muse-glimmer` image, each request with `logprobs` on, `top_logprobs`
10, up to 512 new tokens and `chat_template_kwargs` `{"reasoning_strength": "low"}`.
Loading the adapter with vLLM's `--enable-lora` did not start for this model on that
image. The merge, as the project's `te export` runs it:

```python
import torch
from peft import PeftModel
from transformers import AutoModelForImageTextToText, AutoProcessor
base, revision = "meta-models/Muse-Glimmer-30B", "a4e59da52a7bc87ae7251dd5545c0dd437c44b68"
model = AutoModelForImageTextToText.from_pretrained(
    base, revision=revision, torch_dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(model, "amaan784/trained-eye").merge_and_unload()
model.save_pretrained("trained-eye-merged")
AutoProcessor.from_pretrained(base, revision=revision).save_pretrained("trained-eye-merged")
```

## Results

In short: on other kitchens filmed first-person on phones, the fine-tuned model spotted the condition about as accurately as base Glimmer and less accurately than Muse Spark (frame F1 0.87, 0.88 and 0.94), answering in 1.64 s a frame against 2.53 s and 11.80 s, at $4.00 an hour against $16.53 for Spark.

By skill, the fine-tuned model was the least accurate of the three on burner_left_on (frame F1 0.64, against 0.79 for base Glimmer and 0.97 for Muse Spark) and the most accurate of the three on fridge_left_open (frame F1 0.95, against 0.91 and 0.92). burner_left_on has no calibration clip, so the fine-tuned model runs it at the default threshold, 0.50.

| Skill | Model | Frame precision | Frame recall | Frame F1 | Frame accuracy | Seconds per frame | Dollars per hour |
|---|---|---|---|---|---|---|---|
| burner_left_on | this adapter | 1.00 | 0.47 | 0.64 | 0.70 | 1.69 | $4.10 |
| burner_left_on | base Muse Glimmer 30B | 1.00 | 0.65 | 0.79 | 0.80 | 3.22 | $4.26 |
| burner_left_on | Muse Spark (API) | 0.98 | 0.96 | 0.97 | 0.97 | 11.93 | $16.53 |
| fridge_left_open | this adapter | 0.93 | 0.96 | 0.95 | 0.95 | 1.58 | $3.95 |
| fridge_left_open | base Muse Glimmer 30B | 0.86 | 0.97 | 0.91 | 0.91 | 2.34 | $3.95 |
| fridge_left_open | Muse Spark (API) | 0.87 | 0.98 | 0.92 | 0.91 | 11.67 | $16.53 |

Test footage: other kitchens filmed first-person on phones (handheld; four fridges and two stoves the model was not taught on; labels drafted by an AI assistant, see docs/evaluation.md). From `te eval`'s `results.json` (SHA-256 `6704865a78632b3e1d28aa5b50c1f56ca68537b9307949ead80969b32ecb1d8a`); evaluations scored: 6, pending: 0. n/a means there was nothing to compute it from. Each skill alerts only once its condition has stayed true for its hold: event_recall_in_window counts an alert within the skill's window after the hold ends, and time_to_alert_median_s is counted from the end of the hold. Holds and windows: burner_left_on, a 10 s hold and a 5 s window; fridge_left_open, a 20 s hold and a 10 s window. Thresholds were calibrated for frame F1 (this footage has too few events past the hold to judge alerts).
<details><summary>Every scored evaluation</summary>
| evaluation | experiment | skill | variant | hours | precision | recall | f1 | event_recall_in_window | false_alarms_per_hour | time_to_alert_median_s | latency_median_s | usd_per_hour |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| phone-exp4-finetuned-fridge_left_open | 4 | fridge_left_open | finetuned | 0.1322 | 0.9328 | 0.9633 | 0.9478 | 0.6667 | 7.563 | 0.0 | 1.5845 | 3.95 |
| phone-exp4-base-fridge_left_open | 4 | fridge_left_open | base | 0.1322 | 0.8614 | 0.9746 | 0.9145 | 0.6667 | 7.563 | 0.0 | 2.341 | 3.95 |
| phone-exp4-spark-fridge_left_open | 4 | fridge_left_open | spark | 0.1322 | 0.8723 | 0.9755 | 0.921 | 0.6667 | 7.563 | 0.0 | 11.6665 | 16.5276 |
| phone-exp4-finetuned-burner_left_on | 4 | burner_left_on | finetuned | 0.0594 | 1.0 | 0.4667 | 0.6364 | 0.0 | 0.0 | 13.0 | 1.689 | 4.1045 |
| phone-exp4-base-burner_left_on | 4 | burner_left_on | base | 0.0594 | 1.0 | 0.65 | 0.7879 | 0.5 | 0.0 | 0.0 | 3.2215 | 4.2589 |
| phone-exp4-spark-burner_left_on | 4 | burner_left_on | spark | 0.0594 | 0.9829 | 0.9583 | 0.9705 | 0.5 | 16.8224 | 9.5 | 11.9315 | 16.5276 |
</details>
The full tables, the EPIC-KITCHENS-100 results and how it was measured are on GitHub:
[results](https://github.com/amaan784/trained-eye/blob/main/eval/results.md), [EPIC-KITCHENS](https://github.com/amaan784/trained-eye/blob/main/eval/epic/results.md),
[method](https://github.com/amaan784/trained-eye/blob/main/docs/evaluation.md).
## Training
Meta's cookbook QLoRA recipe for Muse Glimmer (meta-oss-cookbook at commit b30ca2c,
muse-glimmer/post-training/sft/axolotl) with Axolotl 0.19.0, unchanged except the paths, the
W&B names and the gradient accumulation that keeps its global batch of 32 on one GPU.
The training data was made through the Meta Model API:
compiler `muse-wonder-1.0`, sam `sam-3.1`, image `muse-image-1.0`, labeler `muse-spark-1.3`.
| Setting | Value |
|---|---|
| `adapter` | `qlora` |
| `load_in_4bit` | `true` |
| `lora_r` | `32` |
| `lora_alpha` | `16` |
| `lora_dropout` | `0.05` |
| `lora_target_modules` | `model\.language_model\.layers\.\d+\.(self_attn\.(q\|k\|v\|o)_proj\|mlp\.(gate\|up\|down)_proj)` |
| `sequence_len` | `4096` |
| `micro_batch_size` | `1` |
| `num_epochs` | `1` |
| `optimizer` | `adamw_bnb_8bit` |
| `lr_scheduler` | `cosine` |
| `learning_rate` | `0.0002` |
| `warmup_ratio` | `0.1` |
| `weight_decay` | `0.0` |
| `bf16` | `true` |
| `attn_implementation` | `flash_attention_2` |
| `seed` | `42` |

## Training data

Bundle `mixed`: 603 training and 65 validation examples (train.jsonl SHA-256 `c056bbd61a66ece9bc9b4ec2af87f8b587a6cf952bf0d020fef29f4624473e49`). Real: keyframes of the user's own teaching clips, labelled by the user or else by Muse Spark. Generated: images Muse Image made by editing those same keyframes (the condition, near-misses, other light and angles), labelled by Muse Spark and kept only where Spark agreed with the label it was made for.

| Skill | Split | Real yes | Real no | Generated yes | Generated no |
|---|---|---|---|---|---|
| burner_left_on | train | 33 | 0 | 125 | 135 |
| burner_left_on | val | 7 | 0 | 12 | 15 |
| fridge_left_open | train | 45 | 0 | 134 | 131 |
| fridge_left_open | val | 1 | 0 | 16 | 14 |

The generated images are not included. `manifests/` lists every edit prompt with the SHA-256 of its keyframe and of its image. For burner_left_on, fridge_left_open, `te regenerate` needs the original keyframes, which are not published.

## Uses

Intended: yes/no watch skills on first-person frames from smart glasses or a phone, each
taught by its user from their own footage: the state of an object, a display, a sign or a
vehicle (has the fridge been left open? is the tap still running?), with an alert threshold
calibrated on the user's own validation clips and a hold before each alert. An alert is a
reminder: it never replaces a smoke alarm or turning a burner off.

Out of scope: people (identifying, recognising or tracking anyone), bodies, health,
medication and other sensitive traits, covert monitoring, sensitive places, sports analysis
or coaching, and anything where a missed or false alert could put someone at risk.

## Limitations

- Each skill comes from one user's scenes and camera; others are untested beyond the
  evaluation above. Most training images are Muse Image edits with Muse Spark's labels, which can be wrong.
- It answers per frame: alerts need the watcher's smoothing, each skill's hold, a
  calibrated threshold and a cooldown. The vision tower stayed frozen, so small, distant or glary targets stay hard.
- Scoring reads the server's top logprobs for the yes or no token, so the server must return them.
- In plain words: overall it is about as accurate as base Glimmer and faster; it is the best
  of the three on the fridge and the weakest on the burner, so do not read it as an accuracy
  gain from fine-tuning on every skill.
- It watches only what the glasses can see: look away and it cannot tell whether the
  condition is still true.
- If Muse Spark's check fails outright (an error, not a no), the alert is still spoken and
  logged as unconfirmed, on purpose, so a cloud outage never silences an alert; a no from
  Spark holds it back.
- The test labels were drafted by an AI assistant, too few events lasted past the hold to
  judge alerts well, and Experiments 2 and 3 did not run.
- Validation during training was split by example, not by source frame, so related images
  can sit on both sides; the separate test footage is not affected.

## Privacy and provenance

- Faces are blurred in everything published with this model. Watch mode does not store
  frames unless the user saves a clip, and never hides the glasses' capture LED.
- Muse Image outputs keep their watermark and provenance metadata, byte for byte, and are
  never published.

## License

`apache-2.0` as the base model's (meta-models/Muse-Glimmer-30B at revision `a4e59da52a7bc87ae7251dd5545c0dd437c44b68`; its license text is in `LICENSE`). These weights are a modification of that model: QLoRA fine-tuning on the data above (for a merged build, that adapter merged into the base). Any use is also subject to Meta's [Muse Glimmer Usage Policy](https://huggingface.co/meta-models/Muse-Glimmer-30B/blob/main/USAGE_POLICY.md).

## Citation

```bibtex
@software{sheikh2026trainedeye,
  author = {Sheikh, Amaan},
  title  = {Trained Eye: Teach Your Glasses to Notice Anything},
  year   = {2026},
  month  = nov,
  url    = {https://github.com/amaan784/trained-eye}
}
```

	---
	base_model: meta-models/Muse-Glimmer-30B
	base_model_relation: adapter
	library_name: peft
	license: apache-2.0
	language: [en]
	pipeline_tag: image-text-to-text
	tags: [trained-eye, qlora, lora, axolotl, smart-glasses, egocentric-vision]
	---

	# Trained Eye: watch skills for Muse Glimmer 30B (LoRA adapter)

	What this is. A QLoRA adapter (PEFT) for `meta-models/Muse-Glimmer-30B` (revision `a4e59da52a7bc87ae7251dd5545c0dd437c44b68`), from
	[Trained Eye](https://github.com/amaan784/trained-eye), which lets Meta glasses watch for something and say when it
	happens. Given one first-person camera frame and a condition in the prompt, it answers
	yes or no. It was trained on two skills at once (the training run is called
	`product`): `burner_left_on` ("Tell me when a burner has been left on"), `fridge_left_open` ("Tell me when the fridge door is left open"). It is not a model on its own: it needs the base model's weights.

	[Code](https://github.com/amaan784/trained-eye) \| [Playground](https://aas2438--trained-eye-playground-web.us-east.modal.run) \| [Training runs (W&B)](https://wandb.ai/amaan784-columbia-university/trained-eye) \|
	[How it works](https://aas2438--trained-eye-playground-web.us-east.modal.run/how-it-works)

	## What is in this repository

	\| File \| What it is \|
	\|---\|---\|
	\| `adapter_model.safetensors` \| The LoRA weights: rank 32 on every attention and MLP projection of the language model's layers; the vision tower is unchanged. 767 MB. \|
	\| `adapter_config.json` \| PEFT's settings for those weights: the base model, the rank, alpha and the target modules. \|
	\| `config.json` \| The model configuration the training run saved with the adapter. \|
	\| `processor_config.json` \| The image processor's settings, saved with the adapter. \|
	\| `tokenizer.json` \| The tokenizer, saved with the adapter. \|
	\| `tokenizer_config.json` \| The tokenizer's settings, saved with the adapter. \|
	\| `chat_template.jinja` \| The chat template the prompts were built with. \|
	\| `manifests/<skill>.jsonl` \| One line per generated training image (300 for burner_left_on, 300 for fridge_left_open): Muse Image's edit prompt, the answer it was made for (yes, or no for a near-miss), and the SHA-256 of the frame it edited and of the image. The images themselves are not here. \|
	\| `LICENSE` \| The base model's license text (Apache-2.0). \|
	\| `release.json` \| Every file here with its size and SHA-256. \|
	\| `README.md` \| This card. \|

	Not in this repository:

	- The base model's weights: get `meta-models/Muse-Glimmer-30B` at revision `a4e59da52a7bc87ae7251dd5545c0dd437c44b68` from Hugging Face, under its own terms.
	- The training images. The generated images are never published (they keep Muse Image's
	provenance and stay with the author), and neither are the teaching frames they were
	edited from. `manifests/` says how each generated image was made.
	- The test clips and their labels, and the training code, which is on [GitHub](https://github.com/amaan784/trained-eye).

	## How to use it

	Send each frame the way the training examples were built:

	- System message: `You watch first-person camera frames for one condition. Answer only yes or no.`
	- User message: the frame, padded (not cropped) to 512 x 512 pixels with black
	bars, then this text: `Condition: <what to watch for>. Not: <a near-miss>; <another>.
	Is the condition true in this frame?`

	The conditions it was trained with, word for word:

	\| Skill \| Asked as \| Condition \| Not \|
	\|---\|---\|---\|---\|
	\| burner_left_on \| Tell me when a burner has been left on \| a burner is visibly on with a blue gas flame or glowing red hot and no pot or pan on it \| a burner is off, dark and unlit with no flame or glow; a burner is on but has a pot or pan sitting on it; an unlit neighbouring burner on the same cooktop; shiny foil burner liner or metal grate with no flame or red glow \|
	\| fridge_left_open \| Tell me when the fridge door is left open \| the fridge door stands open with its brightly lit interior, food shelves, and door bins exposed \| the fridge doors are fully closed, showing only the dark flat exterior with no interior light visible; the bottom freezer drawer is closed beneath the fridge compartment; a closed kitchen cabinet above the refrigerator; an open cabinet or pantry door that is not the refrigerator \|

	Glimmer writes a short reasoning before it answers, so read the last `yes` or `no` token it
	generates, and take the chance of yes from that token's top logprobs (yes over yes plus
	no). An alert should wait until that chance has stayed over a threshold, calibrated on
	your own labelled clips, for the skill's hold (Trained Eye's watcher does this).

	What ran on Nov 4: the adapter merged into the bf16 base with PEFT's
	`merge_and_unload()` on one 80 GB H100, then served with vLLM's
	`vllm/vllm-openai:muse-glimmer` image, each request with `logprobs` on, `top_logprobs`
	10, up to 512 new tokens and `chat_template_kwargs` `{"reasoning_strength": "low"}`.
	Loading the adapter with vLLM's `--enable-lora` did not start for this model on that
	image. The merge, as the project's `te export` runs it:

	```python
	import torch
	from peft import PeftModel
	from transformers import AutoModelForImageTextToText, AutoProcessor

	base, revision = "meta-models/Muse-Glimmer-30B", "a4e59da52a7bc87ae7251dd5545c0dd437c44b68"
	model = AutoModelForImageTextToText.from_pretrained(
	base, revision=revision, torch_dtype=torch.bfloat16, device_map="auto")
	model = PeftModel.from_pretrained(model, "amaan784/trained-eye").merge_and_unload()
	model.save_pretrained("trained-eye-merged")
	AutoProcessor.from_pretrained(base, revision=revision).save_pretrained("trained-eye-merged")
	```

	## Results

	In short: on other kitchens filmed first-person on phones, the fine-tuned model spotted the condition about as accurately as base Glimmer and less accurately than Muse Spark (frame F1 0.87, 0.88 and 0.94), answering in 1.64 s a frame against 2.53 s and 11.80 s, at $4.00 an hour against $16.53 for Spark.

	By skill, the fine-tuned model was the least accurate of the three on burner_left_on (frame F1 0.64, against 0.79 for base Glimmer and 0.97 for Muse Spark) and the most accurate of the three on fridge_left_open (frame F1 0.95, against 0.91 and 0.92). burner_left_on has no calibration clip, so the fine-tuned model runs it at the default threshold, 0.50.

	\| Skill \| Model \| Frame precision \| Frame recall \| Frame F1 \| Frame accuracy \| Seconds per frame \| Dollars per hour \|
	\|---\|---\|---\|---\|---\|---\|---\|---\|
	\| burner_left_on \| this adapter \| 1.00 \| 0.47 \| 0.64 \| 0.70 \| 1.69 \| $4.10 \|
	\| burner_left_on \| base Muse Glimmer 30B \| 1.00 \| 0.65 \| 0.79 \| 0.80 \| 3.22 \| $4.26 \|
	\| burner_left_on \| Muse Spark (API) \| 0.98 \| 0.96 \| 0.97 \| 0.97 \| 11.93 \| $16.53 \|
	\| fridge_left_open \| this adapter \| 0.93 \| 0.96 \| 0.95 \| 0.95 \| 1.58 \| $3.95 \|
	\| fridge_left_open \| base Muse Glimmer 30B \| 0.86 \| 0.97 \| 0.91 \| 0.91 \| 2.34 \| $3.95 \|
	\| fridge_left_open \| Muse Spark (API) \| 0.87 \| 0.98 \| 0.92 \| 0.91 \| 11.67 \| $16.53 \|

	Test footage: other kitchens filmed first-person on phones (handheld; four fridges and two stoves the model was not taught on; labels drafted by an AI assistant, see docs/evaluation.md). From `te eval`'s `results.json` (SHA-256 `6704865a78632b3e1d28aa5b50c1f56ca68537b9307949ead80969b32ecb1d8a`); evaluations scored: 6, pending: 0. n/a means there was nothing to compute it from. Each skill alerts only once its condition has stayed true for its hold: event_recall_in_window counts an alert within the skill's window after the hold ends, and time_to_alert_median_s is counted from the end of the hold. Holds and windows: burner_left_on, a 10 s hold and a 5 s window; fridge_left_open, a 20 s hold and a 10 s window. Thresholds were calibrated for frame F1 (this footage has too few events past the hold to judge alerts).

	<details><summary>Every scored evaluation</summary>

	\| evaluation \| experiment \| skill \| variant \| hours \| precision \| recall \| f1 \| event_recall_in_window \| false_alarms_per_hour \| time_to_alert_median_s \| latency_median_s \| usd_per_hour \|
	\|---\|---\|---\|---\|---\|---\|---\|---\|---\|---\|---\|---\|---\|
	\| phone-exp4-finetuned-fridge_left_open \| 4 \| fridge_left_open \| finetuned \| 0.1322 \| 0.9328 \| 0.9633 \| 0.9478 \| 0.6667 \| 7.563 \| 0.0 \| 1.5845 \| 3.95 \|
	\| phone-exp4-base-fridge_left_open \| 4 \| fridge_left_open \| base \| 0.1322 \| 0.8614 \| 0.9746 \| 0.9145 \| 0.6667 \| 7.563 \| 0.0 \| 2.341 \| 3.95 \|
	\| phone-exp4-spark-fridge_left_open \| 4 \| fridge_left_open \| spark \| 0.1322 \| 0.8723 \| 0.9755 \| 0.921 \| 0.6667 \| 7.563 \| 0.0 \| 11.6665 \| 16.5276 \|
	\| phone-exp4-finetuned-burner_left_on \| 4 \| burner_left_on \| finetuned \| 0.0594 \| 1.0 \| 0.4667 \| 0.6364 \| 0.0 \| 0.0 \| 13.0 \| 1.689 \| 4.1045 \|
	\| phone-exp4-base-burner_left_on \| 4 \| burner_left_on \| base \| 0.0594 \| 1.0 \| 0.65 \| 0.7879 \| 0.5 \| 0.0 \| 0.0 \| 3.2215 \| 4.2589 \|
	\| phone-exp4-spark-burner_left_on \| 4 \| burner_left_on \| spark \| 0.0594 \| 0.9829 \| 0.9583 \| 0.9705 \| 0.5 \| 16.8224 \| 9.5 \| 11.9315 \| 16.5276 \|

	</details>

	The full tables, the EPIC-KITCHENS-100 results and how it was measured are on GitHub:
	[results](https://github.com/amaan784/trained-eye/blob/main/eval/results.md), [EPIC-KITCHENS](https://github.com/amaan784/trained-eye/blob/main/eval/epic/results.md),
	[method](https://github.com/amaan784/trained-eye/blob/main/docs/evaluation.md).

	## Training

	Meta's cookbook QLoRA recipe for Muse Glimmer (meta-oss-cookbook at commit b30ca2c,
	muse-glimmer/post-training/sft/axolotl) with Axolotl 0.19.0, unchanged except the paths, the
	W&B names and the gradient accumulation that keeps its global batch of 32 on one GPU.
	The training data was made through the Meta Model API:
	compiler `muse-wonder-1.0`, sam `sam-3.1`, image `muse-image-1.0`, labeler `muse-spark-1.3`.

	\| Setting \| Value \|
	\|---\|---\|
	\| `adapter` \| `qlora` \|
	\| `load_in_4bit` \| `true` \|
	\| `lora_r` \| `32` \|
	\| `lora_alpha` \| `16` \|
	\| `lora_dropout` \| `0.05` \|
	\| `lora_target_modules` \| `model\.language_model\.layers\.\d+\.(self_attn\.(q\\|k\\|v\\|o)_proj\\|mlp\.(gate\\|up\\|down)_proj)` \|
	\| `sequence_len` \| `4096` \|
	\| `micro_batch_size` \| `1` \|
	\| `num_epochs` \| `1` \|
	\| `optimizer` \| `adamw_bnb_8bit` \|
	\| `lr_scheduler` \| `cosine` \|
	\| `learning_rate` \| `0.0002` \|
	\| `warmup_ratio` \| `0.1` \|
	\| `weight_decay` \| `0.0` \|
	\| `bf16` \| `true` \|
	\| `attn_implementation` \| `flash_attention_2` \|
	\| `seed` \| `42` \|

	## Training data

	Bundle `mixed`: 603 training and 65 validation examples (train.jsonl SHA-256 `c056bbd61a66ece9bc9b4ec2af87f8b587a6cf952bf0d020fef29f4624473e49`). Real: keyframes of the user's own teaching clips, labelled by the user or else by Muse Spark. Generated: images Muse Image made by editing those same keyframes (the condition, near-misses, other light and angles), labelled by Muse Spark and kept only where Spark agreed with the label it was made for.

	\| Skill \| Split \| Real yes \| Real no \| Generated yes \| Generated no \|
	\|---\|---\|---\|---\|---\|---\|
	\| burner_left_on \| train \| 33 \| 0 \| 125 \| 135 \|
	\| burner_left_on \| val \| 7 \| 0 \| 12 \| 15 \|
	\| fridge_left_open \| train \| 45 \| 0 \| 134 \| 131 \|
	\| fridge_left_open \| val \| 1 \| 0 \| 16 \| 14 \|

	The generated images are not included. `manifests/` lists every edit prompt with the SHA-256 of its keyframe and of its image. For burner_left_on, fridge_left_open, `te regenerate` needs the original keyframes, which are not published.

	## Uses

	Intended: yes/no watch skills on first-person frames from smart glasses or a phone, each
	taught by its user from their own footage: the state of an object, a display, a sign or a
	vehicle (has the fridge been left open? is the tap still running?), with an alert threshold
	calibrated on the user's own validation clips and a hold before each alert. An alert is a
	reminder: it never replaces a smoke alarm or turning a burner off.

	Out of scope: people (identifying, recognising or tracking anyone), bodies, health,
	medication and other sensitive traits, covert monitoring, sensitive places, sports analysis
	or coaching, and anything where a missed or false alert could put someone at risk.

	## Limitations

	- Each skill comes from one user's scenes and camera; others are untested beyond the
	evaluation above. Most training images are Muse Image edits with Muse Spark's labels, which can be wrong.
	- It answers per frame: alerts need the watcher's smoothing, each skill's hold, a
	calibrated threshold and a cooldown. The vision tower stayed frozen, so small, distant or glary targets stay hard.
	- Scoring reads the server's top logprobs for the yes or no token, so the server must return them.
	- In plain words: overall it is about as accurate as base Glimmer and faster; it is the best
	of the three on the fridge and the weakest on the burner, so do not read it as an accuracy
	gain from fine-tuning on every skill.
	- It watches only what the glasses can see: look away and it cannot tell whether the
	condition is still true.
	- If Muse Spark's check fails outright (an error, not a no), the alert is still spoken and
	logged as unconfirmed, on purpose, so a cloud outage never silences an alert; a no from
	Spark holds it back.
	- The test labels were drafted by an AI assistant, too few events lasted past the hold to
	judge alerts well, and Experiments 2 and 3 did not run.
	- Validation during training was split by example, not by source frame, so related images
	can sit on both sides; the separate test footage is not affected.

	## Privacy and provenance

	- Faces are blurred in everything published with this model. Watch mode does not store
	frames unless the user saves a clip, and never hides the glasses' capture LED.
	- Muse Image outputs keep their watermark and provenance metadata, byte for byte, and are
	never published.

	## License

	`apache-2.0` as the base model's (meta-models/Muse-Glimmer-30B at revision `a4e59da52a7bc87ae7251dd5545c0dd437c44b68`; its license text is in `LICENSE`). These weights are a modification of that model: QLoRA fine-tuning on the data above (for a merged build, that adapter merged into the base). Any use is also subject to Meta's [Muse Glimmer Usage Policy](https://huggingface.co/meta-models/Muse-Glimmer-30B/blob/main/USAGE_POLICY.md).

	## Citation

	```bibtex
	@software{sheikh2026trainedeye,
	author = {Sheikh, Amaan},
	title = {Trained Eye: Teach Your Glasses to Notice Anything},
	year = {2026},
	month = nov,
	url = {https://github.com/amaan784/trained-eye}
	}
	```