amaan784/trained-eye

Trained Eye: watch skills for Muse Glimmer 30B (LoRA adapter)

What this is. A QLoRA adapter (PEFT) for meta-models/Muse-Glimmer-30B (revision a4e59da52a7bc87ae7251dd5545c0dd437c44b68), from Trained Eye, which lets Meta glasses watch for something and say when it happens. Given one first-person camera frame and a condition in the prompt, it answers yes or no. It was trained on two skills at once (the training run is called product): burner_left_on ("Tell me when a burner has been left on"), fridge_left_open ("Tell me when the fridge door is left open"). It is not a model on its own: it needs the base model's weights.

Code | Playground | Training runs (W&B) | How it works

What is in this repository

File	What it is
adapter_model.safetensors	The LoRA weights: rank 32 on every attention and MLP projection of the language model's layers; the vision tower is unchanged. 767 MB.
adapter_config.json	PEFT's settings for those weights: the base model, the rank, alpha and the target modules.
config.json	The model configuration the training run saved with the adapter.
processor_config.json	The image processor's settings, saved with the adapter.
tokenizer.json	The tokenizer, saved with the adapter.
tokenizer_config.json	The tokenizer's settings, saved with the adapter.
chat_template.jinja	The chat template the prompts were built with.
manifests/<skill>.jsonl	One line per generated training image (300 for burner_left_on, 300 for fridge_left_open): Muse Image's edit prompt, the answer it was made for (yes, or no for a near-miss), and the SHA-256 of the frame it edited and of the image. The images themselves are not here.
LICENSE	The base model's license text (Apache-2.0).
release.json	Every file here with its size and SHA-256.
README.md	This card.

Not in this repository:

• The base model's weights: get meta-models/Muse-Glimmer-30B at revision a4e59da52a7bc87ae7251dd5545c0dd437c44b68 from Hugging Face, under its own terms.
• The training images. The generated images are never published (they keep Muse Image's provenance and stay with the author), and neither are the teaching frames they were edited from. manifests/ says how each generated image was made.
• The test clips and their labels, and the training code, which is on GitHub.

How to use it

Send each frame the way the training examples were built:

• System message: You watch first-person camera frames for one condition. Answer only yes or no.
• User message: the frame, padded (not cropped) to 512 x 512 pixels with black bars, then this text: Condition: <what to watch for>. Not: <a near-miss>; <another>. Is the condition true in this frame?

The conditions it was trained with, word for word:

Skill	Asked as	Condition	Not
burner_left_on	Tell me when a burner has been left on	a burner is visibly on with a blue gas flame or glowing red hot and no pot or pan on it	a burner is off, dark and unlit with no flame or glow; a burner is on but has a pot or pan sitting on it; an unlit neighbouring burner on the same cooktop; shiny foil burner liner or metal grate with no flame or red glow
fridge_left_open	Tell me when the fridge door is left open	the fridge door stands open with its brightly lit interior, food shelves, and door bins exposed	the fridge doors are fully closed, showing only the dark flat exterior with no interior light visible; the bottom freezer drawer is closed beneath the fridge compartment; a closed kitchen cabinet above the refrigerator; an open cabinet or pantry door that is not the refrigerator

Glimmer writes a short reasoning before it answers, so read the last yes or no token it generates, and take the chance of yes from that token's top logprobs (yes over yes plus no). An alert should wait until that chance has stayed over a threshold, calibrated on your own labelled clips, for the skill's hold (Trained Eye's watcher does this).

What ran on Nov 4: the adapter merged into the bf16 base with PEFT's merge_and_unload() on one 80 GB H100, then served with vLLM's vllm/vllm-openai:muse-glimmer image, each request with logprobs on, top_logprobs 10, up to 512 new tokens and chat_template_kwargs {"reasoning_strength": "low"}. Loading the adapter with vLLM's --enable-lora did not start for this model on that image. The merge, as the project's te export runs it:

import torch
from peft import PeftModel
from transformers import AutoModelForImageTextToText, AutoProcessor

base, revision = "meta-models/Muse-Glimmer-30B", "a4e59da52a7bc87ae7251dd5545c0dd437c44b68"
model = AutoModelForImageTextToText.from_pretrained(
    base, revision=revision, torch_dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(model, "amaan784/trained-eye").merge_and_unload()
model.save_pretrained("trained-eye-merged")
AutoProcessor.from_pretrained(base, revision=revision).save_pretrained("trained-eye-merged")

Results

In short: on other kitchens filmed first-person on phones, the fine-tuned model spotted the condition about as accurately as base Glimmer and less accurately than Muse Spark (frame F1 0.87, 0.88 and 0.94), answering in 1.64 s a frame against 2.53 s and 11.80 s, at $4.00 an hour against $16.53 for Spark.

Skill	Model	Frame precision	Frame recall	Frame F1	Frame accuracy	Seconds per frame	Dollars per hour
burner_left_on	this adapter	1.00	0.47	0.64	0.70	1.69	$4.10
burner_left_on	base Muse Glimmer 30B	1.00	0.65	0.79	0.80	3.22	$4.26
burner_left_on	Muse Spark (API)	0.98	0.96	0.97	0.97	11.93	$16.53
fridge_left_open	this adapter	0.93	0.96	0.95	0.95	1.58	$3.95
fridge_left_open	base Muse Glimmer 30B	0.86	0.97	0.91	0.91	2.34	$3.95
fridge_left_open	Muse Spark (API)	0.87	0.98	0.92	0.91	11.67	$16.53

Test footage: other kitchens filmed first-person on phones (handheld; four fridges and two stoves the model was not taught on; labels drafted by an AI assistant, see docs/evaluation.md). From te eval's results.json (SHA-256 6704865a78632b3e1d28aa5b50c1f56ca68537b9307949ead80969b32ecb1d8a); evaluations scored: 6, pending: 0. n/a means there was nothing to compute it from. Each skill alerts only once its condition has stayed true for its hold: event_recall_in_window counts an alert within the skill's window after the hold ends, and time_to_alert_median_s is counted from the end of the hold. Holds and windows: burner_left_on, a 10 s hold and a 5 s window; fridge_left_open, a 20 s hold and a 10 s window. Thresholds were calibrated for frame F1 (this footage has too few events past the hold to judge alerts).

Every scored evaluation

evaluation	experiment	skill	variant	hours	precision	recall	f1	event_recall_in_window	false_alarms_per_hour	time_to_alert_median_s	latency_median_s	usd_per_hour
phone-exp4-finetuned-fridge_left_open	4	fridge_left_open	finetuned	0.1322	0.9328	0.9633	0.9478	0.6667	7.563	0.0	1.5845	3.95
phone-exp4-base-fridge_left_open	4	fridge_left_open	base	0.1322	0.8614	0.9746	0.9145	0.6667	7.563	0.0	2.341	3.95
phone-exp4-spark-fridge_left_open	4	fridge_left_open	spark	0.1322	0.8723	0.9755	0.921	0.6667	7.563	0.0	11.6665	16.5276
phone-exp4-finetuned-burner_left_on	4	burner_left_on	finetuned	0.0594	1.0	0.4667	0.6364	0.0	0.0	13.0	1.689	4.1045
phone-exp4-base-burner_left_on	4	burner_left_on	base	0.0594	1.0	0.65	0.7879	0.5	0.0	0.0	3.2215	4.2589
phone-exp4-spark-burner_left_on	4	burner_left_on	spark	0.0594	0.9829	0.9583	0.9705	0.5	16.8224	9.5	11.9315	16.5276

The full tables, the EPIC-KITCHENS-100 results and how it was measured are on GitHub: results, EPIC-KITCHENS, method.

Training

Meta's cookbook QLoRA recipe for Muse Glimmer (meta-oss-cookbook at commit b30ca2c, muse-glimmer/post-training/sft/axolotl) with Axolotl 0.19.0, unchanged except the paths, the W&B names and the gradient accumulation that keeps its global batch of 32 on one GPU. The training data was made through the Meta Model API: compiler muse-wonder-1.0, sam sam-3.1, image muse-image-1.0, labeler muse-spark-1.3.

Setting	Value
adapter	qlora
load_in_4bit	true
lora_r	32
lora_alpha	16
lora_dropout	0.05
lora_target_modules	model\.language_model\.layers\.\d+\.(self_attn\.(q|k|v|o)_proj|mlp\.(gate|up|down)_proj)
sequence_len	4096
micro_batch_size	1
num_epochs	1
optimizer	adamw_bnb_8bit
lr_scheduler	cosine
learning_rate	0.0002
warmup_ratio	0.1
weight_decay	0.0
bf16	true
attn_implementation	flash_attention_2
seed	42

Training data

Bundle mixed: 603 training and 65 validation examples (train.jsonl SHA-256 c056bbd61a66ece9bc9b4ec2af87f8b587a6cf952bf0d020fef29f4624473e49). Real: keyframes of the user's own teaching clips, labelled by the user or else by Muse Spark. Generated: images Muse Image made by editing those same keyframes (the condition, near-misses, other light and angles), labelled by Muse Spark and kept only where Spark agreed with the label it was made for.

Skill	Split	Real yes	Real no	Generated yes	Generated no
burner_left_on	train	33	0	125	135
burner_left_on	val	7	0	12	15
fridge_left_open	train	45	0	134	131
fridge_left_open	val	1	0	16	14

The generated images are not included. manifests/ lists every edit prompt with the SHA-256 of its keyframe and of its image. For burner_left_on, fridge_left_open, te regenerate needs the original keyframes, which are not published.

Uses

Intended: yes/no watch skills on first-person frames from smart glasses or a phone, each taught by its user from their own footage: the state of an object, a display, a sign or a vehicle (has the fridge been left open? is the tap still running?), with an alert threshold calibrated on the user's own validation clips and a hold before each alert. An alert is a reminder: it never replaces a smoke alarm or turning a burner off.

Out of scope: people (identifying, recognising or tracking anyone), bodies, health, medication and other sensitive traits, covert monitoring, sensitive places, sports analysis or coaching, and anything where a missed or false alert could put someone at risk.

Limitations

• Each skill comes from one user's scenes and camera; others are untested beyond the evaluation above. Most training images are Muse Image edits with Muse Spark's labels, which can be wrong.
• It answers per frame: alerts need the watcher's smoothing, each skill's hold, a calibrated threshold and a cooldown. The vision tower stayed frozen, so small, distant or glary targets stay hard.
• Scoring reads the server's top logprobs for the yes or no token, so the server must return them.

Privacy and provenance

• Faces are blurred in everything published with this model. Watch mode does not store frames unless the user saves a clip, and never hides the glasses' capture LED.
• Muse Image outputs keep their watermark and provenance metadata, byte for byte, and are never published.

License

apache-2.0 as the base model's (meta-models/Muse-Glimmer-30B at revision a4e59da52a7bc87ae7251dd5545c0dd437c44b68; its license text is in LICENSE). These weights are a modification of that model: QLoRA fine-tuning on the data above (for a merged build, that adapter merged into the base). Any use is also subject to Meta's Muse Glimmer Usage Policy.

Citation

@software{sheikh2026trainedeye,
  author = {Sheikh, Amaan},
  title  = {Trained Eye: Teach Your Glasses to Notice Anything},
  year   = {2026},
  month  = nov,
  url    = {https://github.com/amaan784/trained-eye}
}