bhanureddychada/thingtrail-35b-a3b

ThingTrail: reference models

Two LoRAs on Qwen3.6-35B-A3B, released as reference baselines for the ThingTrail long-gap benchmark. Neither is a working object tracker. What we learned training them is in FINDINGS.md.

Model	Setting	Use it when
rl_vocab_onpolicy2	Closed vocabulary: the video's object list is in the prompt	You know which objects exist (a known inventory)
rl_is	Open vocabulary: the model names objects itself	You don't have an object list

Both watch first-person kitchen video 16 s at a time (16 frames, 1 fps, 448 px), read their current memory and write changes as JSON ops:

{"ops": [{"op": "MOVE", "object": "chopping board", "place": "sink.001", "t": 957.2},
         {"op": "SET_STATE", "object": "tap", "state": "on", "t": 959.8}]}

Ops are ADD, MOVE, SET_STATE and RETIRE; places are the kitchen's fixture ids (counter.003, held_by:wearer).

Results

Full tables with 95% confidence intervals: results/RESULTS.md.

	Untrained	SFT	rl_is	rl_vocab_onpolicy2 (list given)
Per clip, probe exact (500 clips, correct memory given)	17.5%	34.9%	42.3%	43.4%
Long gap, needs tracking (642 questions, own memory)	10.1%	14.6%	12.0%	26.6%
Long gap, answer is the busiest fixture (211)	2.8%	4.7%	26.5%	35.1%
Long gap, all (853)	8.3%	12.2%	15.6%	28.7%
HD-EPIC object motion, 129 test questions, box-grounded	20.2%	26.4%	31.8%	30.2%

• rl_vocab_onpolicy2 nearly doubles tracking over SFT without the list, and is the only model that beats always guessing the busiest fixture pooled (24.7%). Its accuracy doesn't fall as the time since the object moved grows (26.8% / 25.2% / 28.9% at 1–5 / 5–15 / 15+ min). It needs the object list: that is an easier setting than the open benchmark.
• rl_is is clearly better than SFT per clip (+7.3 points, CI +3.9 to +10.6) but not over long videos where tracking is needed (−2.6 points, CI −5.9 to +0.2). Its higher pooled score comes from learning that objects end up on the main counter.
• A long-context model re-watching raw frames (Qwen3.8-27B) gets 25.9% on tracking (200-question sample): better than rl_is, tied with rl_vocab_onpolicy2, at about 60× the tokens per question.
• Always guessing the kitchen's busiest fixture scores 24.7% pooled. Report the tracking split.

Known failure modes

• Open vocabulary: names objects differently from the annotators and confuses duplicates (bowl / bowl2); in about half of rl_is's "not in memory" misses it saw the move but used another name.
• Picks the neighbouring fixture of the right type (counter.005 for counter.004): 19% of questions for rl_is.
• Writes ops in clips where nothing changed: 31% of quiet clips for rl_is, 29% for rl_vocab_onpolicy2 (SFT: 15%).
• Open/closed and on/off states are less reliable than locations (and so are HD-EPIC's own state labels, which sometimes miss the close).
• Tested on two kitchens (25 videos); kitchens only, one camera.

Training

• Data: HD-EPIC object-movement tracks and narrations, as 16 s clips with the memory before and the gold ops. Train: 7,401 clips, 131 videos, kitchens P01–P07. Test: 1,979 clips, 25 videos, P08–P09.
• SFT: LoRA rank 32, AdamW, lr 5e-4, batch 32, 1 epoch (218 steps).
• rl_is, RL from the SFT state: importance-sampling policy gradient, lr 2e-5, 16 clips × 8 samples, 100 steps. Reward 0.1 valid JSON + 0.5 op F1 + 0.4 location-probe accuracy, partial credit for the right fixture type; half the clips start from a memory with objects removed.
• rl_vocab_onpolicy2: SFT with the object list in the prompt, then two rounds of 100 RL steps (same loss and reward). Before each round the latest model is run over all training videos and its own memory before every clip is kept; half of each RL batch starts from that memory instead of the gold one (scripts/build_onpolicy.py, train_rl.py --onpolicy-clips). A third round made long-video tracking worse (−5.0 points) and is not released.

Use

On Tinker:

• rl_is (public checkpoint): tinker://309b6481-4f96-59dc-b563-31fae305aeb5:train:0/sampler_weights/rl_is-s100
• rl_vocab_onpolicy2 (public checkpoint): tinker://ef06dfba-7ff5-52ef-9be0-a5ac8e15292f:train:0/sampler_weights/rl_vocab_onpolicy2-s100

On Hugging Face: bhanureddychada/thingtrail-35b-a3b holds both as PEFT LoRA adapters (rl_is/, rl_vocab_onpolicy2/) for Qwen/Qwen3.6-35B-A3B, e.g. vllm serve Qwen/Qwen3.6-35B-A3B --enable-lora --lora-modules thingtrail=<path>/rl_vocab_onpolicy2. LoRA on the MoE expert layers is experimental in vLLM and not yet supported in SGLang; the Tinker checkpoints are the reference. The benchmark is bhanureddychada/thingtrail-bench.

python scripts/eval_rollout.py --mode model --model-path <tinker path> --name my_run   # long gap
python scripts/eval_clips.py --model-path <tinker path> --n 500 --name my_run_clips    # per clip
python scripts/make_results.py

Add --vocab to both evals for rl_vocab_onpolicy2. python scripts/export_release.py --adapter builds a PEFT adapter for vLLM or SGLang on top of Qwen/Qwen3.6-35B-A3B. python scripts/export_memory_db.py turns a model's memories into a Postgres timeline you can query in SQL (demo/ontology).

Licence

Weights and derived labels: CC BY-NC 4.0, following HD-EPIC (non-commercial research only). Code: Apache-2.0. Please cite HD-EPIC if you use this model.