ArashNasr/ai2thor-kitchen-perception-floorplan17

AI2-THOR kitchen perception models: FloorPlan17

Status: public release, part of MISE (project page; an English paper is in preparation). Part of an MSc thesis (University of Tehran): an LLM + RAG embodied agent that plans L1 atomic / L2 sequential / L3 abstract kitchen commands in AI2-THOR, with an honest perception stack (no simulator ground truth at test time).

FloorPlan17 is a held-out replication kitchen. Its detectors were initialized from the frozen FloorPlan1 models (A38) and trained on its own capture; the numbers below are its registered TEST evaluation (2026-10-21; each step run once, sealed holdout opened once).

Models

File	Role
FloorPlan17.yolo26s.pt	primary detector, YOLO26s, 49 type-only classes, every frame
FloorPlan17.amb26m.pt	look-alike specialist, YOLO26m (Fork, Spoon, ButterKnife, Knife, Spatula, SaltShaker, PepperShaker, Plate, Pot, Pan); replaces the primary's predictions for those classes only while the agent holds an object up (gated)
FloorPlan17.yolo26s.control.pt	control: YOLO26s fine-tuned on primary + specialist data (the comparison the specialist is tested against)
FloorPlan17.state_*.pt	7 state classifiers (broken, cooked, dirty, filled, on, open, sliced), YOLO11s-cls

SHA256SUMS lists the exact deployed bytes (they match the kitchen's frozen record).

Results

Measure	Value
primary detector (YOLO26s), TEST / CLOSE-TEST / LIVE coverage mAP50	0.910 / 0.900 / 0.983
primary detector, TEST / CLOSE-TEST / LIVE coverage mAP50-95	0.823 / 0.823 / 0.844
LIVE realistic set mAP50 / mAP50-95 (127 poses, only 34 ground-truth boxes)	0.868 / 0.619
sealed holdout (opened once): held-target naming success, primary / gated (with specialist) / control	0.571 / 0.915 / 0.787
gated minus primary; gated minus control (paired episode bootstrap, 95 % CI)	+34.5 pp [+30.2, +38.3]; +12.8 pp [+9.0, +15.8]
note	realistic LIVE used 127 of the registered 150 poses (the Kitchen 1 pose rule's 20,000-draw cap)

Every number belongs to this kitchen's own adapted detectors and is never pooled across kitchens.

Licence and credits

AGPL-3.0 (fine-tuned from Ultralytics YOLO models). Training images rendered with AI2-THOR (Apache-2.0).