ArashNasr/ai2thor-kitchen-perception-floorplan15

AI2-THOR kitchen perception models: FloorPlan15

Status: public release, part of MISE (project page; an English paper is in preparation). Part of an MSc thesis (University of Tehran): an LLM + RAG embodied agent that plans L1 atomic / L2 sequential / L3 abstract kitchen commands in AI2-THOR, with an honest perception stack (no simulator ground truth at test time).

FloorPlan15 is a held-out replication kitchen. Its detectors were initialized from the frozen FloorPlan1 models (A38) and trained on its own capture; the numbers below are its registered TEST evaluation (2026-10-21; each step run once, sealed holdout opened once).

Models

File	Role
FloorPlan15.yolo26s.pt	primary detector, YOLO26s, 49 type-only classes, every frame
FloorPlan15.amb26m.pt	look-alike specialist, YOLO26m (Fork, Spoon, ButterKnife, Knife, Spatula, SaltShaker, PepperShaker, Plate, Pot, Pan); replaces the primary's predictions for those classes only while the agent holds an object up (gated)
FloorPlan15.yolo26s.control.pt	control: YOLO26s fine-tuned on primary + specialist data (the comparison the specialist is tested against)
FloorPlan15.state_*.pt	7 state classifiers (broken, cooked, dirty, filled, on, open, sliced), YOLO11s-cls
FloorPlan15.yolo26s_coco.pt	registered sensitivity only: the same primary trained from the COCO checkpoint (never deployed)

SHA256SUMS lists the exact deployed bytes (they match the kitchen's frozen record).

Results

Measure	Value
primary detector (YOLO26s), TEST / CLOSE-TEST / LIVE coverage mAP50	0.934 / 0.943 / 0.984
primary detector, TEST / CLOSE-TEST / LIVE coverage mAP50-95	0.854 / 0.885 / 0.861
LIVE realistic set mAP50 / mAP50-95 (150 poses, only 123 ground-truth boxes)	0.627 / 0.551
sealed holdout (opened once): held-target naming success, primary / gated (with specialist) / control	0.662 / 0.948 / 0.849
gated minus primary; gated minus control (paired episode bootstrap, 95 % CI)	+28.6 pp [+23.9, +32.8]; +9.9 pp [+6.0, +14.6]
COCO-start sensitivity: TEST mAP50-95 COCO minus Kitchen-1 start	-0.5 pp [-0.8, -0.1] (the deployed model is the Kitchen-1 start)

Every number belongs to this kitchen's own adapted detectors and is never pooled across kitchens.

Licence and credits

AGPL-3.0 (fine-tuned from Ultralytics YOLO models). Training images rendered with AI2-THOR (Apache-2.0).