Spaces:
Running
Running
| <html><head><meta charset=utf-8><meta name=viewport content="width=device-width,initial-scale=1"><title>Thali — voice-controlled two-arm table setting</title> | |
| <style>body{margin:0;background:#0b0e14;color:#e5e7eb;font-family:Inter,Segoe UI,Helvetica,Arial,sans-serif} .w{max-width:1100px;margin:0 auto;padding:40px 24px} h1{font-size:44px;margin:0 0 8px} h2{font-size:26px;margin:40px 0 12px;color:#7dd3fc} p{font-size:18px;line-height:1.5;color:#c7cbd3} a{color:#7dd3fc} video,img{width:100%;border-radius:12px} table{border-collapse:collapse;width:100%;font-size:15px} td{padding:8px 10px;border-bottom:1px solid #222;vertical-align:top} td:first-child{color:#7dd3fc;white-space:nowrap} .grid{display:grid;grid-template-columns:1fr 1fr;gap:20px} .card{background:#151a23;border:1px solid #232a36;border-radius:12px;padding:18px} .card b{font-size:34px;color:#7dd3fc;display:block} .tag{display:inline-block;background:#7dd3fc;color:#0b0e14;font-weight:700;padding:4px 12px;border-radius:999px;font-size:14px;margin-right:8px}</style></head><body><div class=w> | |
| <span class=tag>Intel Physical AI · Bimanual VLA</span><span class=tag>Speechmatics</span> | |
| <h1>Thali — two arms that set your table by voice</h1> | |
| <p>Say what you want and two SO-101 arms open the drawer, lay out cutlery and plate, hold the mug and pour — for people who can speak but can't reach. Speechmatics realtime speech · local Qwen2-VL planner on Intel OpenVINO · deterministic safety verifier with a hash-chained audit log · learned skills with a scripted-IK fallback. Everything except speech recognition runs on an Intel Core i7-13650HX laptop (no NPU).</p> | |
| <p id="live2"><a href="https://thali-live.streamlit.app" style="display:inline-block;background:#34d399;color:#0b0e14;font-weight:800;padding:12px 20px;border-radius:10px;text-decoration:none;font-size:18px">▶ Open Thali Live — always on (Streamlit)</a> <span style="color:#8b93a3;font-size:13px">simulator, verifier, audit chain, your microphone via Speechmatics; the rule planner (the OpenVINO planner and PatchCore run on the lab machine, first button)</span></p> | |
| <p id="live"><a href="https://lip-stops-algorithms-fleece.trycloudflare.com" style="display:inline-block;background:#7dd3fc;color:#0b0e14;font-weight:800;padding:12px 20px;border-radius:10px;text-decoration:none;font-size:18px">▶ Open Thali Live — drive the simulator yourself</a> <span style="color:#8b93a3;font-size:13px">served from the lab machine; if it does not answer, the machine is off — the video below shows the same loop</span></p> | |
| <h2>Demo (5:36) — the full closed loop, clearing the table, corrections, both arms at once</h2><video controls src="thali_demo.mp4" poster="architecture.png"></video> | |
| <h2>Architecture</h2><img src="architecture.png"> | |
| <h2>Results at a glance <span style='font-size:14px;color:#8b93a3'></span></h2><div class=grid> | |
| <div class=card><b>9/10</b>full 7-skill task on held-out seeds (via-table handoff + hold-while-pour); clear the table 9/10; follow-ups ("again", "a bit more", "other side") 20/20</div> | |
| <div class=card><b>20/20</b>injected unsafe plans blocked; audit chain verified</div> | |
| <div class=card><b>49.13 → 17.06 ms</b>ACT policy call, OpenVINO fp32 → NNCF INT8 on the CPU, behaviour preserved</div> | |
| <div class=card><b>44.6 tok/s</b>Qwen2-VL-2B INT4 planner on the Intel CPU, 100% verifier-approved plans</div> | |
| <div class=card><b>4/4</b>mid-task recovery: plate knocked 10 cm off its zone after its step passed; final-state check noticed and re-placed it</div> | |
| <div class=card><b>44%</b>fewer sim steps with both arms moving at once (drawer + mug), success unchanged</div> | |
| <div class=card><b>18/20 · 15/20 · 14/20</b>ACT policies alone at 50k training steps — hold, plate, fork (median 1–2 cm placement); 8/20 at 12k: training length was the bottleneck</div> | |
| <div class=card><b>AUROC 0.943</b>Anomalib PatchCore table-state check on OpenVINO (70.23 ms CPU); knocked plate flagged live 3/4</div> | |
| <div class=card><b>4/4</b>voice samples (clear, tired, Hindi, noisy room) with the skill sequence recovered; background speaker ignored</div> | |
| <div class=card><b>10.447 s</b>speech end → arms moving, end to end</div></div> | |
| <h2>Robustness — seeds × perturbation axis (held-out split)</h2><img src="heatmap_expert.png" style="max-width:760px"> | |
| <h2>Evidence index — every number and the script that produced it</h2><table><tr><td>Expert full task, test split (10 seeds)</td><td>9/10</td><td>`eval/run_seeds.py --policy expert --split test`</td><td>results/seeds_expert_expert_test.json</td></tr><tr><td>Expert full task, train split</td><td>9/10</td><td>`eval/run_seeds.py --policy expert --split train`</td><td>results/seeds_expert_expert_train.json</td></tr><tr><td>ACT policy-only, test</td><td>0/10</td><td>`eval/run_seeds.py --policy act --mode policy_only`</td><td>results/seeds_act_policy_only_test.json</td></tr><tr><td>ACT + retry, test</td><td>0/10</td><td>`eval/run_seeds.py --policy act --mode policy_retry`</td><td>results/seeds_act_policy_retry_test.json</td></tr><tr><td>ACT + retry + expert fallback, test</td><td>0/10</td><td>`eval/run_seeds.py --policy act --mode policy_fallback`</td><td>results/seeds_act_policy_fallback_test.json</td></tr><tr><td>SmolVLA rows</td><td>pending SmolVLA run (docs/KAGGLE_TODO.md)</td><td>`eval/run_seeds.py --policy smolvla`</td><td>results/seeds.json</td></tr><tr><td>Heatmap seeds × axis, expert</td><td>hardest axis shape</td><td>`eval/heatmap.py --policy expert`</td><td>results/heatmap_expert.json / .png</td></tr><tr><td>Heatmap seeds × axis, ACT</td><td>pending</td><td>`eval/heatmap.py --policy act`</td><td>results/heatmap_act.json / .png</td></tr><tr><td>Per-skill ACT policy-only (60 ep)</td><td>open_drawer **20/20** · pick_place_fork **2/20** (median 29.93 cm from zone) · pick_place_plate **8/20** (median 15.85 cm from zone) · pick_place_mug **1/20** (median 24.45 cm from zone) · handoff_spoon **0/20** (median 39.89 cm from zone) · hold_mug **10/20** · pour **1/20**</td><td>`python -m eval.skill_eval --act-root outputs_60ep --tag act_60ep`</td><td>`results/skill_eval_act_60ep.json`</td></tr><tr><td>Per-skill ACT policy-only (1050 ep)</td><td>open_drawer **20/20** · pick_place_fork **6/20** (median 29.16 cm from zone) · pick_place_plate **8/20** (median 14.12 cm from zone) · pick_place_mug **1/20** (median 26.68 cm from zone) · handoff_spoon **0/20** (median 39.3 cm from zone) · hold_mug **10/20** · pour **1/20**</td><td>`python -m eval.skill_eval --tag act_1050ep`</td><td>`results/skill_eval_act_1050ep.json`</td></tr><tr><td>Per-skill ACT policy-only (50k steps)</td><td>pick_place_fork **16/20** (median 1.04 cm from zone) · pick_place_plate **15/20** (median 1.85 cm from zone)</td><td>`lerobot-train … --steps=50000 --output_dir=outputs_50k/act_<skill>`; `python -m eval.skill_eval --act-root outputs_50k --tag act_50k`</td><td>`results/skill_eval_act_*50k.json`</td></tr><tr><td>Per-skill SmolVLA policy-only (newest checkpoint)</td><td>step 5k: open_drawer **1/20** · pick_place_fork **0/20** (median 31.5 cm from zone) · pick_place_plate **0/20** (median 20.07 cm from zone) · pick_place_mug **0/20** (median 27.21 cm from zone) · handoff_spoon **0/20** (median 40.36 cm from zone) · hold_mug **3/20** · pour **0/20**</td><td>`python -m eval.skill_eval --kind smolvla --smolvla-path <Prashant-77/thali_smolvla> --tag smolvla_5k`</td><td>`results/skill_eval_smolvla_5k.json`</td></tr><tr><td>Table-state anomaly check</td><td>abs(frame − reset reference) crop, resnet18, nominal set from real expert runs (post-skill states): image AUROC **0.943** · 72/75 disturbances flagged with 13/57 false alarms (spill 13/15, tipped mug 15/15, knocked plate 15/15, fallen bottle 15/15, dropped cutlery 14/15) · at 10% false alarms 61/75 · IR p50 CPU 70.23 ms — **live, in the loop** (plate knocked mid-task, 10%-FPR threshold): flagged at the next check in 3/4 runs, 1/4 false alarms on clean steps, table nominal again after the redo — ablation full frame, wide_resnet50: AUROC 0.762, 38/60 false alarms — ablation table crop, resnet18: AUROC 0.78, 42/60 false alarms — ablation diff on the synthetic nominal set: AUROC 0.938, 5/60 false alarms</td><td>`python -m anomaly.make_data_real …; THALI_ANOMALY_VARIANT=diffreal .venv-anomalib/bin/python -m anomaly.train_patchcore --backbone resnet18 --coreset 0.05; … export_patchcore; python -m anomaly.check --score; python -m eval.recovery --anomaly`</td><td>`results/anomaly_diffreal.json`, `results/anomaly*.json`, `results/recovery_anomaly.json`</td></tr><tr><td>Follow-ups and corrections</td><td>**20/20** corrections resolved to the expected plan; 14/16 resolved plans verifier-approved (the rest are refused with a reason, e.g. the other arm cannot reach)</td><td>`python -m eval.followups`</td><td>`results/followups.json`</td></tr><tr><td>Clear the table</td><td>**9/10** held-out seeds fully cleared (fork stowed 90%, spoon stowed 90%, drawer closed 90%)</td><td>`python -m eval.clear_table --seeds 10 --split test`</td><td>`results/clear_table.json`</td></tr><tr><td>Target-volume pour</td><td>little (target 3): mean 3.0 spheres, within ±2 in **6/6**, reached 6/6 · normal (target 6): mean 6.3 spheres, within ±2 in **6/6**, reached 6/6 · full (target 12): mean 12.8 spheres, within ±2 in **5/6**, reached 6/6</td><td>`python -m eval.pour_amount`</td><td>`results/pour_amount.json`</td></tr><tr><td>Both arms at once vs sequential</td><td>drawer_and_mug: sequential 4/5 in 1484 sim steps → concurrent **5/5 in 687** (54% fewer) · plate_and_mug: sequential 5/5 in 1263 sim steps → concurrent **5/5 in 965** (24% fewer)</td><td>`python -m eval.concurrency`</td><td>`results/concurrency.json`</td></tr><tr><td>Mid-task perturbation recovery</td><td>4/4 recovered (4 detected)</td><td>`python -m eval.recovery`</td><td>`results/recovery.json`</td></tr><tr><td>Instruction swap</td><td>7/10</td><td>`eval/instruction_swap.py`</td><td>results/instruction_swap.json</td></tr><tr><td>Camera vs oracle</td><td>pixels 84%, vlm 38%</td><td>`eval/camera_vs_oracle.py`</td><td>results/camera_vs_oracle.json</td></tr><tr><td>Planner: VLM-accepted / approved / tok/s</td><td>4/8 · 100% · 44.6</td><td>`eval/planner_eval.py`</td><td>results/planner_eval.json</td></tr><tr><td>Verifier injection</td><td>20/20 caught, 6/6 sane passed</td><td>`verifier/inject_bad_plans.py`</td><td>results/verifier_injection.json + audit jsonl</td></tr><tr><td>Audit chain</td><td>chain intact</td><td>`make verify-log`</td><td>results/audit.jsonl</td></tr><tr><td>Voice test (4 samples)</td><td>mean WER 0.438, normalised 0.295, skills 100%</td><td>`eval/voice_test.py`</td><td>results/voice_test.json</td></tr><tr><td>Voice → arm latency</td><td>10.447 s (seed 3, noisy.wav)</td><td>`runtime/demo.py --voice`</td><td>results/demo_seed3.json</td></tr><tr><td>Barge-in</td><td>1 stop/resume, step completed: True</td><td>`runtime/demo.py --barge-in stop@3`</td><td>results/demo_bargein_stop.json</td></tr><tr><td>OpenVINO latency</td><td>fp32/CPU 49.13 ms, fp32/GPU 283.24 ms, fp16/CPU 49.51 ms, fp16/GPU 282.82 ms, int8/CPU 17.06 ms, int8/GPU 201.85 ms</td><td>`bench/run.py`</td><td>results/bench.json / bench.md</td></tr><tr><td>Precision preservation</td><td>torch 0, fp32 0, fp16 0, int8 0</td><td>`bench/preserve.py`</td><td>results/preserve.json</td></tr><tr><td>IR sizes / max Δ</td><td>open_drawer: fp32 132.7MB/fp16 67.1MB/int8 34.9MB, pick_place_fork: fp32 132.7MB/fp16 67.1MB/int8 34.9MB, pick_place_plate: fp32 132.7MB/fp16 67.1MB/int8 34.9MB, pick_place_mug: fp32 132.7MB/fp16 67.1MB/int8 34.9MB, handoff_spoon: fp32 132.7MB/fp16 67.1MB/int8 34.9MB, hold_mug: fp32 132.7MB/fp16 67.1MB/int8 34.9MB, pour: fp32 132.7MB/fp16 67.1MB/int8 34.9MB</td><td>`bench/export_ir.py`, `bench/quantize.py`</td><td>results/ir_export.json</td></tr><tr><td>Dataset</td><td>1050 episodes, 742837 frames</td><td>`expert/make_demos.py`</td><td>results/demos.json</td></tr><tr><td>Reach / handoff geometry</td><td>handoff [-0.010000000000000009, 0.0, 0.03] (via_table), 330 doubly-reachable cells</td><td>`souschef_env/reach.py`</td><td>results/reach_envelope.json</td></tr><tr><td>Jaw pads</td><td>0.23° tilt, 15.8 mm gap</td><td>`souschef_env/build_scene.py`</td><td>results/jaw_pads.json</td></tr></table> | |
| <h2>Links</h2><p><a href="https://github.com/Prashant-thakur77/THALI">GitHub repository</a> · <a href="Thali_slides.pdf">Slides (PDF)</a> · <a href="https://huggingface.co/datasets/Prashant-77/thali_all">Dataset (1050 episodes)</a> · <a href="https://github.com/Prashant-thakur77/THALI/blob/main/docs/WRITEUP.md">Write-up</a></p> | |
| <h2>Run it yourself</h2><p><code>git clone https://github.com/Prashant-thakur77/THALI && cd THALI && uv venv .venv --python 3.11 && uv pip install -r requirements.txt && uv pip install -e . && make test && make demo VOICE=voice/test_samples/noisy.wav SEED=3</code></p> | |
| </div></body></html> |