Spaces:
Running
Running
| # plan.md — Reachy Mini "Wave Hello" app | |
| ## What you want | |
| A Reachy Mini app that: | |
| 1. Watches the robot's camera for faces and keeps the head pointed at the nearest/largest face. | |
| 2. Detects when a person waves at the robot. | |
| 3. Responds to a wave by wagging its antennae and making happy noises. | |
| Robot: **Wireless** (onboard CM4). Reaction source: **prebuilt emotions library** (`pollen-robotics/reachy-mini-emotions-library`). | |
| ## App flavor: JS / Hugging Face Space | |
| Chosen over Python because the Wireless CM4 is CPU- and memory-constrained. In the JS flavor all vision runs in the browser on your laptop or phone, the robot only streams video and receives motion commands. This matches the pattern used by the official `hand_tracker_v2` and `reachy_mini_telepresence` reference apps. | |
| Cost of this choice: WebRTC adds roughly 100-300 ms of latency between what the camera sees and the head command. Face tracking will feel slightly laggy compared to an on-robot loop. This is acceptable for social tracking; it would not be for fast target following. Graduating to Python later is easy if you decide you want tighter tracking. | |
| ## Technical approach | |
| **Scaffold** | |
| - Clone `pollen-robotics/reachy_mini_telepresence` (React + Vite, already handles the video stream) and trim it. Alternative: `reachy_mini_minimal_conversation` (vanilla TS) if you prefer no React. I lean telepresence because the camera plumbing is already correct. | |
| - Pin `@pollen-robotics/reachy-mini-sdk@1.8.0`. | |
| - Standard host shell: `mountHost()` for OAuth, robot picker, top bar; `connectToHost()` in the embed. | |
| - `enableMicrophone: false` (no user-to-robot audio needed). | |
| - Mobile-first layout, per the guide. | |
| **Vision (in browser)** | |
| - `@mediapipe/tasks-vision` from npm: Face Detector (short-range, lightweight) and Hand Landmarker (1 hand). Both run in the browser via WebGL/WASM. Frames come from the `<video>` element attached with `reachy.attachVideo()`. Never `getUserMedia()`; that would grab your own laptop camera. | |
| - Run detection at ~15 fps in a `requestVideoFrameCallback` loop; skip frames if the previous inference hasn't finished. | |
| **Face tracking loop** | |
| - Pick the largest detected face. Compute its center offset from frame center, normalized to [-1, 1]. | |
| - Proportional control on head yaw and pitch, with a small deadband (~4% of frame) and exponential smoothing so the head doesn't twitch. | |
| - Command via `reachy.setTarget()` at 10-20 Hz using a locally kept "last commanded pose" as the baseline (the guide warns against using telemetry as the baseline because it lags one RTT). | |
| - Clamp: pitch/roll within ±40°, keep yaw well inside limits. Body yaw stays fixed at 0 in v1; if the face drifts past ~35° head yaw, optionally rotate the body to follow (v2). | |
| - If no face is seen for ~3 s, ease back to a neutral pose. | |
| **Wave detection** | |
| - From Hand Landmarker: require an open hand (fingers extended, palm roughly facing camera) with the wrist above the hand's midpoint. | |
| - Track wrist x over a 1.2 s sliding window. A wave is: at least 2 direction reversals with horizontal travel above a threshold proportional to hand size. | |
| - Debounce: after a wave triggers, ignore new waves for ~4 s so one wave produces one reaction. | |
| **Happy reaction** | |
| - At startup, fetch a small set of "happy" moves from the emotions dataset (each is `<name>.json` + `<name>.ogg` at the dataset root) and cache them in memory. | |
| - On wave: pause the tracking loop, call `reachy.playMove(motionJson, { audioBlob })`, await `finished`, then resume tracking. `playMove` runs daemon-side with a single clock, so motion and sound stay in sync. | |
| - The recorded emotions already include antenna motion. If the library's "happy" clips are not antenna-heavy enough, I'll layer a short custom antenna wag via `setAntennasDeg` before or after the clip. | |
| **Cleanup** | |
| - `handle.onLeave()` → `safelyReturnToPose(reachy)`, cancel any in-flight `playMove`, stop the vision loop, close MediaPipe. | |
| **UI (minimal)** | |
| - Robot camera feed with face box and hand landmarks overlaid. | |
| - Status line: "Looking for faces" / "Tracking" / "Wave detected!" | |
| - Toggles: tracking on/off, reaction sensitivity, mute. | |
| ## Files I will produce | |
| Per the guide's 3-file contract: `index.html`, `src/dispatch.ts`, `src/embed.tsx`, plus `src/vision/` helpers, `public/icon.svg`, `README.md` frontmatter with `sdk: static`, `hf_oauth: true`, and the `reachy_mini_js_app` tag. | |
| ## Questions (answer inline) | |
| 1. Which emotion clips should play on a wave? Options: (a) just `happy`, (b) a random pick from a short list I curate (happy, amazed, excited-type clips) so it doesn't feel identical every time. | |
| Answer: random pick from a curated list (welcoming1, welcoming2, enthusiastic2, cheerful1, success2, loving1, helpful1) | |
| 2. Should the robot also react when it first sees a face, before any wave (e.g. a small "noticed you" head tilt)? Or stay quiet until waved at? | |
| Answer: yes, a short nod and a chirp when a face first appears | |
| 3. Body rotation: keep the body fixed and only move the head (simpler, v1), or let the body turn to follow a person walking around it? | |
| Answer: body turns to follow | |
| 4. Your Hugging Face username and the Space name you want (e.g. `yourname/reachy_wave_hello`). | |
| Answer: RickDakan/howdy_friend | |
| 5. Do you have Node.js + npm on your machine? If not, I can switch to the bare-HTML + CDN path, but MediaPipe is easier via npm. | |
| Answer: unknown, so use bare HTML + CDN path (no Node needed) | |
| ## Not doing in v1 | |
| - Multi-person tracking or identity recognition. | |
| - Custom recorded sounds. | |
| - Python fallback. | |