Spaces:
Running
Running
Commit History
-
feat: sync robot preferences to HF, thinking movement + wait/announce on plan
Juan Jimenez Carrero +1: Claude Opus 4.8Settings sync (user noticed prefs didn't sync): the HF dataset only carried AppData. Now a NON-SECRET settings subset (language, robotName, voice, activationMode, allowInterrupt, inventorySort) rides along in a separate cookaiware-settings.json. Secrets (openaiKey, hfToken) and device-local state (lastSyncedAt, hfDatasetSync) are deliberately never uploaded. pushToCloud sends them; "Load from cloud" and the auto-sync "keep cloud" branch pull + validate them (applySyncableSettings keeps the current value for any invalid field). Silent-thinking fix: the menu designer takes ~10-20 s during which the robot sat still and mute. Now: - VoiceController.startThinking/stopThinking pulses a VARIED thinking body language (mostly "thinking", an occasional glance) every 2.6 s; app wraps both designAndCommitPlan and designMeal awaits with it (finally-stopped, also cleared on session stop). Only active during a live voice session. - Prompt now tells Reachy to ask the user to wait BEFORE calling…b5352be
-
feat: pick robot voice, lock/undo by voice, protect locked meals; fix key text
Juan Jimenez Carrero +1: Claude Opus 4.8Voice picker (like the official conversation app): new "Robot voice" setting in Talk to Reachy lists the OpenAI gpt-realtime voices (marin & cedar — the newest, most natural — plus alloy/ash/ballad/coral/echo/sage/shimmer/verse). Threaded settings.voice -> controller -> RealtimeSession.output.voice (was hardcoded shimmer). Default is now marin; applies on the next Talk-to-Reachy connect. Lock + undo by voice (parity with the app buttons): - set_meal_lock tool -> app.setMealLock(day, meal, locked): keep/free one meal. - undo_plan_change tool -> app.undoPlan(): restore the previous menu. - Reused resolveDay so 'today'/'tomorrow'/weekday all work. Locked meals are now protected: set_day_meal (and the app's designMeal) refuse a locked slot, returning a 'locked' error; the prompt tells Reachy to ask the user and only unlock+replace on confirmation. Previously voice could silently overwrite a kept meal. Accuracy fix (user-flagged): the OpenAI key hint wrongly said vision uses gpt-4o-mini …f080bc3
-
feat: expressive body — play_emotion tool + idle policy (match official app feel)
Juan Jimenez Carrero +1: Claude Opus 4.8Studied pollen-robotics/reachy_mini_conversation_app to close the naturalness gap. Its liveliness comes from the LLM driving a play_emotion tool mid-speech (intent → recorded move) plus a between-turns idle policy — movement tied to what it's SAYING, not just audio loudness. Ported both to our web/JS stack. - New play_emotion tool: the model calls it constantly (happy, excited, thinking, confused, sad, attentive, loving, angry, surprised, nod). Free-text intents resolve through an alias table (laughing→happy, curious→attentive…). - Emotions are procedural accents ADDED onto the existing 20 Hz motion loop — one writer to the head, so no motor jitter (the reason we own motion and disable the daemon wobble). Each has a bell envelope so it blends in/out. - Idle policy: after ~7 s of listening silence the robot throws in a small attentive/thinking glance so it never looks frozen. - Prompt: explicit BODY rule ('move while you talk, never stand still'), tighter reply length (~25 …011329d
-
feat: voice proactivity/interrupt settings + intelligent matrix-based menus
Juan Jimenez Carrero +1: Claude Opus 4.8Voice settings (apply on the next session): - Activation mode: respond to any speech, or ONLY when addressed by the robot's name (prompt-enforced wake rule; stays fully silent otherwise). - Allow interruptions: maps to server VAD interrupt_response, keeps the mic pump live while the robot speaks, cuts local TTS on speech_started, and skips input_buffer clears that would clip a barge-in. Meal planning overhaul: - Family size is now REQUIRED to plan (gate + CTA to Settings; voice tools return an instructive error that makes the model ask and call set_family). - 'Who eats what' 7×3 matrix (days × breakfast/lunch/dinner): each cell holds adults+children for that meal; tap → steppers, 'not planned' clears; stored as family_profile.plan_matrix (legacy weekday/weekend schedule auto-derives the default matrix, preserving the original app's behavior). - Intelligent generation through gpt-realtime (the key's only scope): * Button: short headless TEXT realtime session (planLLM.ts…cd6a098
-
fix: garbled voice on tool calls, sync safety, edit-not-duplicate, calmer persona
Juan Jimenez Carrero +1: Claude Opus 4.8- Garbled/'buzzing' speech before plan creation: we requested a new response while the previous one was still streaming, so two responses' audio chunks interleaved on one playback timeline. Responses are now gated: track response.created/done and defer response.create until the active one ends. - Auto-sync: enabling the toggle now downloads the cloud copy first and, if it differs, asks which side to keep (cloud vs this device) before the first auto-save — it can no longer silently overwrite. Switch styling fixed (the global input rules were deforming the checkbox). - New update_inventory_item tool: edits an existing item in place (location, expiration, quantity, rename); prompt now forbids add_inventory for changes, which was creating duplicates. - Sort by Location added (groups by location, then name; no-location last). - Robot name setting (default Reachy) fed into the voice persona; persona rewritten to be REACTIVE: do only what was asked, one-two short sentences, …ff39cb2
-
feat: real vision — take_picture tool + photo in the conversation
Juan Jimenez Carrero +1: Claude Fable 5Stop the model hallucinating sight and give it actual eyes: - Hard prompt rule: no continuous vision; the ONLY way to see is the new take_picture tool; never describe surroundings except from its result. - take_picture(question): grabs ONE still frame from the robot's WebRTC video track (off-screen sink, nothing streamed or recorded), shows the photo as a polaroid-style bubble in the conversation panel, then sends it to gpt-4o-mini (same user key) which returns a description + candidate food items {name, quantity?, unit?} the model can offer to add_inventory — the Python app's camera.py flow, ported (two-step architecture; gpt-realtime takes no image input). - Tool layer + Realtime handler now support async tools (await before sending function_call_output). - Diagnostics: video-receiver count at voice start, capture size, vision latency; graceful {error: camera unavailable} so the model admits it can't see instead of inventing. Co-Authored-By: Claude Fable 5 <norep…086eac3
-
feat: robot-only voice — remove phone mode, prime robot mic, visible motion
Juan Jimenez Carrero +1: Claude Fable 5Per user decision, the conversation is with the robot, full stop: - Removed the mic-source toggle and all phone-mic/phone-speaker paths. Voice requires the robot: its mic in, its speaker out (replaceTrack). Clear errors if either is unavailable instead of silent fallbacks. - Robot hearing fix: remote WebRTC audio tracks feed SILENCE into WebAudio on Chrome unless also consumed by a media element — prime the robot mic track with a hidden muted <audio> before the PCM pump reads it. This is the prime suspect for 'Reachy can't hear me'. - Motion: keep the 20 Hz head loop but make it clearly visible — speech envelope has a 0.35 floor while speaking (up to ~9° pitch), slow attentive sway while listening, antennas ease up smoothly while talking (slow lerp, never audio-rate). [motion] console logs every 3 s report level/env/ok so a failing send is visible in the console. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>45d3b03
-
fix: smooth head-only motion, no audio duplication, mic diagnostics
Juan Jimenez Carrero +1: Claude Opus 4.8- Movement: port the reachy-stories approach — a 20 Hz (not 60 Hz) client-side loop that wobbles the HEAD only (roll/pitch/yaw) from smoothed speech energy while talking, gentle head breath when idle. Antennas are no longer driven from the wobble — oscillating them at audio rate was the buzz. Daemon wobbler stays off; motion is client-side (it doesn't react to streamed WebRTC audio). - Duplicate audio: play TTS on the phone speaker ONLY when the robot isn't connected; when the robot plays it, phone output is muted (no double sound, no phone-mic feedback loop). - 'Reachy can't hear me': added mic-level logging (avg sample magnitude) to reveal whether the robot mic track actually carries audio; keep the worklet in the render graph via a 0-gain sink so process() reliably runs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>f71bdf7 -
fix: send TTS to robot speaker via replaceTrack (SDK 1.8 audio path)
Juan Jimenez Carrero +1: Claude Opus 4.8Root cause of 'no sound/movement from the robot, only phone mode works': SDK 1.8 no longer calls getUserMedia — it puts a silent placeholder on the WebRTC audio sender and apps must replaceTrack() their real audio onto it after streaming. Our getUserMedia patch was dead code, so TTS never reached the robot (phone mode worked only because it uses phone mic+speaker, bypassing the robot entirely — hence no robot movement either). Fix: drop the getUserMedia patch; on voice start, find the audio sender on reachy._pc and replaceTrack() our TTS bus (robotSpeakerDest) onto it; restore the placeholder on stop. Now the robot actually plays the speech, so it's audible AND the daemon wobbler has audio to animate the head from. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>8c09fe3
-
fix: reliable robot-mic capture + restore talking motion (daemon wobbler)
Juan Jimenez Carrero +1: Claude Opus 4.8- Read the robot mic track directly from reachy._pc receivers at session start (reliable) instead of only the connect-time interceptor global; if it's truly absent, fall back to the phone mic so 'Reachy's mic' never dead-ends (it was failing to start the session, so the robot neither heard nor talked). - Re-enable motion: with our client wobble gone there's no longer a conflict, so enable the daemon's purpose-built head wobbler during voice (set_wobbling true on start, false on stop). Single motion master = movement without the earlier two-master buzz. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>d9d0156 -
fix: disable daemon head-wobbler from app; restore robot-mic capture
Juan Jimenez Carrero +1: Claude Opus 4.8Two fixes from the deep dive: - Antenna buzz: the daemon's audio-reactive head wobbler (head-only, opt-in, likely enabled by Reachy Mini Control) is the remaining motion master that moved the head/antenna when our TTS played. Turn it off from the app via the data-channel command {type:set_wobbling, enabled:false} on connect and at voice start, so talking motion stays calm (no second master, no buzz). - 'Reachy's mic' didn't work (only phone did): setAudioMuted(true) on connect suppressed the robot's inbound mic track we capture for the voice loop. Stop muting inbound audio — keep the robot mic flowing so robot-mic mode works; only gate our outbound bus with setMicMuted. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>6edd0a8 -
fix: stop client-side head-wobble fighting the daemon's native wobbler
Juan Jimenez Carrero +1: Claude Opus 4.8Deep dive on the high-frequency antenna buzz: the reachy_mini DAEMON already runs a HeadWobbler that taps the speaker-playback audio and drives 6-DOF head offsets automatically (media/audio_gstreamer.py). Our client-side rAF wobble was a second master writing head/antenna targets at ~60 Hz, so the two fought over the motors — the buzz. Smoothing our values couldn't fix a two-master conflict. Fix: remove the client-side wobble entirely and let the daemon animate the head from the audio it's already playing. Also suppress tool-call gestures while a voice session is live (same two-master reason); touch gestures unaffected. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>e531223 -
fix: smooth speaking head-wobble + persist API key on paste
Juan Jimenez Carrero +1: Claude Opus 4.8- Head-wobble fed the antenna a raw frame-rate amplitude (~60 Hz of jitter) → high-frequency buzz. Run TTS energy through an envelope follower (fast attack / slow release) and drive slow sways scaled by that smooth energy; antennas perk symmetrically instead of flapping. Natural motion, no buzz. - OpenAI key (and HF token) saved only on 'change' (fires on blur), so pasting and tapping Talk to Reachy without leaving the field never persisted it. Save on every input event (incl. paste) instead. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>802251d -
fix: keep robot audio link muted until a voice session is active
Juan Jimenez Carrero +1: Claude Opus 4.8enableMicrophone opens the robot's mic/audio path for the whole session, which made the robot react to a live-but-idle link (one antenna vibrating from connect until end-session, before voice was even started). Mute mic+audio on connect; the VoiceController unmutes on start and re-mutes on stop, so the robot only opens its ears while you're actually talking. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>ec87c6b
-
feat: voice — talk to Reachy via OpenAI Realtime, wired to the kitchen tools
Juan Jimenez Carrero +1: Claude Fable 5Browser voice loop modeled on Pollen's reachy-stories (static-Space pattern): - OpenAI Realtime (gpt-realtime) over WebSocket with the key in the Sec-WebSocket-Protocol slot; key pasted in Settings, kept on-device. - Audio: getUserMedia patched to a synthetic destination so TTS plays on the robot speaker; robot mic captured via an RTCPeerConnection interceptor in index.html; 24 kHz PCM16 mic pump (AudioWorklet) → input buffer. - Function calling wired to the ported TS domain (add/cook/list inventory, set family, generate plan, get day, generate/list shopping) — voice and touch share state and update live. - Mic-source toggle (Reachy's mic at home / phone mic away); AI voice to the robot speaker with client-side head-wobble, or the phone speaker away. - Floating 'Talk to Reachy' button with connecting/listening/speaking states. - enableMicrophone enabled so the SDK sets up the outbound audio path. Verified in preview: UI, no-key guidance, settings. Live voice needs the user…e6c8aff