Spaces:
Running
Running
Commit History
-
feat: redo for the meal plan (button + voice), completing undo/redo
Juan Jimenez Carrero +1: Claude Opus 4.8Undo discarded the state it undid from, so there was no way forward. Added a proper redo stack: - undoPlan now pushes the current plan onto planFuture before restoring; redoPlan pops it back. snapshotPlan (any fresh edit) clears planFuture, so a new change correctly invalidates the redo future — standard undo/redo semantics. - Redo button appears in the week header next to Undo whenever there's something to redo (mirrored icon, plan_redo string EN/ES). - Voice parity: redo_plan_change tool -> ctx.redoPlan; prompt mentions it alongside undo. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>88ed190 -
feat: sync robot preferences to HF, thinking movement + wait/announce on plan
Juan Jimenez Carrero +1: Claude Opus 4.8Settings sync (user noticed prefs didn't sync): the HF dataset only carried AppData. Now a NON-SECRET settings subset (language, robotName, voice, activationMode, allowInterrupt, inventorySort) rides along in a separate cookaiware-settings.json. Secrets (openaiKey, hfToken) and device-local state (lastSyncedAt, hfDatasetSync) are deliberately never uploaded. pushToCloud sends them; "Load from cloud" and the auto-sync "keep cloud" branch pull + validate them (applySyncableSettings keeps the current value for any invalid field). Silent-thinking fix: the menu designer takes ~10-20 s during which the robot sat still and mute. Now: - VoiceController.startThinking/stopThinking pulses a VARIED thinking body language (mostly "thinking", an occasional glance) every 2.6 s; app wraps both designAndCommitPlan and designMeal awaits with it (finally-stopped, also cleared on session stop). Only active during a live voice session. - Prompt now tells Reachy to ask the user to wait BEFORE calling…b5352be
-
feat: pick robot voice, lock/undo by voice, protect locked meals; fix key text
Juan Jimenez Carrero +1: Claude Opus 4.8Voice picker (like the official conversation app): new "Robot voice" setting in Talk to Reachy lists the OpenAI gpt-realtime voices (marin & cedar — the newest, most natural — plus alloy/ash/ballad/coral/echo/sage/shimmer/verse). Threaded settings.voice -> controller -> RealtimeSession.output.voice (was hardcoded shimmer). Default is now marin; applies on the next Talk-to-Reachy connect. Lock + undo by voice (parity with the app buttons): - set_meal_lock tool -> app.setMealLock(day, meal, locked): keep/free one meal. - undo_plan_change tool -> app.undoPlan(): restore the previous menu. - Reused resolveDay so 'today'/'tomorrow'/weekday all work. Locked meals are now protected: set_day_meal (and the app's designMeal) refuse a locked slot, returning a 'locked' error; the prompt tells Reachy to ask the user and only unlock+replace on confirmation. Previously voice could silently overwrite a kept meal. Accuracy fix (user-flagged): the OpenAI key hint wrongly said vision uses gpt-4o-mini …f080bc3 -
fix: precise single-meal voice editing (set_day_meal) — stop full-week regen
Juan Jimenez Carrero +1: Claude Opus 4.8Regression from the previous batch: removing the inline planning tools left voice with ONLY design_meal_plan, so any spoken request ("a dinner for tonight with the item that just expired") regenerated the whole week and ignored the specific ask. Added a set_day_meal voice tool + app-side designMeal(): - Edits ONE slot (day + breakfast/lunch/dinner) and leaves the rest of the week untouched; creates a plan if none exists yet. - Resolves spoken days: today/tomorrow/hoy/mañana or any weekday. - Passes the user's exact words as `request`, turned into a strong must-honor note ("include any ingredient they name even if it is expiring/expired") so the concrete item actually lands in that meal. - Servings come from the matrix cell, or the family size if that slot isn't normally planned. Snapshots for undo; spins that meal's regen icon; avoids repeating the week's other dishes. Prompt now tells Reachy to choose scope: one meal/day -> set_day_meal, whole new week -> design_meal_plan,…f1b541d
-
fix: language lock, date+time, voice uses thinking-menu, plan undo + spin
Juan Jimenez Carrero +1: Claude Opus 4.8Language drift: the 'always Spanish/English' rule was buried mid-prompt and the model wandered. Moved it to the FIRST line as an absolute rule ('no matter what language you hear, never switch'). Date/time: nothing was ever injected — the model guessed. Now the session prompt states the current weekday, date, TIME and timezone at connect, so it can answer 'what day/time is it'. Voice menus were basic: the in-session realtime model hand-wrote weak menus. Voice now calls a single design_meal_plan tool that routes through the SAME headless 'thinking' generator the Generate button uses (get_planning_context + save_meal_plan removed so the weak inline path is gone). It takes an optional guidance arg for one-off spoken requests ('vegetarian this week') merged with the saved wishes, and the prompt tells Reachy to say it's thinking + play the thinking emotion before the call. Plan destruction safety: - When a new plan is generated (button OR voice), the regenerate icons on the meals being …6212603
-
feat: lock-all, plan wishes, location tags in form, shopping usage, PWA
Juan Jimenez Carrero +1: Claude Opus 4.8Plan: - 'Keep all' / 'Free all' buttons next to the week header lock or unlock every meal at once. - 'Wishes for this week' free-text field ('Friday is pizza night; paella on Sunday') stored in family_profile.plan_notes (synced with the family data) and honored by BOTH planners: the button's headless LLM session and the voice flow (get_planning_context exposes user_wishes). Inventory form: - Existing storage locations shown as tappable tags under the field — one tap reuses the exact label; saves also canonicalize case-insensitively (inv.canonicalizeLocation) so 'Nevera'/'nevera' never fork. Shopping: - Tap an item to see exactly which planned meals need it and when (day, date, meal, dish, quantity) — or a note that it was added manually. Professional polish: - PWA manifest + apple-touch-icon + theme-color: the app is installable to the home screen with the CookAIware icon and standalone display. - Settings footer shows the build date + Space link for support. Co-Aut…9f40540 -
feat: matrix bulk edit, per-meal lock/regenerate, smart voice planning only
Juan Jimenez Carrero +1: Claude Opus 4.8Matrix: - Tap a day name to edit its whole row, a meal header for its whole column, and the ✎ corner for every cell at once (prefilled with the shared value when uniform, else the family size). Cells still edit individually. Plan editing: - Every meal has a padlock: locked meals ('keep') survive any regeneration, and the regenerate flows exclude their slots + pass them as context so the model doesn't repeat them. New per-meal 🔄 regenerates just that one meal (single-slot LLM call, everything else untouched; algorithmic fallback). - save_meal_plan enforces locks server-side: whatever the model sends, locked meals are merged back. Voice planning: - Removed the quick generate_meal_plan tool from the voice toolset — the model was taking the lazy path and producing rotation-quality plans. The ONLY voice path is now get_planning_context (lock-aware, includes kept_meals) + save_meal_plan, so voice plans are designed, like the button. Shopping: - Trash button removes an i…595eb1e -
feat: voice proactivity/interrupt settings + intelligent matrix-based menus
Juan Jimenez Carrero +1: Claude Opus 4.8Voice settings (apply on the next session): - Activation mode: respond to any speech, or ONLY when addressed by the robot's name (prompt-enforced wake rule; stays fully silent otherwise). - Allow interruptions: maps to server VAD interrupt_response, keeps the mic pump live while the robot speaks, cuts local TTS on speech_started, and skips input_buffer clears that would clip a barge-in. Meal planning overhaul: - Family size is now REQUIRED to plan (gate + CTA to Settings; voice tools return an instructive error that makes the model ask and call set_family). - 'Who eats what' 7×3 matrix (days × breakfast/lunch/dinner): each cell holds adults+children for that meal; tap → steppers, 'not planned' clears; stored as family_profile.plan_matrix (legacy weekday/weekend schedule auto-derives the default matrix, preserving the original app's behavior). - Intelligent generation through gpt-realtime (the key's only scope): * Button: short headless TEXT realtime session (planLLM.ts…cd6a098
-
fix: garbled voice on tool calls, sync safety, edit-not-duplicate, calmer persona
Juan Jimenez Carrero +1: Claude Opus 4.8- Garbled/'buzzing' speech before plan creation: we requested a new response while the previous one was still streaming, so two responses' audio chunks interleaved on one playback timeline. Responses are now gated: track response.created/done and defer response.create until the active one ends. - Auto-sync: enabling the toggle now downloads the cloud copy first and, if it differs, asks which side to keep (cloud vs this device) before the first auto-save — it can no longer silently overwrite. Switch styling fixed (the global input rules were deforming the checkbox). - New update_inventory_item tool: edits an existing item in place (location, expiration, quantity, rename); prompt now forbids add_inventory for changes, which was creating duplicates. - Sort by Location added (groups by location, then name; no-location last). - Robot name setting (default Reachy) fed into the voice persona; persona rewritten to be REACTIVE: do only what was asked, one-two short sentences, …ff39cb2
-
feat: conversation minimize, HF sync UX, inventory editing/sort/drag/tags
Juan Jimenez Carrero +1: Claude Opus 4.8Conversation: - Minimize/expand the transcript panel (chevron + tap-header-to-expand; minimized header shows the message count). HF sync: - Save/Load side by side; each asks for confirmation and states what will be overwritten (item count, plan y/n, shopping count) before replacing. - Auto-sync toggle: debounced upload ~4s after each change (reuses hfDatasetSync); shows 'last synced' relative time. Inventory: - Edit any item in a styled modal (name/qty/unit/expiration/location); new inv.normalizeItemInput replaces the item in place. - Restyled 'Use' modal (was window.prompt) with an amount field + 'use all'. - Delete asks for confirmation. - Expiration date now shown on every card (clock icon + date, red if overdue); sort control: Expiration / Name / Custom. - Custom mode: pointer-based drag-to-reorder (works on touch), persisted. - Storage location shown as a chip; tapping it filters the list to that location; add/edit location field autocompletes from existing locations…a03aea0
-
fix: HF sync save — don't fail on 'repo already exists' + surface real errors
Juan Jimenez Carrero +1: Claude Fable 5'Load from my HF account' worked but 'Save' always failed once the dataset existed: ensureRepo ran createRepo first and only swallowed errors matching /409|exist/i, but the Hub words the already-exists error differently, so every save after the first aborted before uploading. createRepo failures are now never fatal (uploadFiles is the authority and raises the real error if the repo is truly missing), and sync toasts now include the actual error message instead of the generic 'check the token'. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>37cc891 -
feat: vision via native gpt-realtime image input (no extra key scope)
Juan Jimenez Carrero +1: Claude Fable 5Juan's OpenAI key is restricted to the Realtime API (401 'Missing scopes: model.request' on chat/completions) and can't be replaced. gpt-realtime supports image input natively (conversation.item.create with input_image data URL) — the earlier 'unsupported' finding was Azure-only. So drop the gpt-4o-mini sidecar: take_picture now returns the captured frame, and the Realtime handler attaches it as a user message (input_image + guidance text) right after the function output, then requests an audio response. One model, one scope, native vision; the model can chain add_inventory directly. identifyWithVision kept exported as a future fallback. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>7f0bb1c -
fix: surface the real vision-request error + camera exposure warm-up
Juan Jimenez Carrero +1: Claude Fable 5The screenshot case: photo captured and displayed, then the model claimed 'the camera isn't available' — because every failure collapsed into {error: camera unavailable}. Now capture errors and vision-request errors are separated: vision failures toast the real reason (401/429/model access/…) and return it in the tool output so the model explains honestly ('photo was taken, but analyzing it failed: …'). Also let auto-exposure settle 450 ms before grabbing the frame — first decoded frames come out dark. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>9c60e23 -
feat: real vision — take_picture tool + photo in the conversation
Juan Jimenez Carrero +1: Claude Fable 5Stop the model hallucinating sight and give it actual eyes: - Hard prompt rule: no continuous vision; the ONLY way to see is the new take_picture tool; never describe surroundings except from its result. - take_picture(question): grabs ONE still frame from the robot's WebRTC video track (off-screen sink, nothing streamed or recorded), shows the photo as a polaroid-style bubble in the conversation panel, then sends it to gpt-4o-mini (same user key) which returns a description + candidate food items {name, quantity?, unit?} the model can offer to add_inventory — the Python app's camera.py flow, ported (two-step architecture; gpt-realtime takes no image input). - Tool layer + Realtime handler now support async tools (await before sending function_call_output). - Diagnostics: video-receiver count at voice start, capture size, vision latency; graceful {error: camera unavailable} so the model admits it can't see instead of inventing. Co-Authored-By: Claude Fable 5 <norep…086eac3 -
fix: user transcription model + brand icons + date-input overflow
Juan Jimenez Carrero +1: Claude Fable 5- User speech still not transcribed with whisper-1 on GA sessions; the community-verified GA recipe uses gpt-4o-mini-transcribe under session.audio.input.transcription — switched to it. Also log session.updated (shows the accepted config), server error events, and each unseen event type once, so the console now pinpoints any remaining transcription gap. - Replace platform emojis (tabs + talk button) with rounded line SVG icons in the landing's hand-drawn style: basket, calendar, cart, sliders; mic, stop, speaker, spinner (animated while connecting). - Fix date input overflowing its card on iOS/Safari (appearance reset + min-width:0 on inputs and form-grid cells). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>66844d3 -
feat: brand-styled app + GA transcription fix + smoothness pass
Juan Jimenez Carrero +1: Claude Fable 5- User speech transcription: GA Realtime sessions configure transcription under session.audio.input.transcription — our flat input_audio_transcription was silently ignored, so transcription defaulted OFF and the user's words never appeared. Switched session.update to the documented GA nested shape (audio.input format/transcription/turn_detection, audio.output voice/format); also handle transcription.failed with a console warning. - Restyle: the app now uses the hand-made landing's visual language (cream gradient, Fraunces headings, Nunito body, teal/peach palette, white 24px-radius cards, pill FAB). The conversation panel is the landing's chat: peach assistant bubbles + teal user bubbles with the family avatars. - Smoothness: preserve scroll across re-renders, tap-highlight removed, touch-action manipulation, :active scale feedback, slide-in bubbles, animated toasts, focus rings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>d22f4d1
-
feat: live conversation transcript panel during voice
Juan Jimenez Carrero +1: Claude Fable 5Bring back the old app's conversation view: a panel above the talk button shows what Reachy hears (🧑, user STT) and answers (🤖, assistant transcript) in real time, with auto-scroll and a Clear action. Appears when a voice session starts (empty-state hint until the first exchange) and stays while there is history — doubling as a diagnostic: if your words show up as text, the robot mic path is working even before audio comes back. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>55261bc -
feat: robot-only voice — remove phone mode, prime robot mic, visible motion
Juan Jimenez Carrero +1: Claude Fable 5Per user decision, the conversation is with the robot, full stop: - Removed the mic-source toggle and all phone-mic/phone-speaker paths. Voice requires the robot: its mic in, its speaker out (replaceTrack). Clear errors if either is unavailable instead of silent fallbacks. - Robot hearing fix: remote WebRTC audio tracks feed SILENCE into WebAudio on Chrome unless also consumed by a media element — prime the robot mic track with a hidden muted <audio> before the PCM pump reads it. This is the prime suspect for 'Reachy can't hear me'. - Motion: keep the 20 Hz head loop but make it clearly visible — speech envelope has a 0.35 floor while speaking (up to ~9° pitch), slow attentive sway while listening, antennas ease up smoothly while talking (slow lerp, never audio-rate). [motion] console logs every 3 s report level/env/ok so a failing send is visible in the console. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>45d3b03
-
fix: stop client-side head-wobble fighting the daemon's native wobbler
Juan Jimenez Carrero +1: Claude Opus 4.8Deep dive on the high-frequency antenna buzz: the reachy_mini DAEMON already runs a HeadWobbler that taps the speaker-playback audio and drives 6-DOF head offsets automatically (media/audio_gstreamer.py). Our client-side rAF wobble was a second master writing head/antenna targets at ~60 Hz, so the two fought over the motors — the buzz. Smoothing our values couldn't fix a two-master conflict. Fix: remove the client-side wobble entirely and let the daemon animate the head from the audio it's already playing. Also suppress tool-call gestures while a voice session is live (same two-master reason); touch gestures unaffected. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>e531223 -
fix: smooth speaking head-wobble + persist API key on paste
Juan Jimenez Carrero +1: Claude Opus 4.8- Head-wobble fed the antenna a raw frame-rate amplitude (~60 Hz of jitter) → high-frequency buzz. Run TTS energy through an envelope follower (fast attack / slow release) and drive slow sways scaled by that smooth energy; antennas perk symmetrically instead of flapping. Natural motion, no buzz. - OpenAI key (and HF token) saved only on 'change' (fires on blur), so pasting and tapping Talk to Reachy without leaving the field never persisted it. Save on every input event (incl. paste) instead. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>802251d
-
feat: voice — talk to Reachy via OpenAI Realtime, wired to the kitchen tools
Juan Jimenez Carrero +1: Claude Fable 5Browser voice loop modeled on Pollen's reachy-stories (static-Space pattern): - OpenAI Realtime (gpt-realtime) over WebSocket with the key in the Sec-WebSocket-Protocol slot; key pasted in Settings, kept on-device. - Audio: getUserMedia patched to a synthetic destination so TTS plays on the robot speaker; robot mic captured via an RTCPeerConnection interceptor in index.html; 24 kHz PCM16 mic pump (AudioWorklet) → input buffer. - Function calling wired to the ported TS domain (add/cook/list inventory, set family, generate plan, get day, generate/list shopping) — voice and touch share state and update live. - Mic-source toggle (Reachy's mic at home / phone mic away); AI voice to the robot speaker with client-side head-wobble, or the phone speaker away. - Floating 'Talk to Reachy' button with connecting/listening/speaking states. - enableMicrophone enabled so the SDK sets up the outbound audio path. Verified in preview: UI, no-key guidance, settings. Live voice needs the user…e6c8aff -
feat: onboarding journey — prerequisites on landing + first-run setup
Juan Jimenez Carrero +1: Claude Fable 5Smooth the start of the user journey before and during first use: - Landing 'What you'll need' section: robot online (+ Lite daemon note), HF account, OpenAI key for voice, same-network expectation, and a 'can't see your robot?' troubleshooting note (powered on / daemon / one-app-at-a-time). - App first-run welcome card: sets family size up front (so meal plans are sized right) with a pointer to add the OpenAI key in Settings for voice. - Catalog thumbnail now resolves to the hand-made hero image. - New users start fresh (web app reads localStorage; repo's dev data unused). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>36e652b -
feat: restore the original hand-made landing design + wire app launch
Juan Jimenez Carrero +1: Claude Fable 5Bring back the real CookAIware landing (hero illustration, conversation demo with the custom avatars, how-it-works tabs, install steps, footer) and its style.css/fonts, instead of the minimal placeholder. The hero and topbar 'Connect your Reachy' buttons now launch the web/mobile app via the host shell handoff. Privacy/install copy updated to mention the web app + optional HF sync. Also fix deploy:root to copy the full build output (incl. public images) to the repo root. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>576b390 -
feat: informational landing page + clear new-user journey
Juan Jimenez Carrero +1: Claude Fable 5Opening the Space now lands on an info page (what CookAIware does, the 4 steps, the two flavours web+voice, what you need) instead of dropping straight into the sign-in. A prominent 'Connect your Reachy' button hands off to the host shell (sign-in -> picker -> app). dispatch.ts routes the OAuth redirect (?code=) and already-launched sessions straight to the host so the round-trip still completes. Bilingual ES/EN, theme-aware, mobile-first. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>82a4679
-
feat: add mobile-first JS web app (Pollen's new Reachy Mini JS workflow)
Juan Jimenez Carrero +1: Claude Fable 5New web/ TS+Vite app served as this Space's page, alongside the existing Python on-robot app — one Space, two flavours: - Host-shell integration per ts/APP_CREATION_GUIDE.md: dispatch/embed split, @pollen-robotics/reachy-mini-sdk pinned 1.8.1, onLeave -> safelyReturnToPose - Domain logic ported 1:1 from Python (inventory, algorithmic meal planner, shopping shortfalls, family profile); parity-checked against the Python implementation on the repo's real data (scripts/parity-check.ts) - Spanish keywords added to meal categorization (data is in Spanish) - Mobile-first UI: bottom tabs, 44px targets, dark/light via host theme, bilingual ES/EN; localStorage + JSON export/import + opt-in HF dataset sync - Robot gestures: rAF-tweened antenna wiggle, happy dance, concerned tilt, sweep look, greeting - Frontmatter: app_build_command/app_file/hf_oauth + reachy_mini_js_app tag Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>a519434