Commit History
-
fix: increase OpenClaw bridge timeout + improve ask_openclaw error handling
tomrikert- Increase bridge timeout from 120s to 300s to accommodate slow local LLM (Qwen3-32B takes ~50s per inference; tool-heavy queries need 2-3 cycles) - Add auto-reconnect in ask_openclaw if bridge is disconnected - Improve error messages so OpenAI Realtime model reports accurate issues instead of misinterpreting timeouts as 'permissions problems' - Add logging for ask_openclaw queries3785255 -
feat: add OpenAI Vision API for camera analysis + fix OpenClaw bridge auth
tomrikert- Add _analyze_image_with_openai() using gpt-4o-mini for reliable cloud vision - Camera tool now tries: 1) local SmolVLM2, 2) OpenAI Vision API, 3) OpenClaw - Fix OpenClaw bridge client ID to 'openclaw-control-ui' to preserve scopes - This resolves the 'missing scope: operator.write' error for chat.send373ee8f -
Fix crash resilience and OpenClaw gateway connectivity
tomrikert- openai_realtime.py: Replace 3-attempt retry with infinite reconnect loop so the robot stays alive through network blips and idle timeouts. Uses exponential backoff capped at 30s with jitter. - openclaw_bridge.py: Send Origin header on WebSocket connect so the gateway's controlUi origin check accepts the robot client.0843272 -
Update default OpenAI realtime model to gpt-realtime-1.5
tomrikert08a3c9c
-
Switch OpenClaw bridge from HTTP to WebSocket protocol
tomrikertThe OpenClaw gateway speaks WebSocket natively, not HTTP REST. The old HTTP-based bridge (httpx POST to /v1/chat/completions) would connect but hang forever waiting for a response. Rewrite openclaw_bridge.py to use the gateway's WebSocket protocol: - Connect handshake with challenge/auth/scopes - Persistent connection with background listener task - chat.send with streaming event collection (agent deltas, lifecycle) - Proper disconnect cleanup on shutdown Also includes: room-scanning when no face detected, improved face tracking EMA tuning, and thinking animation offsets in movement system.375b02a
-
Add EMA smoothing to face tracking to reduce jitter
tomrikert1607ad8 -
Fix missing Any import in main.py
tomrikertc8c58de -
Add local face tracking (YOLO) and vision processing (SmolVLM2)
tomrikertFace Tracking: - Initialize YOLO HeadTracker for local face detection - Face tracking now works out of the box with --no-face-tracking to disable - CameraWorker properly integrates with head tracker Local Vision (optional): - Add VisionProcessor using SmolVLM2 for on-device image understanding - VisionManager for periodic scene description - Enable with --local-vision flag or ENABLE_LOCAL_VISION=true - Falls back to OpenClaw for vision if local not available Configuration: - ENABLE_FACE_TRACKING (default: true) - ENABLE_LOCAL_VISION (default: false) - LOCAL_VISION_MODEL (default: SmolVLM2-256M-Video-Instruct) - VISION_DEVICE (auto, cuda, mps, cpu) - HF_HOME for model cache directory Based on pollen-robotics/reachy_mini_conversation_app implementation.c9c7adc -
Refactor to native OpenAI Realtime with OpenClaw identity
tomrikertMajor architecture changes: - Use OpenAI Realtime API natively instead of hybrid approach - Fetch OpenClaw agent context (personality, memories) at startup - Robot now IS the OpenClaw agent with full context - Sync conversations back to OpenClaw for memory continuity Camera/Vision improvements: - Initialize camera worker properly for video streaming - Camera tool now sends images to OpenClaw for vision analysis - Added detailed logging for camera debugging Key changes: - openai_realtime.py: Native conversation flow with tools - openclaw_bridge.py: Added get_agent_context() and sync_conversation() - main.py: Proper camera worker initialization and lifecycle - core_tools.py: Camera tool uses OpenClaw for image descriptionab3858a
-
Fix speech cutoff and echo: track speaking state, wait for responses
tomrikertIssues fixed: - Robot's speech was being picked up by mic (echo/feedback) - New responses started before previous ones finished Changes: - Add _speaking flag to ignore mic during TTS - Add _response_done_event to wait for completion - Ignore speech_started and transcripts while speaking - Wait for active response before starting new TTSc163d58 -
Fix TTS: cancel OpenAI auto-response, route through OpenClaw
tomrikertPrevious approach (create_response: False) broke the TTS system. New approach: - Allow OpenAI auto-response to keep WebSocket healthy - Cancel it when transcript arrives - Route to OpenClaw for AI response - Use simpler TTS call with instructions93366b6 -
Add acknowledgment for long-running queries to prevent timeout
tomrikertWhen detecting likely long-running queries (weather, search, etc.), the robot now says "Let me check on that..." before calling OpenClaw. This keeps the OpenAI WebSocket alive during tool execution.f78756e -
Fix modalities error: must use ['text', 'audio'] not ['audio']
tomrikertdf8b068 -
Fix dual-response issue: disable OpenAI auto-response
tomrikertThe robot was responding with both OpenAI and OpenClaw because: - turn_detection was auto-generating responses - tools were enabled allowing OpenAI to act independently Fix: - Set create_response: False in turn_detection - Remove tools from OpenAI session (OpenClaw handles AI) - Improve TTS to add text as message first, then generate audio Now OpenAI Realtime is purely STT + TTS, all intelligence from OpenClaw.7e9c528 -
Use x-openclaw-session-key header for proper session routing
tomrikertThe robot now uses the x-openclaw-session-key header to target the main session (agent:main:main), which is the same session used by WhatsApp and other DM channels. Per OpenClaw docs, the 'user' field creates a separate session (openai-user:<user>), while the session-key header provides direct control over session routing. Changes: - Add x-openclaw-session-key header to HTTP requests - Replace OPENCLAW_USER_ID with OPENCLAW_SESSION_KEY config - Default to "main" session for cross-channel context sharing0c3977c -
Fix cross-channel conversation awareness with shared user ID
tomrikertThe robot now uses the same user ID as other channels (WhatsApp, web) so OpenClaw can share conversation context across all communication methods. Changes: - Add OPENCLAW_USER_ID config option (default: "tom") - Update bridge to use configurable user ID instead of hardcoded value - Update .env.example with documentation Set OPENCLAW_USER_ID in your .env to match your WhatsApp user ID.70b1eb3 -
Fix conversation awareness across channels (WhatsApp, web, robot)
tomrikertRemove local message history tracking - let OpenClaw's memory system handle conversation context. This allows the robot to be aware of messages sent/received on other channels like WhatsApp. Previously ClawBody maintained its own message history, which meant it had no awareness of conversations happening elsewhere. Now each message is sent individually and OpenClaw provides context from its unified memory system.8acae21 -
Add face tracking for robot to look at person speaking
tomrikertFeatures: - Camera worker with 25Hz frame capture and face tracking - MediaPipe and YOLO head tracker options (auto-detect) - Smooth interpolation back to neutral when face is lost - Integration with movement system as secondary offsets at 100Hz New files: - camera_worker.py: Thread-safe camera worker with face tracking - vision/: Head tracker module with YOLO and MediaPipe options CLI options: - --head-tracker [yolo|mediapipe]: Select tracker type - --no-face-tracking: Disable face tracking Config: - ENABLE_FACE_TRACKING: Enable/disable via env var - HEAD_TRACKER_TYPE: Set tracker type via env var379a66d
-
Initial release of ClawBody
tomrikertGive your OpenClaw AI agent a physical robot body with Reachy Mini. Features: - Real-time voice conversation via OpenAI Realtime API - OpenClaw/Clawson intelligence for AI responses - Vision through robot camera - Expressive movements (head, emotions, dances)c1956d8