Switch OpenClaw bridge from HTTP to WebSocket protocol
tomrikert
375b02a
The OpenClaw gateway speaks WebSocket natively, not HTTP REST. The old
HTTP-based bridge (httpx POST to /v1/chat/completions) would connect but
hang forever waiting for a response.
Rewrite openclaw_bridge.py to use the gateway's WebSocket protocol:
- Connect handshake with challenge/auth/scopes
- Persistent connection with background listener task
- chat.send with streaming event collection (agent deltas, lifecycle)
- Proper disconnect cleanup on shutdown
Also includes: room-scanning when no face detected, improved face
tracking EMA tuning, and thinking animation offsets in movement system.
Add EMA smoothing to face tracking to reduce jitter
tomrikert
1607ad8
Fix missing Any import in main.py
tomrikert
c8c58de
Add local face tracking (YOLO) and vision processing (SmolVLM2)
tomrikert
c9c7adc
Face Tracking:
- Initialize YOLO HeadTracker for local face detection
- Face tracking now works out of the box with --no-face-tracking to disable
- CameraWorker properly integrates with head tracker
Local Vision (optional):
- Add VisionProcessor using SmolVLM2 for on-device image understanding
- VisionManager for periodic scene description
- Enable with --local-vision flag or ENABLE_LOCAL_VISION=true
- Falls back to OpenClaw for vision if local not available
Configuration:
- ENABLE_FACE_TRACKING (default: true)
- ENABLE_LOCAL_VISION (default: false)
- LOCAL_VISION_MODEL (default: SmolVLM2-256M-Video-Instruct)
- VISION_DEVICE (auto, cuda, mps, cpu)
- HF_HOME for model cache directory
Based on pollen-robotics/reachy_mini_conversation_app implementation.
Refactor to native OpenAI Realtime with OpenClaw identity
tomrikert
ab3858a
Major architecture changes:
- Use OpenAI Realtime API natively instead of hybrid approach
- Fetch OpenClaw agent context (personality, memories) at startup
- Robot now IS the OpenClaw agent with full context
- Sync conversations back to OpenClaw for memory continuity
Camera/Vision improvements:
- Initialize camera worker properly for video streaming
- Camera tool now sends images to OpenClaw for vision analysis
- Added detailed logging for camera debugging
Key changes:
- openai_realtime.py: Native conversation flow with tools
- openclaw_bridge.py: Added get_agent_context() and sync_conversation()
- main.py: Proper camera worker initialization and lifecycle
- core_tools.py: Camera tool uses OpenClaw for image description
Fix speech cutoff and echo: track speaking state, wait for responses
tomrikert
c163d58
Issues fixed:
- Robot's speech was being picked up by mic (echo/feedback)
- New responses started before previous ones finished
Changes:
- Add _speaking flag to ignore mic during TTS
- Add _response_done_event to wait for completion
- Ignore speech_started and transcripts while speaking
- Wait for active response before starting new TTS
Fix TTS: cancel OpenAI auto-response, route through OpenClaw
tomrikert
93366b6
Previous approach (create_response: False) broke the TTS system.
New approach:
- Allow OpenAI auto-response to keep WebSocket healthy
- Cancel it when transcript arrives
- Route to OpenClaw for AI response
- Use simpler TTS call with instructions
Add acknowledgment for long-running queries to prevent timeout
tomrikert
f78756e
When detecting likely long-running queries (weather, search, etc.),
the robot now says "Let me check on that..." before calling OpenClaw.
This keeps the OpenAI WebSocket alive during tool execution.
Fix modalities error: must use ['text', 'audio'] not ['audio']
The robot was responding with both OpenAI and OpenClaw because:
- turn_detection was auto-generating responses
- tools were enabled allowing OpenAI to act independently
Fix:
- Set create_response: False in turn_detection
- Remove tools from OpenAI session (OpenClaw handles AI)
- Improve TTS to add text as message first, then generate audio
Now OpenAI Realtime is purely STT + TTS, all intelligence from OpenClaw.
Use x-openclaw-session-key header for proper session routing
tomrikert
0c3977c
The robot now uses the x-openclaw-session-key header to target the
main session (agent:main:main), which is the same session used by
WhatsApp and other DM channels.
Per OpenClaw docs, the 'user' field creates a separate session
(openai-user:<user>), while the session-key header provides direct
control over session routing.
Changes:
- Add x-openclaw-session-key header to HTTP requests
- Replace OPENCLAW_USER_ID with OPENCLAW_SESSION_KEY config
- Default to "main" session for cross-channel context sharing
Fix cross-channel conversation awareness with shared user ID
tomrikert
70b1eb3
The robot now uses the same user ID as other channels (WhatsApp, web)
so OpenClaw can share conversation context across all communication
methods.
Changes:
- Add OPENCLAW_USER_ID config option (default: "tom")
- Update bridge to use configurable user ID instead of hardcoded value
- Update .env.example with documentation
Set OPENCLAW_USER_ID in your .env to match your WhatsApp user ID.
Fix conversation awareness across channels (WhatsApp, web, robot)
tomrikert
8acae21
Remove local message history tracking - let OpenClaw's memory system
handle conversation context. This allows the robot to be aware of
messages sent/received on other channels like WhatsApp.
Previously ClawBody maintained its own message history, which meant
it had no awareness of conversations happening elsewhere. Now each
message is sent individually and OpenClaw provides context from its
unified memory system.
Add face tracking for robot to look at person speaking
tomrikert
379a66d
Features:
- Camera worker with 25Hz frame capture and face tracking
- MediaPipe and YOLO head tracker options (auto-detect)
- Smooth interpolation back to neutral when face is lost
- Integration with movement system as secondary offsets at 100Hz
New files:
- camera_worker.py: Thread-safe camera worker with face tracking
- vision/: Head tracker module with YOLO and MediaPipe options
CLI options:
- --head-tracker [yolo|mediapipe]: Select tracker type
- --no-face-tracking: Disable face tracking
Config:
- ENABLE_FACE_TRACKING: Enable/disable via env var
- HEAD_TRACKER_TYPE: Set tracker type via env var
Initial release of ClawBody
tomrikert
c1956d8
Give your OpenClaw AI agent a physical robot body with Reachy Mini.
Features:
- Real-time voice conversation via OpenAI Realtime API
- OpenClaw/Clawson intelligence for AI responses
- Vision through robot camera
- Expressive movements (head, emotions, dances)
