README.md

metadata

title: Reachy Mini Dance Tutor App
emoji: 💃
colorFrom: red
colorTo: blue
sdk: static
pinned: false
short_description: Dance with Reachy Mini!
suggested_storage: large
tags:
  - reachy_mini
  - reachy_mini_python_app

Reachy Mini Dance Tutor App

For students: This README tells you what this project is, how we built the Dance Challenge booth idea, and exactly what to run on your computer to try it with a simulator (fake robot) or a real robot.

For the science fair or business fair: Read DESIGN.md next. It has simple diagrams and words you can use to explain the project to visitors.

Reachy Mini Dance

---

What is this? (simple version)

Reachy Mini is a small robot that can listen, talk, move, and use its camera.

This dance tutor app is Python code that:

1. Connects to the robot (or simulator) through the official Reachy Mini software.
2. Connects to an AI in the cloud so the robot can have a voice chat.
3. Lets the AI call tools — special actions like "play a dance" or "watch the camera and give points."

Our team added a Dance Challenge mode: the robot hosts a mini game (demo dance → kid dances → points for the booth). How that mode works — in words you can repeat at a fair — is in DESIGN.md.

---

How it is built (big picture)

You speak  →  App sends audio  →  AI thinks and replies  →  App plays sound on the robot
                              ↘  AI can call tools  →  App moves robot / uses camera

• Instructions for the AI live in a profile (a folder with instructions.txt and tools.txt).
• Custom tools for our booth live in src/reachy_dance_tutor_app/profiles/dance_challenge/ (for example scoring).
• Shared tools (camera, generic dance, head move, …) live in src/reachy_dance_tutor_app/tools/.

The architecture diagram in the repo (docs/assets/dance_tutor_app_arch.svg) shows the same idea in more detail for adults.

---

What you need before you run anything

Need	Why
Python 3.10+ (3.12 works well)	Runs the app.
This repo installed	pip install -e . or uv sync — see below.
Reachy Mini SDK + daemon	Robot or simulator talks through the daemon.
API key	Copy .env.example → .env and add OPENAI_API_KEY (or Gemini — see table below).
Dance Challenge profile	Set REACHY_MINI_CUSTOM_PROFILE=dance_challenge in .env.

---

Run locally and test with the simulator (no physical robot)

The simulator is a 3D physics window on your computer. It follows the official guide: Reachy Mini simulation – get started.

Step 1 — Install the project

Option A — uv (fast):

uv venv --python 3.12
# Windows PowerShell:
.venv\Scripts\Activate.ps1
# macOS / Linux:
# source .venv/bin/activate

uv sync

Option B — pip:

python -m venv .venv
# Windows PowerShell:
.venv\Scripts\Activate.ps1
# macOS / Linux:
# source .venv/bin/activate

pip install -e .

Step 2 — Install MuJoCo support for the simulator

Still in the same environment:

pip install "reachy-mini[mujoco]"

Step 3 — Configure environment

1. Copy .env.example to .env.
2. Add your OPENAI_API_KEY (or GEMINI_API_KEY if you use Gemini).
3. Set the Dance Challenge profile, for example:

REACHY_MINI_CUSTOM_PROFILE=dance_challenge

Step 4 — Start the simulated robot

In one terminal (venv activated):

reachy-mini-daemon --sim

You should see a MuJoCo window. Optional: reachy-mini-daemon --sim --scene minimal for a table and props.

macOS: if the normal command fails, use:

mjpython -m reachy_mini.daemon.app.main --sim

Step 5 — Start the dance tutor app

Open a second terminal (same venv, same .env). Simulation needs the web UI:

reachy-mini-dance-tutor-app --gradio

Open the URL it prints (usually http://127.0.0.1:7860/) and use the browser to talk to the robot.

If it says "timeout" or "cannot connect"

The daemon from Step 4 must be running before you start the app. See Reachy Mini troubleshooting.

Note: On Windows, Reachy Mini support is still experimental — if the simulator fails, try another computer or use real hardware.

---

Run with a real Reachy Mini

1. Start the daemon without --sim (follow the hardware guide from Reachy Mini docs).
2. Same .env and profile as above.
3. Run:

reachy-mini-dance-tutor-app --gradio

(Console-only mode exists, but Gradio is easier for demos.)

Add a head tracker only if you installed the matching extras, for example:

reachy-mini-dance-tutor-app --gradio --head-tracker mediapipe

---

Install on a Reachy Mini (wireless hardware)

Follow the Reachy Mini get started guide to set up the robot first, then install this app via the Reachy Mini Control dashboard (it appears as reachy_mini_dance_tutor_app in the Applications list).

If the app crashes on start, check the logs for a TypeError: 'DaemonStatus' object is not subscriptable error — this has been fixed in the current version. Make sure you have the latest code installed.

---

Configuration (short list)

Variable	What it does
OPENAI_API_KEY	Voice + tools with OpenAI realtime (common setup).
GEMINI_API_KEY	Alternative if you use Gemini instead.
MODEL_NAME	Optional: change which model speaks.
REACHY_MINI_CUSTOM_PROFILE	Set to dance_challenge for the booth personality and tools.
HF_TOKEN	May be needed for some emotion clips (play_emotion).

Full list: see .env.example and the Configuration section below for developers.

---

Try automated checks (optional)

With dev dependencies installed (pip install -e .[dev] or uv sync --group dev):

pytest tests/test_dance_challenge_profile.py -q

That checks that the Dance Challenge tools and scoring math load correctly (no robot required).

---

LLM tools (Dance Challenge profile)

The AI only sees the tools listed in profiles/dance_challenge/tools.txt. Important ones for the fair:

Tool	Kid-friendly explanation
dance_routine	Plays the scripted demo for the style the kid picked.
kid_dance_score	Watches the camera for a timed window and returns points and prize tier hints.
play_emotion	Makes the robot show a happy or proud motion.
camera	Optional fun comments from a picture (scoring uses kid_dance_score, not this).

Other tools: dance, stop_dance, move_head, head_tracking, do_nothing, etc.

---

Overview (technical)

• Real-time audio conversation loop powered by the OpenAI realtime API and fastrtc for low-latency streaming.
• Vision processing uses gpt-realtime by default when the camera tool is used, with optional local vision using SmolVLM2 via --local-vision and the local_vision extra.
• Layered motion system queues primary moves (dances, emotions, goto poses, breathing) while blending speech-reactive wobble and face-tracking.
• Async tool dispatch integrates robot motion, camera capture, and optional face-tracking through a Gradio web UI with live transcripts.

---

Installation (developers)

Before using this app, install Reachy Mini's SDK.
Windows support is experimental.

Using uv

uv venv --python 3.12.1
source .venv/bin/activate   # or Windows: .venv\Scripts\Activate.ps1
uv sync

To match the lockfile exactly: uv sync --locked (or --frozen).

Optional extras:

uv sync --extra reachy_mini_wireless
uv sync --extra local_vision
uv sync --extra yolo_vision
uv sync --extra mediapipe_vision
uv sync --extra all_vision
uv sync --extra all_vision --group dev

Using pip

python -m venv .venv
source .venv/bin/activate   # or Windows: .venv\Scripts\Activate.ps1
pip install -e .
pip install -e .[reachy_mini_wireless]
pip install -e .[local_vision]
pip install -e .[yolo_vision]
pip install -e .[mediapipe_vision]
pip install -e .[all_vision]
pip install -e .[dev]

Some wheels (e.g. PyTorch) are large — pick extras that match your CPU/GPU.

Optional dependency groups

Extra	Purpose
reachy_mini_wireless	Wireless Reachy Mini + GStreamer.
local_vision	Local VLM (SmolVLM2).
yolo_vision	--head-tracker yolo.
mediapipe_vision	--head-tracker mediapipe.
all_vision	All vision extras.
dev	pytest, ruff, etc.

---

Configuration (full table)

1. Copy .env.example to .env.
2. Fill in API keys and options.

Variable	Description
OPENAI_API_KEY	OpenAI API key for OpenAI realtime.
GEMINI_API_KEY	Gemini API key if not using OpenAI.
GEMINI_BASE_URL	Optional OpenAI-compatible Gemini base URL.
MODEL_NAME	Override realtime model.
INPUT_TRANSCRIPTION_MODEL	Optional transcription model.
HF_HOME	Cache dir for Hugging Face (local vision).
HF_TOKEN	Optional HF token (emotions dataset, models).
LOCAL_VISION_MODEL	Model path for --local-vision.

---

Running the app (CLI reference)

reachy-mini-dance-tutor-app

Option	Default	Description
--head-tracker {yolo,mediapipe}	off	Face tracking (needs matching extra).
--no-camera	off	No camera or face tracking.
--local-vision	off	Local SmolVLM2 instead of cloud vision for image tools.
--gradio	off	Web UI; use with simulator.
--debug	off	Verbose logs.

Examples:

reachy-mini-dance-tutor-app --head-tracker mediapipe
reachy-mini-dance-tutor-app --local-vision
reachy-mini-dance-tutor-app --no-camera
reachy-mini-dance-tutor-app --gradio

---

Using custom profiles

Set REACHY_MINI_CUSTOM_PROFILE=<name> to load src/reachy_dance_tutor_app/profiles/<name>/. Each profile needs instructions.txt and tools.txt; optional Python tools in the same folder.

Placeholders in instructions.txt like [identities/witty_identity] pull snippets from src/reachy_dance_tutor_app/prompts/.

With --gradio, the Personality panel can switch instructions live; tool lists load at startup.

---

Development workflow

• uv sync --group dev or pip install -e .[dev]
• ruff check .
• pytest

License

Apache 2.0