--- title: Reachy Mini Dance Tutor App emoji: 💃 colorFrom: red colorTo: blue sdk: static pinned: false short_description: Dance with Reachy Mini! suggested_storage: large tags: - reachy_mini - reachy_mini_python_app --- # Reachy Mini Dance Tutor App **For students:** This README tells you **what this project is**, **how we built the Dance Challenge booth idea**, and **exactly what to run** on your computer to try it with a **simulator** (fake robot) or a **real robot**. **For the science fair or business fair:** Read **[DESIGN.md](DESIGN.md)** next. It has simple diagrams and words you can use to explain the project to visitors. Reachy Mini Dance --- ## What is this? (simple version) Reachy Mini is a small robot that can **listen**, **talk**, **move**, and use its **camera**. This **dance tutor app** is Python code that: 1. Connects to the robot (or simulator) through the official **Reachy Mini software**. 2. Connects to an **AI in the cloud** so the robot can have a voice chat. 3. Lets the AI call **tools** — special actions like "play a dance" or "watch the camera and give points." Our team added a **Dance Challenge** mode: the robot hosts a mini game (demo dance → kid dances → points for the booth). How that mode works — in words you can repeat at a fair — is in **[DESIGN.md](DESIGN.md)**. --- ## How it is built (big picture) ``` You speak → App sends audio → AI thinks and replies → App plays sound on the robot ↘ AI can call tools → App moves robot / uses camera ``` - **Instructions** for the AI live in a **profile** (a folder with `instructions.txt` and `tools.txt`). - **Custom tools** for our booth live in `src/reachy_dance_tutor_app/profiles/dance_challenge/` (for example scoring). - **Shared tools** (camera, generic dance, head move, …) live in `src/reachy_dance_tutor_app/tools/`. The architecture diagram in the repo (`docs/assets/dance_tutor_app_arch.svg`) shows the same idea in more detail for adults. --- ## What you need before you run anything | Need | Why | | ---------------------------------- | ------------------------------------------------------------------------------------ | | **Python 3.10+** (3.12 works well) | Runs the app. | | **This repo installed** | `pip install -e .` or `uv sync` — see below. | | **Reachy Mini SDK + daemon** | Robot or simulator talks through the daemon. | | **API key** | Copy `.env.example` → `.env` and add `OPENAI_API_KEY` (or Gemini — see table below). | | **Dance Challenge profile** | Set `REACHY_MINI_CUSTOM_PROFILE=dance_challenge` in `.env`. | --- ## Run locally and test with the **simulator** (no physical robot) The simulator is a **3D physics window** on your computer. It follows the official guide: [Reachy Mini simulation – get started](https://huggingface.co/docs/reachy_mini/platforms/simulation/get_started). ### Step 1 — Install the project **Option A — uv (fast):** ```bash uv venv --python 3.12 # Windows PowerShell: .venv\Scripts\Activate.ps1 # macOS / Linux: # source .venv/bin/activate uv sync ``` **Option B — pip:** ```bash python -m venv .venv # Windows PowerShell: .venv\Scripts\Activate.ps1 # macOS / Linux: # source .venv/bin/activate pip install -e . ``` ### Step 2 — Install MuJoCo support for the simulator Still in the same environment: ```bash pip install "reachy-mini[mujoco]" ``` ### Step 3 — Configure environment 1. Copy `.env.example` to `.env`. 2. Add your `OPENAI_API_KEY` (or `GEMINI_API_KEY` if you use Gemini). 3. Set the Dance Challenge profile, for example: ```env REACHY_MINI_CUSTOM_PROFILE=dance_challenge ``` ### Step 4 — Start the simulated robot In **one** terminal (venv activated): ```bash reachy-mini-daemon --sim ``` You should see a **MuJoCo** window. Optional: `reachy-mini-daemon --sim --scene minimal` for a table and props. **macOS:** if the normal command fails, use: ```bash mjpython -m reachy_mini.daemon.app.main --sim ``` ### Step 5 — Start the dance tutor app Open a **second** terminal (same venv, same `.env`). Simulation needs the **web UI**: ```bash reachy-mini-dance-tutor-app --gradio ``` Open the URL it prints (usually **[http://127.0.0.1:7860/](http://127.0.0.1:7860/)**) and use the browser to talk to the robot. ### If it says "timeout" or "cannot connect" The **daemon** from Step 4 must be running **before** you start the app. See [Reachy Mini troubleshooting](https://huggingface.co/docs/reachy_mini/troubleshooting). **Note:** On **Windows**, Reachy Mini support is still **experimental** — if the simulator fails, try another computer or use real hardware. --- ## Run with a **real** Reachy Mini 1. Start the daemon **without** `--sim` (follow the hardware guide from [Reachy Mini docs](https://huggingface.co/docs/reachy_mini)). 2. Same `.env` and profile as above. 3. Run: ```bash reachy-mini-dance-tutor-app --gradio ``` (Console-only mode exists, but Gradio is easier for demos.) Add a head tracker only if you installed the matching extras, for example: ```bash reachy-mini-dance-tutor-app --gradio --head-tracker mediapipe ``` --- ## Install on a Reachy Mini (wireless hardware) Follow the [Reachy Mini get started guide](https://huggingface.co/docs/reachy_mini/platforms/reachy_mini/get_started) to set up the robot first, then install this app via the Reachy Mini Control dashboard (it appears as **reachy_mini_dance_tutor_app** in the Applications list). If the app crashes on start, check the logs for a `TypeError: 'DaemonStatus' object is not subscriptable` error — this has been fixed in the current version. Make sure you have the latest code installed. --- ## Configuration (short list) | Variable | What it does | | ---------------------------- | ------------------------------------------------------------- | | `OPENAI_API_KEY` | Voice + tools with OpenAI realtime (common setup). | | `GEMINI_API_KEY` | Alternative if you use Gemini instead. | | `MODEL_NAME` | Optional: change which model speaks. | | `REACHY_MINI_CUSTOM_PROFILE` | Set to `dance_challenge` for the booth personality and tools. | | `HF_TOKEN` | May be needed for some **emotion** clips (`play_emotion`). | Full list: see `.env.example` and the **Configuration** section below for developers. --- ## Try automated checks (optional) With dev dependencies installed (`pip install -e .[dev]` or `uv sync --group dev`): ```bash pytest tests/test_dance_challenge_profile.py -q ``` That checks that the Dance Challenge tools and scoring math load correctly (no robot required). --- ## LLM tools (Dance Challenge profile) The AI only sees the tools listed in `profiles/dance_challenge/tools.txt`. Important ones for the fair: | Tool | Kid-friendly explanation | | ----------------- | -------------------------------------------------------------------------------------- | | `dance_routine` | Plays the **scripted demo** for the style the kid picked. | | `kid_dance_score` | Watches the camera for a timed window and returns **points** and **prize tier** hints. | | `play_emotion` | Makes the robot show a **happy** or proud motion. | | `camera` | Optional fun comments from a picture (scoring uses `kid_dance_score`, not this). | Other tools: `dance`, `stop_dance`, `move_head`, `head_tracking`, `do_nothing`, etc. --- ## Overview (technical) - Real-time audio conversation loop powered by the OpenAI realtime API and `fastrtc` for low-latency streaming. - Vision processing uses gpt-realtime by default when the camera tool is used, with optional local vision using SmolVLM2 via `--local-vision` and the `local_vision` extra. - Layered motion system queues primary moves (dances, emotions, goto poses, breathing) while blending speech-reactive wobble and face-tracking. - Async tool dispatch integrates robot motion, camera capture, and optional face-tracking through a Gradio web UI with live transcripts. --- ## Installation (developers) > [!IMPORTANT] > Before using this app, install [Reachy Mini's SDK](https://github.com/pollen-robotics/reachy_mini/). > **Windows support is experimental.** ### Using uv ```bash uv venv --python 3.12.1 source .venv/bin/activate # or Windows: .venv\Scripts\Activate.ps1 uv sync ``` To match the lockfile exactly: `uv sync --locked` (or `--frozen`). Optional extras: ```bash uv sync --extra reachy_mini_wireless uv sync --extra local_vision uv sync --extra yolo_vision uv sync --extra mediapipe_vision uv sync --extra all_vision uv sync --extra all_vision --group dev ``` ### Using pip ```bash python -m venv .venv source .venv/bin/activate # or Windows: .venv\Scripts\Activate.ps1 pip install -e . pip install -e .[reachy_mini_wireless] pip install -e .[local_vision] pip install -e .[yolo_vision] pip install -e .[mediapipe_vision] pip install -e .[all_vision] pip install -e .[dev] ``` Some wheels (e.g. PyTorch) are large — pick extras that match your CPU/GPU. ### Optional dependency groups | Extra | Purpose | | ---------------------- | --------------------------------- | | `reachy_mini_wireless` | Wireless Reachy Mini + GStreamer. | | `local_vision` | Local VLM (SmolVLM2). | | `yolo_vision` | `--head-tracker yolo`. | | `mediapipe_vision` | `--head-tracker mediapipe`. | | `all_vision` | All vision extras. | | `dev` | `pytest`, `ruff`, etc. | --- ## Configuration (full table) 1. Copy `.env.example` to `.env`. 2. Fill in API keys and options. | Variable | Description | | --------------------------- | --------------------------------------------- | | `OPENAI_API_KEY` | OpenAI API key for OpenAI realtime. | | `GEMINI_API_KEY` | Gemini API key if not using OpenAI. | | `GEMINI_BASE_URL` | Optional OpenAI-compatible Gemini base URL. | | `MODEL_NAME` | Override realtime model. | | `INPUT_TRANSCRIPTION_MODEL` | Optional transcription model. | | `HF_HOME` | Cache dir for Hugging Face (local vision). | | `HF_TOKEN` | Optional HF token (emotions dataset, models). | | `LOCAL_VISION_MODEL` | Model path for `--local-vision`. | --- ## Running the app (CLI reference) ```bash reachy-mini-dance-tutor-app ``` | Option | Default | Description | | --------------------------------- | ------- | ------------------------------------------------------- | | `--head-tracker {yolo,mediapipe}` | off | Face tracking (needs matching extra). | | `--no-camera` | off | No camera or face tracking. | | `--local-vision` | off | Local SmolVLM2 instead of cloud vision for image tools. | | `--gradio` | off | Web UI; **use with simulator**. | | `--debug` | off | Verbose logs. | Examples: ```bash reachy-mini-dance-tutor-app --head-tracker mediapipe reachy-mini-dance-tutor-app --local-vision reachy-mini-dance-tutor-app --no-camera reachy-mini-dance-tutor-app --gradio ``` --- ## Using custom profiles Set `REACHY_MINI_CUSTOM_PROFILE=` to load `src/reachy_dance_tutor_app/profiles//`. Each profile needs `instructions.txt` and `tools.txt`; optional Python tools in the same folder. Placeholders in `instructions.txt` like `[identities/witty_identity]` pull snippets from `src/reachy_dance_tutor_app/prompts/`. With `--gradio`, the Personality panel can switch instructions live; tool lists load at startup. --- ## Development workflow - `uv sync --group dev` or `pip install -e .[dev]` - `ruff check .` - `pytest` ## License Apache 2.0