Blame for README.md

---
title: Reachy Mini Dance Tutor App
emoji: 💃
colorFrom: red
colorTo: blue
sdk: static
pinned: false
short_description: Dance with Reachy Mini!
suggested_storage: large
tags:
  - reachy_mini
  - reachy_mini_python_app
---

# Reachy Mini Dance Tutor App

**For students:** This README tells you **what this project is**, **how we built the Dance Challenge booth idea**, and **exactly what to run** on your computer to try it with a **simulator** (fake robot) or a **real robot**.

**For the science fair or business fair:** Read **[DESIGN.md](DESIGN.md)** next. It has simple diagrams and words you can use to explain the project to visitors.

Reachy Mini Dance

---

## What is this? (simple version)

Reachy Mini is a small robot that can **listen**, **talk**, **move**, and use its **camera**.

This **dance tutor app** is Python code that:

1. Connects to the robot (or simulator) through the official **Reachy Mini software**.
2. Connects to an **AI in the cloud** so the robot can have a voice chat.
3. Lets the AI call **tools** — special actions like "play a dance" or "watch the camera and give points."

Our team added a **Dance Challenge** mode: the robot hosts a mini game (demo dance → kid dances → points for the booth). How that mode works — in words you can repeat at a fair — is in **[DESIGN.md](DESIGN.md)**.

---

## How it is built (big picture)

```
You speak  →  App sends audio  →  AI thinks and replies  →  App plays sound on the robot
                              ↘  AI can call tools  →  App moves robot / uses camera
```

- **Instructions** for the AI live in a **profile** (a folder with `instructions.txt` and `tools.txt`).
- **Custom tools** for our booth live in `src/reachy_dance_tutor_app/profiles/dance_challenge/` (for example scoring).
- **Shared tools** (camera, generic dance, head move, …) live in `src/reachy_dance_tutor_app/tools/`.

The architecture diagram in the repo (`docs/assets/dance_tutor_app_arch.svg`) shows the same idea in more detail for adults.

---

## What you need before you run anything


| Need                               | Why                                                                                  |
| ---------------------------------- | ------------------------------------------------------------------------------------ |
| **Python 3.10+** (3.12 works well) | Runs the app.                                                                        |
| **This repo installed**            | `pip install -e .` or `uv sync` — see below.                                         |
| **Reachy Mini SDK + daemon**       | Robot or simulator talks through the daemon.                                         |
| **API key**                        | Copy `.env.example` → `.env` and add `OPENAI_API_KEY` (or Gemini — see table below). |
| **Dance Challenge profile**        | Set `REACHY_MINI_CUSTOM_PROFILE=dance_challenge` in `.env`.                          |


---

## Run locally and test with the **simulator** (no physical robot)

The simulator is a **3D physics window** on your computer. It follows the official guide: [Reachy Mini simulation – get started](https://huggingface.co/docs/reachy_mini/platforms/simulation/get_started).

### Step 1 — Install the project

**Option A — uv (fast):**

```bash
uv venv --python 3.12
# Windows PowerShell:
.venv\Scripts\Activate.ps1
# macOS / Linux:
# source .venv/bin/activate

uv sync
```

**Option B — pip:**

```bash
python -m venv .venv
# Windows PowerShell:
.venv\Scripts\Activate.ps1
# macOS / Linux:
# source .venv/bin/activate

pip install -e .
```

### Step 2 — Install MuJoCo support for the simulator

Still in the same environment:

```bash
pip install "reachy-mini[mujoco]"
```

### Step 3 — Configure environment

1. Copy `.env.example` to `.env`.
2. Add your `OPENAI_API_KEY` (or `GEMINI_API_KEY` if you use Gemini).
3. Set the Dance Challenge profile, for example:

```env
REACHY_MINI_CUSTOM_PROFILE=dance_challenge
```

### Step 4 — Start the simulated robot

In **one** terminal (venv activated):

```bash
reachy-mini-daemon --sim
```

You should see a **MuJoCo** window. Optional: `reachy-mini-daemon --sim --scene minimal` for a table and props.

**macOS:** if the normal command fails, use:

```bash
mjpython -m reachy_mini.daemon.app.main --sim
```

### Step 5 — Start the dance tutor app

Open a **second** terminal (same venv, same `.env`). Simulation needs the **web UI**:

```bash
reachy-mini-dance-tutor-app --gradio
```

Open the URL it prints (usually **[http://127.0.0.1:7860/](http://127.0.0.1:7860/)**) and use the browser to talk to the robot.

### If it says "timeout" or "cannot connect"

The **daemon** from Step 4 must be running **before** you start the app. See [Reachy Mini troubleshooting](https://huggingface.co/docs/reachy_mini/troubleshooting).

**Note:** On **Windows**, Reachy Mini support is still **experimental** — if the simulator fails, try another computer or use real hardware.

---

## Run with a **real** Reachy Mini

1. Start the daemon **without** `--sim` (follow the hardware guide from [Reachy Mini docs](https://huggingface.co/docs/reachy_mini)).
2. Same `.env` and profile as above.
3. Run:

```bash
reachy-mini-dance-tutor-app --gradio
```

(Console-only mode exists, but Gradio is easier for demos.)

Add a head tracker only if you installed the matching extras, for example:

```bash
reachy-mini-dance-tutor-app --gradio --head-tracker mediapipe
```

---

## Install on a Reachy Mini (wireless hardware)

Follow the [Reachy Mini get started guide](https://huggingface.co/docs/reachy_mini/platforms/reachy_mini/get_started) to set up the robot first, then install this app via the Reachy Mini Control dashboard (it appears as **reachy_mini_dance_tutor_app** in the Applications list).

If the app crashes on start, check the logs for a `TypeError: 'DaemonStatus' object is not subscriptable` error — this has been fixed in the current version. Make sure you have the latest code installed.

---

## Configuration (short list)


| Variable                     | What it does                                                  |
| ---------------------------- | ------------------------------------------------------------- |
| `OPENAI_API_KEY`             | Voice + tools with OpenAI realtime (common setup).            |
| `GEMINI_API_KEY`             | Alternative if you use Gemini instead.                        |
| `MODEL_NAME`                 | Optional: change which model speaks.                          |
| `REACHY_MINI_CUSTOM_PROFILE` | Set to `dance_challenge` for the booth personality and tools. |
| `HF_TOKEN`                   | May be needed for some **emotion** clips (`play_emotion`).    |


Full list: see `.env.example` and the **Configuration** section below for developers.

---

## Try automated checks (optional)

With dev dependencies installed (`pip install -e .[dev]` or `uv sync --group dev`):

```bash
pytest tests/test_dance_challenge_profile.py -q
```

That checks that the Dance Challenge tools and scoring math load correctly (no robot required).

---

## LLM tools (Dance Challenge profile)

The AI only sees the tools listed in `profiles/dance_challenge/tools.txt`. Important ones for the fair:


| Tool              | Kid-friendly explanation                                                               |
| ----------------- | -------------------------------------------------------------------------------------- |
| `dance_routine`   | Plays the **scripted demo** for the style the kid picked.                              |
| `kid_dance_score` | Watches the camera for a timed window and returns **points** and **prize tier** hints. |
| `play_emotion`    | Makes the robot show a **happy** or proud motion.                                      |
| `camera`          | Optional fun comments from a picture (scoring uses `kid_dance_score`, not this).       |


Other tools: `dance`, `stop_dance`, `move_head`, `head_tracking`, `do_nothing`, etc.

---

## Overview (technical)

- Real-time audio conversation loop powered by the OpenAI realtime API and `fastrtc` for low-latency streaming.
- Vision processing uses gpt-realtime by default when the camera tool is used, with optional local vision using SmolVLM2 via `--local-vision` and the `local_vision` extra.
- Layered motion system queues primary moves (dances, emotions, goto poses, breathing) while blending speech-reactive wobble and face-tracking.
- Async tool dispatch integrates robot motion, camera capture, and optional face-tracking through a Gradio web UI with live transcripts.



---

## Installation (developers)

> [!IMPORTANT]
> Before using this app, install [Reachy Mini's SDK](https://github.com/pollen-robotics/reachy_mini/).  
> **Windows support is experimental.**

### Using uv

```bash
uv venv --python 3.12.1
source .venv/bin/activate   # or Windows: .venv\Scripts\Activate.ps1
uv sync
```

To match the lockfile exactly: `uv sync --locked` (or `--frozen`).

Optional extras:

```bash
uv sync --extra reachy_mini_wireless
uv sync --extra local_vision
uv sync --extra yolo_vision
uv sync --extra mediapipe_vision
uv sync --extra all_vision
uv sync --extra all_vision --group dev
```

### Using pip

```bash
python -m venv .venv
source .venv/bin/activate   # or Windows: .venv\Scripts\Activate.ps1
pip install -e .
pip install -e .[reachy_mini_wireless]
pip install -e .[local_vision]
pip install -e .[yolo_vision]
pip install -e .[mediapipe_vision]
pip install -e .[all_vision]
pip install -e .[dev]
```

Some wheels (e.g. PyTorch) are large — pick extras that match your CPU/GPU.

### Optional dependency groups


| Extra                  | Purpose                           |
| ---------------------- | --------------------------------- |
| `reachy_mini_wireless` | Wireless Reachy Mini + GStreamer. |
| `local_vision`         | Local VLM (SmolVLM2).             |
| `yolo_vision`          | `--head-tracker yolo`.            |
| `mediapipe_vision`     | `--head-tracker mediapipe`.       |
| `all_vision`           | All vision extras.                |
| `dev`                  | `pytest`, `ruff`, etc.            |


---

## Configuration (full table)

1. Copy `.env.example` to `.env`.
2. Fill in API keys and options.


| Variable                    | Description                                   |
| --------------------------- | --------------------------------------------- |
| `OPENAI_API_KEY`            | OpenAI API key for OpenAI realtime.           |
| `GEMINI_API_KEY`            | Gemini API key if not using OpenAI.           |
| `GEMINI_BASE_URL`           | Optional OpenAI-compatible Gemini base URL.   |
| `MODEL_NAME`                | Override realtime model.                      |
| `INPUT_TRANSCRIPTION_MODEL` | Optional transcription model.                 |
| `HF_HOME`                   | Cache dir for Hugging Face (local vision).    |
| `HF_TOKEN`                  | Optional HF token (emotions dataset, models). |
| `LOCAL_VISION_MODEL`        | Model path for `--local-vision`.              |


---

## Running the app (CLI reference)

```bash
reachy-mini-dance-tutor-app
```


| Option                            | Default | Description                                             |
| --------------------------------- | ------- | ------------------------------------------------------- |
| `--head-tracker {yolo,mediapipe}` | off     | Face tracking (needs matching extra).                   |
| `--no-camera`                     | off     | No camera or face tracking.                             |
| `--local-vision`                  | off     | Local SmolVLM2 instead of cloud vision for image tools. |
| `--gradio`                        | off     | Web UI; **use with simulator**.                         |
| `--debug`                         | off     | Verbose logs.                                           |


Examples:

```bash
reachy-mini-dance-tutor-app --head-tracker mediapipe
reachy-mini-dance-tutor-app --local-vision
reachy-mini-dance-tutor-app --no-camera
reachy-mini-dance-tutor-app --gradio
```

---

## Using custom profiles

Set `REACHY_MINI_CUSTOM_PROFILE=<name>` to load `src/reachy_dance_tutor_app/profiles/<name>/`. Each profile needs `instructions.txt` and `tools.txt`; optional Python tools in the same folder.

Placeholders in `instructions.txt` like `[identities/witty_identity]` pull snippets from `src/reachy_dance_tutor_app/prompts/`.

With `--gradio`, the Personality panel can switch instructions live; tool lists load at startup.

---

## Development workflow

- `uv sync --group dev` or `pip install -e .[dev]`
- `ruff check .`
- `pytest`

## License

Apache 2.0