mini_dance_tutor_app

Running

App Files Files Community

mini_dance_tutor_app / README.md

ipv6

fixed the startup issue and changed the description from conversation to dance tutor app

5020fa1 7 months ago

preview code

Raw History Blame Contribute Delete

12.7 kB

---
title: Reachy Mini Dance Tutor App
emoji: 💃
colorFrom: red
colorTo: blue
sdk: static
pinned: false
short_description: Dance with Reachy Mini!
suggested_storage: large
tags:
  - reachy_mini
  - reachy_mini_python_app
---
# Reachy Mini Dance Tutor App

**For students:** This README tells you **what this project is**, **how we built the Dance Challenge booth idea**, and **exactly what to run** on your computer to try it with a **simulator** (fake robot) or a **real robot**.

**For the science fair or business fair:** Read **[DESIGN.md](DESIGN.md)** next. It has simple diagrams and words you can use to explain the project to visitors.

Reachy Mini Dance

---

## What is this? (simple version)

Reachy Mini is a small robot that can **listen**, **talk**, **move**, and use its **camera**.

This **dance tutor app** is Python code that:

1. Connects to the robot (or simulator) through the official **Reachy Mini software**.
2. Connects to an **AI in the cloud** so the robot can have a voice chat.
3. Lets the AI call **tools** — special actions like "play a dance" or "watch the camera and give points."

Our team added a **Dance Challenge** mode: the robot hosts a mini game (demo dance → kid dances → points for the booth). How that mode works — in words you can repeat at a fair — is in **[DESIGN.md](DESIGN.md)**.

---

## How it is built (big picture)

```
You speak  →  App sends audio  →  AI thinks and replies  →  App plays sound on the robot
                              ↘  AI can call tools  →  App moves robot / uses camera
```

- **Instructions** for the AI live in a **profile** (a folder with `instructions.txt` and `tools.txt`).
- **Custom tools** for our booth live in `src/reachy_dance_tutor_app/profiles/dance_challenge/` (for example scoring).
- **Shared tools** (camera, generic dance, head move, …) live in `src/reachy_dance_tutor_app/tools/`.

The architecture diagram in the repo (`docs/assets/dance_tutor_app_arch.svg`) shows the same idea in more detail for adults.

---

## What you need before you run anything


| Need                               | Why                                                                                  |
| ---------------------------------- | ------------------------------------------------------------------------------------ |
| **Python 3.10+** (3.12 works well) | Runs the app.                                                                        |
| **This repo installed**            | `pip install -e .` or `uv sync` — see below.                                         |
| **Reachy Mini SDK + daemon**       | Robot or simulator talks through the daemon.                                         |
| **API key**                        | Copy `.env.example` → `.env` and add `OPENAI_API_KEY` (or Gemini — see table below). |
| **Dance Challenge profile**        | Set `REACHY_MINI_CUSTOM_PROFILE=dance_challenge` in `.env`.                          |


---

## Run locally and test with the **simulator** (no physical robot)

The simulator is a **3D physics window** on your computer. It follows the official guide: [Reachy Mini simulation – get started](https://huggingface.co/docs/reachy_mini/platforms/simulation/get_started).

### Step 1 — Install the project

**Option A — uv (fast):**

```bash
uv venv --python 3.12
# Windows PowerShell:
.venv\Scripts\Activate.ps1
# macOS / Linux:
# source .venv/bin/activate
uv sync
```

**Option B — pip:**

```bash
python -m venv .venv
# Windows PowerShell:
.venv\Scripts\Activate.ps1
# macOS / Linux:
# source .venv/bin/activate
pip install -e .
```

### Step 2 — Install MuJoCo support for the simulator

Still in the same environment:

```bash
pip install "reachy-mini[mujoco]"
```

### Step 3 — Configure environment

1. Copy `.env.example` to `.env`.
2. Add your `OPENAI_API_KEY` (or `GEMINI_API_KEY` if you use Gemini).
3. Set the Dance Challenge profile, for example:

```env
REACHY_MINI_CUSTOM_PROFILE=dance_challenge
```

### Step 4 — Start the simulated robot

In **one** terminal (venv activated):

```bash
reachy-mini-daemon --sim
```

You should see a **MuJoCo** window. Optional: `reachy-mini-daemon --sim --scene minimal` for a table and props.

**macOS:** if the normal command fails, use:

```bash
mjpython -m reachy_mini.daemon.app.main --sim
```

### Step 5 — Start the dance tutor app

Open a **second** terminal (same venv, same `.env`). Simulation needs the **web UI**:

```bash
reachy-mini-dance-tutor-app --gradio
```

Open the URL it prints (usually **[http://127.0.0.1:7860/](http://127.0.0.1:7860/)**) and use the browser to talk to the robot.

### If it says "timeout" or "cannot connect"

The **daemon** from Step 4 must be running **before** you start the app. See [Reachy Mini troubleshooting](https://huggingface.co/docs/reachy_mini/troubleshooting).

**Note:** On **Windows**, Reachy Mini support is still **experimental** — if the simulator fails, try another computer or use real hardware.

---

## Run with a **real** Reachy Mini

1. Start the daemon **without** `--sim` (follow the hardware guide from [Reachy Mini docs](https://huggingface.co/docs/reachy_mini)).
2. Same `.env` and profile as above.
3. Run:

```bash
reachy-mini-dance-tutor-app --gradio
```

(Console-only mode exists, but Gradio is easier for demos.)

Add a head tracker only if you installed the matching extras, for example:

```bash
reachy-mini-dance-tutor-app --gradio --head-tracker mediapipe
```

---

## Install on a Reachy Mini (wireless hardware)

Follow the [Reachy Mini get started guide](https://huggingface.co/docs/reachy_mini/platforms/reachy_mini/get_started) to set up the robot first, then install this app via the Reachy Mini Control dashboard (it appears as **reachy_mini_dance_tutor_app** in the Applications list).

If the app crashes on start, check the logs for a `TypeError: 'DaemonStatus' object is not subscriptable` error — this has been fixed in the current version. Make sure you have the latest code installed.

---

## Configuration (short list)


| Variable                     | What it does                                                  |
| ---------------------------- | ------------------------------------------------------------- |
| `OPENAI_API_KEY`             | Voice + tools with OpenAI realtime (common setup).            |
| `GEMINI_API_KEY`             | Alternative if you use Gemini instead.                        |
| `MODEL_NAME`                 | Optional: change which model speaks.                          |
| `REACHY_MINI_CUSTOM_PROFILE` | Set to `dance_challenge` for the booth personality and tools. |
| `HF_TOKEN`                   | May be needed for some **emotion** clips (`play_emotion`).    |


Full list: see `.env.example` and the **Configuration** section below for developers.

---

## Try automated checks (optional)

With dev dependencies installed (`pip install -e .[dev]` or `uv sync --group dev`):

```bash
pytest tests/test_dance_challenge_profile.py -q
```

That checks that the Dance Challenge tools and scoring math load correctly (no robot required).

---

## LLM tools (Dance Challenge profile)

The AI only sees the tools listed in `profiles/dance_challenge/tools.txt`. Important ones for the fair:


| Tool              | Kid-friendly explanation                                                               |
| ----------------- | -------------------------------------------------------------------------------------- |
| `dance_routine`   | Plays the **scripted demo** for the style the kid picked.                              |
| `kid_dance_score` | Watches the camera for a timed window and returns **points** and **prize tier** hints. |
| `play_emotion`    | Makes the robot show a **happy** or proud motion.                                      |
| `camera`          | Optional fun comments from a picture (scoring uses `kid_dance_score`, not this).       |


Other tools: `dance`, `stop_dance`, `move_head`, `head_tracking`, `do_nothing`, etc.

---

## Overview (technical)

- Real-time audio conversation loop powered by the OpenAI realtime API and `fastrtc` for low-latency streaming.
- Vision processing uses gpt-realtime by default when the camera tool is used, with optional local vision using SmolVLM2 via `--local-vision` and the `local_vision` extra.
- Layered motion system queues primary moves (dances, emotions, goto poses, breathing) while blending speech-reactive wobble and face-tracking.
- Async tool dispatch integrates robot motion, camera capture, and optional face-tracking through a Gradio web UI with live transcripts.



---

## Installation (developers)

> [!IMPORTANT]
> Before using this app, install [Reachy Mini's SDK](https://github.com/pollen-robotics/reachy_mini/).  
> **Windows support is experimental.**

### Using uv

```bash
uv venv --python 3.12.1
source .venv/bin/activate   # or Windows: .venv\Scripts\Activate.ps1
uv sync
```

To match the lockfile exactly: `uv sync --locked` (or `--frozen`).

Optional extras:

```bash
uv sync --extra reachy_mini_wireless
uv sync --extra local_vision
uv sync --extra yolo_vision
uv sync --extra mediapipe_vision
uv sync --extra all_vision
uv sync --extra all_vision --group dev
```

### Using pip

```bash
python -m venv .venv
source .venv/bin/activate   # or Windows: .venv\Scripts\Activate.ps1
pip install -e .
pip install -e .[reachy_mini_wireless]
pip install -e .[local_vision]
pip install -e .[yolo_vision]
pip install -e .[mediapipe_vision]
pip install -e .[all_vision]
pip install -e .[dev]
```

Some wheels (e.g. PyTorch) are large — pick extras that match your CPU/GPU.

### Optional dependency groups


| Extra                  | Purpose                           |
| ---------------------- | --------------------------------- |
| `reachy_mini_wireless` | Wireless Reachy Mini + GStreamer. |
| `local_vision`         | Local VLM (SmolVLM2).             |
| `yolo_vision`          | `--head-tracker yolo`.            |
| `mediapipe_vision`     | `--head-tracker mediapipe`.       |
| `all_vision`           | All vision extras.                |
| `dev`                  | `pytest`, `ruff`, etc.            |


---

## Configuration (full table)

1. Copy `.env.example` to `.env`.
2. Fill in API keys and options.


| Variable                    | Description                                   |
| --------------------------- | --------------------------------------------- |
| `OPENAI_API_KEY`            | OpenAI API key for OpenAI realtime.           |
| `GEMINI_API_KEY`            | Gemini API key if not using OpenAI.           |
| `GEMINI_BASE_URL`           | Optional OpenAI-compatible Gemini base URL.   |
| `MODEL_NAME`                | Override realtime model.                      |
| `INPUT_TRANSCRIPTION_MODEL` | Optional transcription model.                 |
| `HF_HOME`                   | Cache dir for Hugging Face (local vision).    |
| `HF_TOKEN`                  | Optional HF token (emotions dataset, models). |
| `LOCAL_VISION_MODEL`        | Model path for `--local-vision`.              |


---

## Running the app (CLI reference)

```bash
reachy-mini-dance-tutor-app
```


| Option                            | Default | Description                                             |
| --------------------------------- | ------- | ------------------------------------------------------- |
| `--head-tracker {yolo,mediapipe}` | off     | Face tracking (needs matching extra).                   |
| `--no-camera`                     | off     | No camera or face tracking.                             |
| `--local-vision`                  | off     | Local SmolVLM2 instead of cloud vision for image tools. |
| `--gradio`                        | off     | Web UI; **use with simulator**.                         |
| `--debug`                         | off     | Verbose logs.                                           |


Examples:

```bash
reachy-mini-dance-tutor-app --head-tracker mediapipe
reachy-mini-dance-tutor-app --local-vision
reachy-mini-dance-tutor-app --no-camera
reachy-mini-dance-tutor-app --gradio
```

---

## Using custom profiles

Set `REACHY_MINI_CUSTOM_PROFILE=<name>` to load `src/reachy_dance_tutor_app/profiles/<name>/`. Each profile needs `instructions.txt` and `tools.txt`; optional Python tools in the same folder.

Placeholders in `instructions.txt` like `[identities/witty_identity]` pull snippets from `src/reachy_dance_tutor_app/prompts/`.

With `--gradio`, the Personality panel can switch instructions live; tool lists load at startup.

---

## Development workflow

- `uv sync --group dev` or `pip install -e .[dev]`
- `ruff check .`
- `pytest`

## License

Apache 2.0

	---
	title: Reachy Mini Dance Tutor App
	emoji: 💃
	colorFrom: red
	colorTo: blue
	sdk: static
	pinned: false
	short_description: Dance with Reachy Mini!
	suggested_storage: large
	tags:
	- reachy_mini
	- reachy_mini_python_app
	---

	# Reachy Mini Dance Tutor App

	For students: This README tells you what this project is, how we built the Dance Challenge booth idea, and exactly what to run on your computer to try it with a simulator (fake robot) or a real robot.

	For the science fair or business fair: Read [DESIGN.md](DESIGN.md) next. It has simple diagrams and words you can use to explain the project to visitors.

	Reachy Mini Dance

	---

	## What is this? (simple version)

	Reachy Mini is a small robot that can listen, talk, move, and use its camera.

	This dance tutor app is Python code that:

	1. Connects to the robot (or simulator) through the official Reachy Mini software.
	2. Connects to an AI in the cloud so the robot can have a voice chat.
	3. Lets the AI call tools — special actions like "play a dance" or "watch the camera and give points."

	Our team added a Dance Challenge mode: the robot hosts a mini game (demo dance → kid dances → points for the booth). How that mode works — in words you can repeat at a fair — is in [DESIGN.md](DESIGN.md).

	---

	## How it is built (big picture)

	```
	You speak → App sends audio → AI thinks and replies → App plays sound on the robot
	↘ AI can call tools → App moves robot / uses camera
	```

	- Instructions for the AI live in a profile (a folder with `instructions.txt` and `tools.txt`).
	- Custom tools for our booth live in `src/reachy_dance_tutor_app/profiles/dance_challenge/` (for example scoring).
	- Shared tools (camera, generic dance, head move, …) live in `src/reachy_dance_tutor_app/tools/`.

	The architecture diagram in the repo (`docs/assets/dance_tutor_app_arch.svg`) shows the same idea in more detail for adults.

	---

	## What you need before you run anything


	\| Need \| Why \|
	\| ---------------------------------- \| ------------------------------------------------------------------------------------ \|
	\| Python 3.10+ (3.12 works well) \| Runs the app. \|
	\| This repo installed \| `pip install -e .` or `uv sync` — see below. \|
	\| Reachy Mini SDK + daemon \| Robot or simulator talks through the daemon. \|
	\| API key \| Copy `.env.example` → `.env` and add `OPENAI_API_KEY` (or Gemini — see table below). \|
	\| Dance Challenge profile \| Set `REACHY_MINI_CUSTOM_PROFILE=dance_challenge` in `.env`. \|


	---

	## Run locally and test with the simulator (no physical robot)

	The simulator is a 3D physics window on your computer. It follows the official guide: [Reachy Mini simulation – get started](https://huggingface.co/docs/reachy_mini/platforms/simulation/get_started).

	### Step 1 — Install the project

	Option A — uv (fast):

	```bash
	uv venv --python 3.12
	# Windows PowerShell:
	.venv\Scripts\Activate.ps1
	# macOS / Linux:
	# source .venv/bin/activate

	uv sync
	```

	Option B — pip:

	```bash
	python -m venv .venv
	# Windows PowerShell:
	.venv\Scripts\Activate.ps1
	# macOS / Linux:
	# source .venv/bin/activate

	pip install -e .
	```

	### Step 2 — Install MuJoCo support for the simulator

	Still in the same environment:

	```bash
	pip install "reachy-mini[mujoco]"
	```

	### Step 3 — Configure environment

	1. Copy `.env.example` to `.env`.
	2. Add your `OPENAI_API_KEY` (or `GEMINI_API_KEY` if you use Gemini).
	3. Set the Dance Challenge profile, for example:

	```env
	REACHY_MINI_CUSTOM_PROFILE=dance_challenge
	```

	### Step 4 — Start the simulated robot

	In one terminal (venv activated):

	```bash
	reachy-mini-daemon --sim
	```

	You should see a MuJoCo window. Optional: `reachy-mini-daemon --sim --scene minimal` for a table and props.

	macOS: if the normal command fails, use:

	```bash
	mjpython -m reachy_mini.daemon.app.main --sim
	```

	### Step 5 — Start the dance tutor app

	Open a second terminal (same venv, same `.env`). Simulation needs the web UI:

	```bash
	reachy-mini-dance-tutor-app --gradio
	```

	Open the URL it prints (usually [http://127.0.0.1:7860/](http://127.0.0.1:7860/)) and use the browser to talk to the robot.

	### If it says "timeout" or "cannot connect"

	The daemon from Step 4 must be running before you start the app. See [Reachy Mini troubleshooting](https://huggingface.co/docs/reachy_mini/troubleshooting).

	Note: On Windows, Reachy Mini support is still experimental — if the simulator fails, try another computer or use real hardware.

	---

	## Run with a real Reachy Mini

	1. Start the daemon without `--sim` (follow the hardware guide from [Reachy Mini docs](https://huggingface.co/docs/reachy_mini)).
	2. Same `.env` and profile as above.
	3. Run:

	```bash
	reachy-mini-dance-tutor-app --gradio
	```

	(Console-only mode exists, but Gradio is easier for demos.)

	Add a head tracker only if you installed the matching extras, for example:

	```bash
	reachy-mini-dance-tutor-app --gradio --head-tracker mediapipe
	```

	---

	## Install on a Reachy Mini (wireless hardware)

	Follow the [Reachy Mini get started guide](https://huggingface.co/docs/reachy_mini/platforms/reachy_mini/get_started) to set up the robot first, then install this app via the Reachy Mini Control dashboard (it appears as reachy_mini_dance_tutor_app in the Applications list).

	If the app crashes on start, check the logs for a `TypeError: 'DaemonStatus' object is not subscriptable` error — this has been fixed in the current version. Make sure you have the latest code installed.

	---

	## Configuration (short list)


	\| Variable \| What it does \|
	\| ---------------------------- \| ------------------------------------------------------------- \|
	\| `OPENAI_API_KEY` \| Voice + tools with OpenAI realtime (common setup). \|
	\| `GEMINI_API_KEY` \| Alternative if you use Gemini instead. \|
	\| `MODEL_NAME` \| Optional: change which model speaks. \|
	\| `REACHY_MINI_CUSTOM_PROFILE` \| Set to `dance_challenge` for the booth personality and tools. \|
	\| `HF_TOKEN` \| May be needed for some emotion clips (`play_emotion`). \|


	Full list: see `.env.example` and the Configuration section below for developers.

	---

	## Try automated checks (optional)

	With dev dependencies installed (`pip install -e .[dev]` or `uv sync --group dev`):

	```bash
	pytest tests/test_dance_challenge_profile.py -q
	```

	That checks that the Dance Challenge tools and scoring math load correctly (no robot required).

	---

	## LLM tools (Dance Challenge profile)

	The AI only sees the tools listed in `profiles/dance_challenge/tools.txt`. Important ones for the fair:


	\| Tool \| Kid-friendly explanation \|
	\| ----------------- \| -------------------------------------------------------------------------------------- \|
	\| `dance_routine` \| Plays the scripted demo for the style the kid picked. \|
	\| `kid_dance_score` \| Watches the camera for a timed window and returns points and prize tier hints. \|
	\| `play_emotion` \| Makes the robot show a happy or proud motion. \|
	\| `camera` \| Optional fun comments from a picture (scoring uses `kid_dance_score`, not this). \|


	Other tools: `dance`, `stop_dance`, `move_head`, `head_tracking`, `do_nothing`, etc.

	---

	## Overview (technical)

	- Real-time audio conversation loop powered by the OpenAI realtime API and `fastrtc` for low-latency streaming.
	- Vision processing uses gpt-realtime by default when the camera tool is used, with optional local vision using SmolVLM2 via `--local-vision` and the `local_vision` extra.
	- Layered motion system queues primary moves (dances, emotions, goto poses, breathing) while blending speech-reactive wobble and face-tracking.
	- Async tool dispatch integrates robot motion, camera capture, and optional face-tracking through a Gradio web UI with live transcripts.



	---

	## Installation (developers)

	> [!IMPORTANT]
	> Before using this app, install [Reachy Mini's SDK](https://github.com/pollen-robotics/reachy_mini/).
	> Windows support is experimental.

	### Using uv

	```bash
	uv venv --python 3.12.1
	source .venv/bin/activate # or Windows: .venv\Scripts\Activate.ps1
	uv sync
	```

	To match the lockfile exactly: `uv sync --locked` (or `--frozen`).

	Optional extras:

	```bash
	uv sync --extra reachy_mini_wireless
	uv sync --extra local_vision
	uv sync --extra yolo_vision
	uv sync --extra mediapipe_vision
	uv sync --extra all_vision
	uv sync --extra all_vision --group dev
	```

	### Using pip

	```bash
	python -m venv .venv
	source .venv/bin/activate # or Windows: .venv\Scripts\Activate.ps1
	pip install -e .
	pip install -e .[reachy_mini_wireless]
	pip install -e .[local_vision]
	pip install -e .[yolo_vision]
	pip install -e .[mediapipe_vision]
	pip install -e .[all_vision]
	pip install -e .[dev]
	```

	Some wheels (e.g. PyTorch) are large — pick extras that match your CPU/GPU.

	### Optional dependency groups


	\| Extra \| Purpose \|
	\| ---------------------- \| --------------------------------- \|
	\| `reachy_mini_wireless` \| Wireless Reachy Mini + GStreamer. \|
	\| `local_vision` \| Local VLM (SmolVLM2). \|
	\| `yolo_vision` \| `--head-tracker yolo`. \|
	\| `mediapipe_vision` \| `--head-tracker mediapipe`. \|
	\| `all_vision` \| All vision extras. \|
	\| `dev` \| `pytest`, `ruff`, etc. \|


	---

	## Configuration (full table)

	1. Copy `.env.example` to `.env`.
	2. Fill in API keys and options.


	\| Variable \| Description \|
	\| --------------------------- \| --------------------------------------------- \|
	\| `OPENAI_API_KEY` \| OpenAI API key for OpenAI realtime. \|
	\| `GEMINI_API_KEY` \| Gemini API key if not using OpenAI. \|
	\| `GEMINI_BASE_URL` \| Optional OpenAI-compatible Gemini base URL. \|
	\| `MODEL_NAME` \| Override realtime model. \|
	\| `INPUT_TRANSCRIPTION_MODEL` \| Optional transcription model. \|
	\| `HF_HOME` \| Cache dir for Hugging Face (local vision). \|
	\| `HF_TOKEN` \| Optional HF token (emotions dataset, models). \|
	\| `LOCAL_VISION_MODEL` \| Model path for `--local-vision`. \|


	---

	## Running the app (CLI reference)

	```bash
	reachy-mini-dance-tutor-app
	```


	\| Option \| Default \| Description \|
	\| --------------------------------- \| ------- \| ------------------------------------------------------- \|
	\| `--head-tracker {yolo,mediapipe}` \| off \| Face tracking (needs matching extra). \|
	\| `--no-camera` \| off \| No camera or face tracking. \|
	\| `--local-vision` \| off \| Local SmolVLM2 instead of cloud vision for image tools. \|
	\| `--gradio` \| off \| Web UI; use with simulator. \|
	\| `--debug` \| off \| Verbose logs. \|


	Examples:

	```bash
	reachy-mini-dance-tutor-app --head-tracker mediapipe
	reachy-mini-dance-tutor-app --local-vision
	reachy-mini-dance-tutor-app --no-camera
	reachy-mini-dance-tutor-app --gradio
	```

	---

	## Using custom profiles

	Set `REACHY_MINI_CUSTOM_PROFILE=<name>` to load `src/reachy_dance_tutor_app/profiles/<name>/`. Each profile needs `instructions.txt` and `tools.txt`; optional Python tools in the same folder.

	Placeholders in `instructions.txt` like `[identities/witty_identity]` pull snippets from `src/reachy_dance_tutor_app/prompts/`.

	With `--gradio`, the Personality panel can switch instructions live; tool lists load at startup.

	---

	## Development workflow

	- `uv sync --group dev` or `pip install -e .[dev]`
	- `ruff check .`
	- `pytest`

	## License

	Apache 2.0