stephenslee commited on
Commit
8a652b8
·
verified ·
1 Parent(s): ec77614

Publish Peachy v0.1.0 for Reachy Mini Control

Browse files
This view is limited to 50 files because it contains too many changes.   See raw diff
Files changed (50) hide show
  1. .env.example +36 -0
  2. .gitattributes +1 -0
  3. LICENSE +201 -0
  4. MANIFEST.in +10 -0
  5. README.md +405 -4
  6. UPSTREAM.md +24 -0
  7. docs/assets/conversation_app_arch.svg +0 -0
  8. docs/assets/reachy_mini_dance.gif +3 -0
  9. docs/legacy-regression-bootstrap.md +91 -0
  10. docs/openai-realtime-smoke.md +86 -0
  11. index.html +120 -14
  12. pyproject.toml +79 -0
  13. src/peachy/.env.example +36 -0
  14. src/peachy/__init__.py +2 -0
  15. src/peachy/app_lifecycle.py +85 -0
  16. src/peachy/audio/__init__.py +1 -0
  17. src/peachy/audio/resampling.py +55 -0
  18. src/peachy/audio/startup_config.py +66 -0
  19. src/peachy/config.py +489 -0
  20. src/peachy/console.py +1005 -0
  21. src/peachy/conversation_handler.py +133 -0
  22. src/peachy/dance_emotion_moves.py +156 -0
  23. src/peachy/huggingface_realtime.py +1133 -0
  24. src/peachy/idle_policy.py +140 -0
  25. src/peachy/main.py +372 -0
  26. src/peachy/mcp_client.py +391 -0
  27. src/peachy/memory.py +213 -0
  28. src/peachy/moves.py +767 -0
  29. src/peachy/openai_realtime.py +1178 -0
  30. src/peachy/personality.py +157 -0
  31. src/peachy/personality_routes.py +313 -0
  32. src/peachy/profiles/bored_teenager/instructions.txt +1 -0
  33. src/peachy/profiles/bored_teenager/tools.txt +10 -0
  34. src/peachy/profiles/captain_circuit/instructions.txt +1 -0
  35. src/peachy/profiles/captain_circuit/tools.txt +10 -0
  36. src/peachy/profiles/chess_coach/instructions.txt +1 -0
  37. src/peachy/profiles/chess_coach/tools.txt +10 -0
  38. src/peachy/profiles/cosmic_kitchen/instructions.txt +49 -0
  39. src/peachy/profiles/cosmic_kitchen/tools.txt +10 -0
  40. src/peachy/profiles/default/instructions.txt +48 -0
  41. src/peachy/profiles/default/sweep_look.py +131 -0
  42. src/peachy/profiles/default/tools.txt +15 -0
  43. src/peachy/profiles/hype_bot/instructions.txt +1 -0
  44. src/peachy/profiles/hype_bot/tools.txt +10 -0
  45. src/peachy/profiles/mad_scientist_assistant/instructions.txt +1 -0
  46. src/peachy/profiles/mad_scientist_assistant/tools.txt +10 -0
  47. src/peachy/profiles/mars_rover/instructions.txt +23 -0
  48. src/peachy/profiles/mars_rover/tools.txt +10 -0
  49. src/peachy/profiles/nature_documentarian/instructions.txt +1 -0
  50. src/peachy/profiles/nature_documentarian/tools.txt +10 -0
.env.example ADDED
@@ -0,0 +1,36 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Optional realtime input transcription language.
2
+ # Defaults to "en". Set to a backend-supported code such as "zh" for Chinese.
3
+ # REALTIME_TRANSCRIPTION_LANGUAGE="en"
4
+
5
+ PEACHY_REALTIME_PROVIDER=openai
6
+ OPENAI_API_KEY=
7
+ OPENAI_REALTIME_MODEL=gpt-realtime
8
+
9
+ # Hugging Face connection mode: "deployed" or "local".
10
+ # Deployed mode uses the built-in Hugging Face server.
11
+ # Defaults to "deployed" when unset.
12
+ # HF_REALTIME_CONNECTION_MODE="deployed"
13
+
14
+ # Direct Hugging Face realtime endpoint for local/LAN backends.
15
+ # Accepts either the base URL (`ws://127.0.0.1:8765/v1`) or the full realtime URL
16
+ # (`ws://127.0.0.1:8765/v1/realtime`). Used when HF_REALTIME_CONNECTION_MODE=local.
17
+ # HF_REALTIME_WS_URL="ws://127.0.0.1:8765/v1/realtime"
18
+
19
+ # Hugging Face token for accessing datasets/models
20
+ HF_TOKEN=
21
+
22
+ # Optional built-in profile selector. Use a folder name from profiles/.
23
+ # If unset, the app uses the default profile unless startup settings were saved from the UI.
24
+ # REACHY_MINI_CUSTOM_PROFILE=mars_rover
25
+
26
+ # Optional external profile/tool directories
27
+ # REACHY_MINI_EXTERNAL_PROFILES_DIRECTORY=external_content/external_profiles
28
+ # REACHY_MINI_EXTERNAL_TOOLS_DIRECTORY=external_content/external_tools
29
+
30
+ # Optional: discover and auto-load all tools found in REACHY_MINI_EXTERNAL_TOOLS_DIRECTORY,
31
+ # even if they are not listed in the selected profile's tools.txt.
32
+ # This is convenient for downloaded tools used with built-in/default profiles.
33
+ # AUTOLOAD_EXTERNAL_TOOLS=1
34
+
35
+ # Minutes of inactivity before Reachy goes to sleep and the app stops. Default 1440 (one day), 0 disables.
36
+ # REACHY_MINI_APP_TIMEOUT_MINUTES=1440
.gitattributes CHANGED
@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ docs/assets/reachy_mini_dance.gif filter=lfs diff=lfs merge=lfs -text
LICENSE ADDED
@@ -0,0 +1,201 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Apache License
2
+ Version 2.0, January 2004
3
+ http://www.apache.org/licenses/
4
+
5
+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
6
+
7
+ 1. Definitions.
8
+
9
+ "License" shall mean the terms and conditions for use, reproduction,
10
+ and distribution as defined by Sections 1 through 9 of this document.
11
+
12
+ "Licensor" shall mean the copyright owner or entity authorized by
13
+ the copyright owner that is granting the License.
14
+
15
+ "Legal Entity" shall mean the union of the acting entity and all
16
+ other entities that control, are controlled by, or are under common
17
+ control with that entity. For the purposes of this definition,
18
+ "control" means (i) the power, direct or indirect, to cause the
19
+ direction or management of such entity, whether by contract or
20
+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
21
+ outstanding shares, or (iii) beneficial ownership of such entity.
22
+
23
+ "You" (or "Your") shall mean an individual or Legal Entity
24
+ exercising permissions granted by this License.
25
+
26
+ "Source" form shall mean the preferred form for making modifications,
27
+ including but not limited to software source code, documentation
28
+ source, and configuration files.
29
+
30
+ "Object" form shall mean any form resulting from mechanical
31
+ transformation or translation of a Source form, including but
32
+ not limited to compiled object code, generated documentation,
33
+ and conversions to other media types.
34
+
35
+ "Work" shall mean the work of authorship, whether in Source or
36
+ Object form, made available under the License, as indicated by a
37
+ copyright notice that is included in or attached to the work
38
+ (an example is provided in the Appendix below).
39
+
40
+ "Derivative Works" shall mean any work, whether in Source or Object
41
+ form, that is based on (or derived from) the Work and for which the
42
+ editorial revisions, annotations, elaborations, or other modifications
43
+ represent, as a whole, an original work of authorship. For the purposes
44
+ of this License, Derivative Works shall not include works that remain
45
+ separable from, or merely link (or bind by name) to the interfaces of,
46
+ the Work and Derivative Works thereof.
47
+
48
+ "Contribution" shall mean any work of authorship, including
49
+ the original version of the Work and any modifications or additions
50
+ to that Work or Derivative Works thereof, that is intentionally
51
+ submitted to Licensor for inclusion in the Work by the copyright owner
52
+ or by an individual or Legal Entity authorized to submit on behalf of
53
+ the copyright owner. For the purposes of this definition, "submitted"
54
+ means any form of electronic, verbal, or written communication sent
55
+ to the Licensor or its representatives, including but not limited to
56
+ communication on electronic mailing lists, source code control systems,
57
+ and issue tracking systems that are managed by, or on behalf of, the
58
+ Licensor for the purpose of discussing and improving the Work, but
59
+ excluding communication that is conspicuously marked or otherwise
60
+ designated in writing by the copyright owner as "Not a Contribution."
61
+
62
+ "Contributor" shall mean Licensor and any individual or Legal Entity
63
+ on behalf of whom a Contribution has been received by Licensor and
64
+ subsequently incorporated within the Work.
65
+
66
+ 2. Grant of Copyright License. Subject to the terms and conditions of
67
+ this License, each Contributor hereby grants to You a perpetual,
68
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
69
+ copyright license to reproduce, prepare Derivative Works of,
70
+ publicly display, publicly perform, sublicense, and distribute the
71
+ Work and such Derivative Works in Source or Object form.
72
+
73
+ 3. Grant of Patent License. Subject to the terms and conditions of
74
+ this License, each Contributor hereby grants to You a perpetual,
75
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
76
+ (except as stated in this section) patent license to make, have made,
77
+ use, offer to sell, sell, import, and otherwise transfer the Work,
78
+ where such license applies only to those patent claims licensable
79
+ by such Contributor that are necessarily infringed by their
80
+ Contribution(s) alone or by combination of their Contribution(s)
81
+ with the Work to which such Contribution(s) was submitted. If You
82
+ institute patent litigation against any entity (including a
83
+ cross-claim or counterclaim in a lawsuit) alleging that the Work
84
+ or a Contribution incorporated within the Work constitutes direct
85
+ or contributory patent infringement, then any patent licenses
86
+ granted to You under this License for that Work shall terminate
87
+ as of the date such litigation is filed.
88
+
89
+ 4. Redistribution. You may reproduce and distribute copies of the
90
+ Work or Derivative Works thereof in any medium, with or without
91
+ modifications, and in Source or Object form, provided that You
92
+ meet the following conditions:
93
+
94
+ (a) You must give any other recipients of the Work or
95
+ Derivative Works a copy of this License; and
96
+
97
+ (b) You must cause any modified files to carry prominent notices
98
+ stating that You changed the files; and
99
+
100
+ (c) You must retain, in the Source form of any Derivative Works
101
+ that You distribute, all copyright, patent, trademark, and
102
+ attribution notices from the Source form of the Work,
103
+ excluding those notices that do not pertain to any part of
104
+ the Derivative Works; and
105
+
106
+ (d) If the Work includes a "NOTICE" text file as part of its
107
+ distribution, then any Derivative Works that You distribute must
108
+ include a readable copy of the attribution notices contained
109
+ within such NOTICE file, excluding those notices that do not
110
+ pertain to any part of the Derivative Works, in at least one
111
+ of the following places: within a NOTICE text file distributed
112
+ as part of the Derivative Works; within the Source form or
113
+ documentation, if provided along with the Derivative Works; or,
114
+ within a display generated by the Derivative Works, if and
115
+ wherever such third-party notices normally appear. The contents
116
+ of the NOTICE file are for informational purposes only and
117
+ do not modify the License. You may add Your own attribution
118
+ notices within Derivative Works that You distribute, alongside
119
+ or as an addendum to the NOTICE text from the Work, provided
120
+ that such additional attribution notices cannot be construed
121
+ as modifying the License.
122
+
123
+ You may add Your own copyright statement to Your modifications and
124
+ may provide additional or different license terms and conditions
125
+ for use, reproduction, or distribution of Your modifications, or
126
+ for any such Derivative Works as a whole, provided Your use,
127
+ reproduction, and distribution of the Work otherwise complies with
128
+ the conditions stated in this License.
129
+
130
+ 5. Submission of Contributions. Unless You explicitly state otherwise,
131
+ any Contribution intentionally submitted for inclusion in the Work
132
+ by You to the Licensor shall be under the terms and conditions of
133
+ this License, without any additional terms or conditions.
134
+ Notwithstanding the above, nothing herein shall supersede or modify
135
+ the terms of any separate license agreement you may have executed
136
+ with Licensor regarding such Contributions.
137
+
138
+ 6. Trademarks. This License does not grant permission to use the trade
139
+ names, trademarks, service marks, or product names of the Licensor,
140
+ except as required for reasonable and customary use in describing the
141
+ origin of the Work and reproducing the content of the NOTICE file.
142
+
143
+ 7. Disclaimer of Warranty. Unless required by applicable law or
144
+ agreed to in writing, Licensor provides the Work (and each
145
+ Contributor provides its Contributions) on an "AS IS" BASIS,
146
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
147
+ implied, including, without limitation, any warranties or conditions
148
+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
149
+ PARTICULAR PURPOSE. You are solely responsible for determining the
150
+ appropriateness of using or redistributing the Work and assume any
151
+ risks associated with Your exercise of permissions under this License.
152
+
153
+ 8. Limitation of Liability. In no event and under no legal theory,
154
+ whether in tort (including negligence), contract, or otherwise,
155
+ unless required by applicable law (such as deliberate and grossly
156
+ negligent acts) or agreed to in writing, shall any Contributor be
157
+ liable to You for damages, including any direct, indirect, special,
158
+ incidental, or consequential damages of any character arising as a
159
+ result of this License or out of the use or inability to use the
160
+ Work (including but not limited to damages for loss of goodwill,
161
+ work stoppage, computer failure or malfunction, or any and all
162
+ other commercial damages or losses), even if such Contributor
163
+ has been advised of the possibility of such damages.
164
+
165
+ 9. Accepting Warranty or Additional Liability. While redistributing
166
+ the Work or Derivative Works thereof, You may choose to offer,
167
+ and charge a fee for, acceptance of support, warranty, indemnity,
168
+ or other liability obligations and/or rights consistent with this
169
+ License. However, in accepting such obligations, You may act only
170
+ on Your own behalf and on Your sole responsibility, not on behalf
171
+ of any other Contributor, and only if You agree to indemnify,
172
+ defend, and hold each Contributor harmless for any liability
173
+ incurred by, or claims asserted against, such Contributor by reason
174
+ of your accepting any such warranty or additional liability.
175
+
176
+ END OF TERMS AND CONDITIONS
177
+
178
+ APPENDIX: How to apply the Apache License to your work.
179
+
180
+ To apply the Apache License to your work, attach the following
181
+ boilerplate notice, with the fields enclosed by brackets "[]"
182
+ replaced with your own identifying information. (Don't include
183
+ the brackets!) The text should be enclosed in the appropriate
184
+ comment syntax for the file format. We also recommend that a
185
+ file or class name and description of purpose be included on the
186
+ same "printed page" as the copyright notice for easier
187
+ identification within third-party archives.
188
+
189
+ Copyright [yyyy] [name of copyright owner]
190
+
191
+ Licensed under the Apache License, Version 2.0 (the "License");
192
+ you may not use this file except in compliance with the License.
193
+ You may obtain a copy of the License at
194
+
195
+ http://www.apache.org/licenses/LICENSE-2.0
196
+
197
+ Unless required by applicable law or agreed to in writing, software
198
+ distributed under the License is distributed on an "AS IS" BASIS,
199
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
200
+ See the License for the specific language governing permissions and
201
+ limitations under the License.
MANIFEST.in ADDED
@@ -0,0 +1,10 @@
 
 
 
 
 
 
 
 
 
 
 
1
+ include LICENSE
2
+ include README.md
3
+ include index.html
4
+ include style.css
5
+ include src/peachy/.env.example
6
+ recursive-include src/peachy/static *
7
+ recursive-include src/peachy/profiles *
8
+ recursive-include docs/assets *
9
+ include tests/_package_build.py
10
+ global-exclude *.py[cod]
README.md CHANGED
@@ -1,10 +1,411 @@
1
  ---
2
  title: Peachy
3
- emoji: 📈
4
- colorFrom: green
5
- colorTo: red
6
  sdk: static
7
  pinned: false
 
 
 
 
 
8
  ---
9
 
10
- Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  title: Peachy
3
+ emoji: 🍑
4
+ colorFrom: yellow
5
+ colorTo: pink
6
  sdk: static
7
  pinned: false
8
+ short_description: Peachy for Reachy Mini
9
+ suggested_storage: large
10
+ tags:
11
+ - reachy_mini
12
+ - reachy_mini_python_app
13
  ---
14
 
15
+ # Peachy
16
+
17
+ Peachy is an on-robot conversational assistant for the Reachy Mini robot, combining realtime voice backends and choreographed motion libraries.
18
+
19
+ ![Reachy Mini Dance](docs/assets/reachy_mini_dance.gif)
20
+
21
+ ## Table of contents
22
+ - [Overview](#overview)
23
+ - [Architecture](#architecture)
24
+ - [Install and configure](#install-and-configure)
25
+ - [Installation](#installation)
26
+ - [Configuration](#configuration)
27
+ - [Running the app](#running-the-app)
28
+ - [LLM tools](#llm-tools-exposed-to-the-assistant)
29
+ - [Advanced features](#advanced-features)
30
+ - [Contributing](#contributing)
31
+ - [License](#license)
32
+
33
+ ## Overview
34
+ - Real-time audio conversation loop for low-latency streaming, powered by **OpenAI Realtime** by default with Pollen's Hugging Face backend preserved as an explicit alternative.
35
+ - Vision is handled by the realtime backend when the `camera` tool is used.
36
+ - Layered motion system queues primary moves (dances, emotions, goto poses, breathing) while blending speech-reactive wobble.
37
+ - Async tool dispatch integrates robot motion and camera capture. An optional web UI (`--ui`) provides personality selection, mic control, and settings.
38
+
39
+ ## Architecture
40
+
41
+ The app follows a layered architecture connecting the user, AI services, and robot hardware:
42
+
43
+ <p align="center">
44
+ <img src="docs/assets/conversation_app_arch.svg" alt="Architecture Diagram" width="600"/>
45
+ </p>
46
+
47
+ ## Install and configure
48
+
49
+ The Reachy Mini app manager installs the built wheel into its shared apps
50
+ environment. Peachy declares `reachy-mini==1.9.0` and does not install the
51
+ MuJoCo extra.
52
+
53
+ Set configuration through the app settings page, or create the owner-only
54
+ state file at `${PEACHY_STATE_DIR:-${XDG_STATE_HOME:-$HOME/.local/state}/peachy}/.env`:
55
+
56
+ ```dotenv
57
+ PEACHY_REALTIME_PROVIDER=openai
58
+ OPENAI_API_KEY=replace-through-local-settings
59
+ OPENAI_REALTIME_MODEL=gpt-realtime
60
+ ```
61
+
62
+ Use `PEACHY_REALTIME_PROVIDER=huggingface` only to select the preserved Pollen
63
+ backend explicitly. Peachy never switches providers automatically.
64
+
65
+ ## Installation
66
+
67
+ > [!IMPORTANT]
68
+ > Before using this app, you need to install [Reachy Mini's SDK](https://github.com/pollen-robotics/reachy_mini/).<br>
69
+ > Windows support is currently experimental and has not been extensively tested. Use with caution.
70
+
71
+ <details open>
72
+ <summary><b>Using uv (recommended)</b></summary>
73
+
74
+ Set up the project quickly using [uv](https://docs.astral.sh/uv/):
75
+
76
+ ```bash
77
+ # macOS (Homebrew)
78
+ uv venv --python /opt/homebrew/bin/python3.12 .venv
79
+
80
+ # Linux / Windows (Python in PATH)
81
+ uv venv --python python3.12 .venv
82
+
83
+ source .venv/bin/activate
84
+ uv sync
85
+ ```
86
+
87
+ > **Note:** To reproduce the exact dependency set from this repo's `uv.lock`, run `uv sync --frozen`. This ensures `uv` installs directly from the lockfile without re-resolving or updating any versions.
88
+
89
+ Include dev dependencies:
90
+ ```bash
91
+ uv sync --group dev
92
+ ```
93
+
94
+ </details>
95
+
96
+ <details>
97
+ <summary><b>Using pip</b></summary>
98
+
99
+ ```bash
100
+ python -m venv .venv
101
+ source .venv/bin/activate
102
+ pip install -e .
103
+ ```
104
+
105
+ **Install dev dependencies with uv (from `onrobot`):**
106
+ ```bash
107
+ uv sync --group dev
108
+ ```
109
+
110
+ </details>
111
+
112
+ ### Dependency groups
113
+
114
+ | Group | Purpose | Notes |
115
+ |-------|---------|-------|
116
+ | `dev` | Developer tooling (`pytest`, `ruff`, `mypy`) | Development-only dependencies. Use `uv sync --group dev` from `onrobot`. |
117
+
118
+ ## Configuration
119
+
120
+ The default setup uses OpenAI Realtime and requires `OPENAI_API_KEY`. Save it
121
+ through the app settings page or the owner-only state file described above.
122
+ Select the Hugging Face backend explicitly before using its connection settings.
123
+
124
+ | Variable | Description |
125
+ |----------|-------------|
126
+ | `PEACHY_REALTIME_PROVIDER` | Realtime backend selector. Defaults to `openai`; set `huggingface` explicitly for the preserved Pollen backend. Peachy does not fail over automatically. |
127
+ | `OPENAI_API_KEY` | Required when the selected provider is `openai`. Store it through local settings or the owner-only state `.env`. |
128
+ | `OPENAI_REALTIME_MODEL` | OpenAI Realtime model. Defaults to `gpt-realtime`. |
129
+ | `REALTIME_TRANSCRIPTION_LANGUAGE` | Optional input transcription language for the realtime backend. Defaults to `en`; set to a backend-supported code such as `zh` for Chinese. |
130
+ | `HF_REALTIME_CONNECTION_MODE` | Hugging Face connection selector: `deployed` uses the built-in Hugging Face server; `local` uses `HF_REALTIME_WS_URL`. Defaults to `deployed`. |
131
+ | `HF_REALTIME_WS_URL` | Direct websocket endpoint for your own Hugging Face backend. Accepts either a base URL like `ws://127.0.0.1:8765/v1` or the full websocket URL `ws://127.0.0.1:8765/v1/realtime`. Used when `HF_REALTIME_CONNECTION_MODE=local`. |
132
+ | `HF_TOKEN` | Optional token for Hugging Face access (for gated/private assets). |
133
+ | `REACHY_MINI_APP_TIMEOUT_MINUTES` | Minutes of inactivity before Reachy goes to sleep and the app stops. Defaults to `1440` (one day); set to `0` to disable. |
134
+
135
+ ### Hugging Face Connection Modes
136
+
137
+ After selecting the Hugging Face provider, use the built-in Hugging Face server through the app-managed Space proxy. This is the default Hugging Face connection mode; set it explicitly only when you want to switch back from a saved local endpoint:
138
+
139
+ ```env
140
+ HF_REALTIME_CONNECTION_MODE=deployed
141
+ ```
142
+
143
+ Run your own realtime voice backend using [speech-to-speech](https://github.com/huggingface/speech-to-speech) on the same machine as the conversation app:
144
+
145
+ ```env
146
+ HF_REALTIME_CONNECTION_MODE=local
147
+ HF_REALTIME_WS_URL=ws://127.0.0.1:8765/v1/realtime
148
+ ```
149
+
150
+ Run your own Hugging Face backend on your laptop and connect to it from Reachy Mini Wireless over the same Wi-Fi network:
151
+
152
+ ```env
153
+ HF_REALTIME_CONNECTION_MODE=local
154
+ HF_REALTIME_WS_URL=ws://<your-laptop-lan-ip>:8765/v1/realtime
155
+ ```
156
+
157
+ For that LAN setup, make sure the backend listens on an address reachable from the robot, not only on `127.0.0.1`.
158
+
159
+ If the backend stays bound to loopback on your laptop, you can forward it into the robot over SSH instead:
160
+
161
+ ```bash
162
+ ssh -N -R 8765:127.0.0.1:8765 <robot-user>@<robot-host>
163
+ ```
164
+
165
+ Then set this on the robot:
166
+
167
+ ```env
168
+ HF_REALTIME_CONNECTION_MODE=local
169
+ HF_REALTIME_WS_URL=ws://127.0.0.1:8765/v1/realtime
170
+ ```
171
+
172
+ In the web UI's Settings view, the Connection section lets you choose either the built-in server or a local `host:port` target. The UI writes `HF_REALTIME_CONNECTION_MODE` for you, and the local path writes `HF_REALTIME_WS_URL` with a default of `localhost:8765`.
173
+
174
+ ## Running the app
175
+
176
+ Activate your virtual environment, then launch:
177
+
178
+ ```bash
179
+ peachy
180
+ ```
181
+
182
+ > [!TIP]
183
+ > Make sure the Reachy Mini daemon is running before launching the app. If you see a `TimeoutError`, it means the daemon isn't started. See [Reachy Mini's SDK](https://github.com/pollen-robotics/reachy_mini/) for setup instructions.
184
+
185
+ The app runs in console mode by default. Add `--ui` to also serve a web UI at http://127.0.0.1:7860/ for picking a personality, controlling the mic, and changing settings. All options are described in the CLI table below.
186
+
187
+ ### CLI options
188
+
189
+ | Option | Default | Description |
190
+ |--------|---------|-------------|
191
+ | `--no-camera` | `False` | Run without camera capture. |
192
+ | `--ui` | `False` | Serve the web UI at http://127.0.0.1:7860/, in addition to console mode. |
193
+ | `--robot-name` | `None` | Optional. Connect to a specific robot by name when running multiple daemons on the same subnet. See [Multiple robots on the same subnet](#advanced-features). |
194
+ | `--debug` | `False` | Enable verbose logging for troubleshooting. |
195
+
196
+ ### Examples
197
+
198
+ ```bash
199
+ # Audio-only conversation (no camera)
200
+ peachy --no-camera
201
+
202
+ # Launch with the minimal web UI for personality/mic/settings control
203
+ peachy --ui
204
+ ```
205
+
206
+ ## LLM tools exposed to the assistant
207
+
208
+ The default profile exposes these tools. Custom profiles can enable a different set in their own `tools.txt`.
209
+
210
+ | Tool | Action | Dependencies |
211
+ |------|--------|--------------|
212
+ | `dance` | Queue a dance from `reachy_mini_dances_library`. | Core install only. |
213
+ | `stop_dance` | Clear queued dances. | Core install only. |
214
+ | `play_emotion` | Play a recorded emotion clip via Hugging Face datasets. | Core install only. Uses the default open emotions dataset: [`pollen-robotics/reachy-mini-emotions-library`](https://huggingface.co/datasets/pollen-robotics/reachy-mini-emotions-library). |
215
+ | `stop_emotion` | Clear queued emotions. | Core install only. |
216
+ | `camera` | Capture the latest camera frame and analyze it with the selected realtime backend. | Core install only. Requires the camera (disable with `--no-camera`). |
217
+ | `idle_do_nothing` | Explicitly remain idle during an idle turn. Not intended for normal conversation turns. | Core install only. |
218
+ | `move_head` | Queue a head pose change (left/right/up/down/front). | Core install only. |
219
+ | `head_tracking` | Follow the user's face with the head, or stop following. | Core install only. Requires a daemon with the `vision` extra and a camera. |
220
+ | `go_to_sleep` | Run Reachy's sleep movement and stop the current app after an explicit user request. | Core install only. |
221
+ | `sweep_look` | Sweep Reachy's head left, right, and back to center. | Bundled default profile tool. |
222
+ | `remember` | Save one short, stable fact about the user for future sessions. | Core install only. Stored in the app instance data directory. |
223
+ | `forget` | Remove a saved memory fact by matching a short query. | Core install only. |
224
+ | `pollen_robotics_reachy_mini_search_tool__search_web` | Search the web and return a short list of results. | Preinstalled MCP Space: `pollen-robotics/reachy-mini-search-tool`. |
225
+ | `pollen_robotics_reachy_mini_weather_tool__get_weather` | Report today's weather for a place: current conditions, high and low temperature, and rain chance. | Preinstalled MCP Space: `pollen-robotics/reachy-mini-weather-tool`. |
226
+ | `pollen_robotics_reachy_mini_time_tool__get_time` | Report the current time for a timezone or the user's local time, or the difference between two timezones. | Preinstalled MCP Space: `pollen-robotics/reachy-mini-time-tool`. |
227
+
228
+ > [!NOTE]
229
+ > `remember`/`forget` facts are stored in `memory.v1.json` inside the app's instance data directory (`~/.local/share/peachy/` by default, or the instance path used by the desktop launcher). `forget` only removes facts matched by query. To reset all remembered facts, delete this file.
230
+
231
+ ## Advanced features
232
+
233
+ Built-in motion content is published as open Hugging Face datasets:
234
+ - Emotions: [`pollen-robotics/reachy-mini-emotions-library`](https://huggingface.co/datasets/pollen-robotics/reachy-mini-emotions-library)
235
+ - Dances: [`pollen-robotics/reachy-mini-dances-library`](https://huggingface.co/datasets/pollen-robotics/reachy-mini-dances-library)
236
+
237
+ <details>
238
+ <summary><b>Custom profiles</b></summary>
239
+
240
+ Create custom profiles with dedicated instructions and enabled tools.
241
+
242
+ For normal usage, select a profile from the UI and save it for startup. That selection is persisted in `startup_settings.json`.
243
+
244
+ If no startup settings have been saved yet, `REACHY_MINI_CUSTOM_PROFILE=<name>` selects a named built-in package profile unless an external profiles directory is configured. If neither is set, Peachy uses its built-in default profile.
245
+
246
+ Each profile should include `instructions.txt` (prompt text). If that file is missing or empty, the app logs a warning and falls back to Peachy's package-local default profile. `greeting.txt` is optional and controls how the robot should start the conversation after the backend connects. `tools.txt` (list of allowed tools) is recommended. If missing for a non-default profile, the app falls back to the built-in package-local default tools. Profiles can optionally contain custom tool implementations.
247
+
248
+ **Startup greeting:**
249
+
250
+ On startup, once the realtime backend is connected and ready, the app sends the active profile's `greeting.txt` as an internal text turn so the model opens with a fresh spoken greeting. Keep this file as a short instruction, not a fixed script, for example:
251
+ ```
252
+ Greet me warmly in one sentence, in character, and vary the wording each time.
253
+ ```
254
+ If `greeting.txt` is missing, the app uses the built-in default greeting prompt.
255
+
256
+ **Enabling tools:**
257
+
258
+ List enabled tools in `tools.txt`, one per line. Prefix with `#` to comment out:
259
+ ```
260
+ play_emotion
261
+ # move_head
262
+
263
+ # My custom tool defined locally
264
+ sweep_look
265
+ ```
266
+ Tools are resolved first from Python files in the profile folder (custom tools), then from the core library `src/peachy/tools/` (like `dance`, `camera`).
267
+ Installed Hugging Face Space tools can also be enabled here after you add them with `tool-spaces`.
268
+
269
+ **Custom tools:**
270
+
271
+ On top of built-in tools found in the core library, you can implement custom tools specific to your profile by adding Python files in the profile folder.
272
+ Custom tools must subclass `peachy.tools.core_tools.Tool` (see that module for the interface).
273
+
274
+ **Edit personalities from the UI:**
275
+
276
+ When running with `--ui`, the Home view lists package-local built-ins plus instance-directory user personalities:
277
+ - Tap a card to apply that personality and start talking.
278
+ - Tap "Custom" to create a new personality by entering a name, instructions, and an optional startup greeting prompt. It copies `tools.txt` from the `default` profile and stores the files under `user_personalities/<name>/` in the app instance directory (next to `.env`/`startup_settings.json`).
279
+
280
+ In a source checkout, built-ins live under `src/peachy/profiles/`; installed built-ins are read-only package data. Author durable profiles through the UI or an external profiles directory.
281
+
282
+ Note: switching a personality reloads its instructions and tools in place via a quick backend reconnect — no app restart. Editing the active profile's files on disk needs a re-select (or restart) to apply.
283
+
284
+ </details>
285
+
286
+ <details>
287
+ <summary><b>Locked profile mode</b></summary>
288
+
289
+ To create a locked variant of the app that cannot switch profiles, edit `src/peachy/config.py` and set the `LOCKED_PROFILE` constant to the desired profile name:
290
+ ```python
291
+ LOCKED_PROFILE: str | None = "mars_rover" # Lock to this profile
292
+ ```
293
+ When `LOCKED_PROFILE` is set, the app always uses that profile, ignoring saved startup settings, `REACHY_MINI_CUSTOM_PROFILE`, and the web UI. The UI shows "(locked)" and disables all profile editing controls.
294
+ This is useful for creating dedicated clones of the app with a fixed personality. Clone scripts can simply edit this constant to lock the variant.
295
+
296
+ </details>
297
+
298
+ <details>
299
+ <summary><b>External profiles and tools</b></summary>
300
+
301
+ You can extend the app with profiles/tools stored outside the repository defaults. Source checkout built-ins are under `src/peachy/profiles/`; installed built-ins are read-only package data. Author durable profiles through the UI or an external profiles directory.
302
+
303
+ - Core tools are under `src/peachy/tools/`.
304
+
305
+ **Recommended layout:**
306
+
307
+ ```text
308
+ external_content/
309
+ ├── external_profiles/
310
+ │ └── my_profile/
311
+ │ ├── instructions.txt
312
+ │ ├── greeting.txt # optional startup greeting prompt
313
+ │ ├── tools.txt # optional (see fallback behavior below)
314
+ │ └── voice.txt # optional
315
+ ├── external_tools/
316
+ │ └── my_custom_tool.py
317
+ └── installed_tool_spaces.json
318
+ ```
319
+
320
+ **Environment variables:**
321
+
322
+ Set these values in your `.env` when you want env-driven external profile/tool selection:
323
+
324
+ ```env
325
+ # Optional fallback/manual profile selector:
326
+ REACHY_MINI_CUSTOM_PROFILE=my_profile
327
+ REACHY_MINI_EXTERNAL_PROFILES_DIRECTORY=./external_content/external_profiles
328
+ REACHY_MINI_EXTERNAL_TOOLS_DIRECTORY=./external_content/external_tools
329
+ # Optional convenience mode:
330
+ # AUTOLOAD_EXTERNAL_TOOLS=1
331
+ ```
332
+
333
+ **Loading behavior:**
334
+
335
+ - **Default/strict mode**: `tools.txt` defines enabled tools explicitly. Every name in `tools.txt` must resolve to either a built-in tool (`src/peachy/tools/`) or an external tool module in `REACHY_MINI_EXTERNAL_TOOLS_DIRECTORY`.
336
+ - **Convenience mode** (`AUTOLOAD_EXTERNAL_TOOLS=1`): all valid `*.py` tool files in `REACHY_MINI_EXTERNAL_TOOLS_DIRECTORY` are auto-added.
337
+ - **External profile fallback**: if the selected external profile has no `tools.txt`, the app falls back to the built-in package-local default tools.
338
+ - **Duplicate safety**: every loaded tool class must expose a unique `Tool.name`. The app now fails fast if two tool implementations claim the same tool name.
339
+
340
+ This supports both:
341
+ 1. Local external tools used with built-in/default profile.
342
+ 2. Local external profiles used with built-in default tools.
343
+
344
+ </details>
345
+
346
+ <details>
347
+ <summary><b>Hugging Face Space tools</b></summary>
348
+
349
+ You can install MCP-compatible Hugging Face Spaces as remote tool sources for this app. Private Spaces work too, as long as `HF_TOKEN` is set (or you have run `hf auth login`) for an account that can access them.
350
+
351
+ ```bash
352
+ # install + enable in active profile
353
+ peachy tool-spaces add <owner/space-name>
354
+
355
+ # enable in a specific profile
356
+ peachy tool-spaces add <owner/space-name> --profile NAME
357
+
358
+ # install without enabling
359
+ peachy tool-spaces add <owner/space-name> --install-only
360
+
361
+ # list installed spaces
362
+ peachy tool-spaces list
363
+
364
+ # remove an installed space
365
+ peachy tool-spaces remove owner/space-name
366
+ ```
367
+
368
+ The bundled Pollen Spaces are enabled by default and resolve from static specs, so startup needs no Hugging Face discovery. For custom Spaces, the app validates the slug through the Hugging Face Hub, probes the standard MCP endpoint (sending the HF token only to private Spaces), discovers tools, enables them in the active profile's `tools.txt`, and writes the installed Space to:
369
+
370
+ - `installed_tool_spaces.json` in the managed app instance directory
371
+ - `external_content/installed_tool_spaces.json` in terminal mode
372
+
373
+ Recommended tags for discoverability on Hugging Face:
374
+
375
+ - `reachy-mini-tool`
376
+ - `mcp`
377
+
378
+ These tags are advisory only. Installation still relies on successful MCP validation, not on tag presence.
379
+
380
+ > [!NOTE]
381
+ > Preinstalled Pollen Spaces can be removed like any other (`tool-spaces remove pollen-robotics/reachy-mini-weather-tool`) or delete `installed_tool_spaces.json` to restore all defaults.
382
+
383
+ </details>
384
+
385
+ <details>
386
+ <summary><b>Multiple robots on the same subnet</b></summary>
387
+
388
+ If you run multiple Reachy Mini daemons on the same network, use:
389
+
390
+ ```bash
391
+ peachy --robot-name <name>
392
+ ```
393
+
394
+ `<name>` must match the daemon's `--robot-name` value so the app connects to the correct robot.
395
+
396
+ </details>
397
+
398
+ ## Contributing
399
+
400
+ We welcome bug fixes, features, profiles, and documentation improvements. From `onrobot`, set up the development environment and run the local checks:
401
+
402
+ ```bash
403
+ uv sync --group dev
404
+ .venv/bin/python -m pytest tests
405
+ .venv/bin/ruff check src tests
406
+ .venv/bin/mypy
407
+ ```
408
+
409
+ ## License
410
+
411
+ Apache 2.0
UPSTREAM.md ADDED
@@ -0,0 +1,24 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Upstream provenance
2
+
3
+ Peachy is a modified source fork of [Pollen Robotics' Reachy Mini Conversation App](https://github.com/pollen-robotics/reachy_mini_conversation_app).
4
+
5
+ - Upstream tag: `v0.10.0`
6
+ - Immutable source commit: `40ea94e49213f77ae83d21a81c1766b2e4f25e49`
7
+
8
+ ## Copied input mapping
9
+
10
+ - `src/reachy_mini_conversation_app/** -> onrobot/src/peachy/**`
11
+ - `tests/** -> onrobot/tests/**`
12
+ - `profiles/** -> onrobot/src/peachy/profiles/**`
13
+ - `.env.example -> onrobot/.env.example`
14
+ - `.env.example -> onrobot/src/peachy/.env.example`
15
+ - `LICENSE -> onrobot/LICENSE`
16
+ - `index.html -> onrobot/index.html`
17
+ - `style.css -> onrobot/style.css`
18
+ - `docs/assets/conversation_app_arch.svg -> onrobot/docs/assets/conversation_app_arch.svg`
19
+ - `docs/assets/reachy_mini_dance.gif -> onrobot/docs/assets/reachy_mini_dance.gif`
20
+
21
+ This Apache-2.0-licensed source has been modified by Peachy contributors for namespace, packaging, and provider integration.
22
+
23
+ External profile/tool Python files that import `reachy_mini_conversation_app.*`
24
+ must use `peachy.*` in this spike. No compatibility namespace is installed.
docs/assets/conversation_app_arch.svg ADDED
docs/assets/reachy_mini_dance.gif ADDED

Git LFS Details

  • SHA256: 75914c3cb7af982e0b1c6369e25fc46d8c08a0ab5ad022240ae9c1a0d93967c3
  • Pointer size: 132 Bytes
  • Size of remote file: 3.93 MB
docs/legacy-regression-bootstrap.md ADDED
@@ -0,0 +1,91 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Legacy root regression bootstrap
2
+
3
+ The repository-root Mac/simulator app targets the old FastRTC engine and SDK
4
+ 1.8.0. Its tests must not run in the `onrobot` SDK 1.9.0 environment.
5
+
6
+ ```bash
7
+ PEACHY_LEGACY_ENGINE_DIR="$(mktemp -d /tmp/peachy-engine-v030.XXXXXX)"
8
+ PEACHY_LEGACY_VENV_DIR="$(mktemp -d /tmp/peachy-legacy-venv.XXXXXX)"
9
+ git clone --depth 1 --branch v0.3.0 \
10
+ https://github.com/pollen-robotics/reachy_mini_conversation_app.git \
11
+ "$PEACHY_LEGACY_ENGINE_DIR"
12
+ test "$(git -C "$PEACHY_LEGACY_ENGINE_DIR" rev-parse HEAD)" = \
13
+ "93674f987cd3172296e503d81674ece3ab58b4df"
14
+ uv venv --python python3.12 "$PEACHY_LEGACY_VENV_DIR"
15
+ uv pip install --python "$PEACHY_LEGACY_VENV_DIR/bin/python" \
16
+ -r requirements.txt -r requirements-dev.txt -e "$PEACHY_LEGACY_ENGINE_DIR"
17
+ uv pip install --python "$PEACHY_LEGACY_VENV_DIR/bin/python" \
18
+ --no-deps -e .
19
+ test "$("$PEACHY_LEGACY_VENV_DIR/bin/python" -c \
20
+ 'import importlib.metadata as m; print(m.version("reachy-mini-conversation-app"))')" = \
21
+ "0.3.0"
22
+ test "$("$PEACHY_LEGACY_VENV_DIR/bin/python" -c \
23
+ 'import importlib.metadata as m; print(m.version("reachy-mini"))')" = \
24
+ "1.8.0"
25
+ "$PEACHY_LEGACY_VENV_DIR/bin/python" -m pytest tests -q -m "not e2e"
26
+ ```
27
+
28
+ Expected: the exact SHA and package-version checks succeed, and this diagnostic
29
+ non-daemon baseline passes. It is not a silent substitute for the full default
30
+ gate described below. The root editable install uses `--no-deps`, so
31
+ the legacy engine remains 0.3.0 and the SDK remains 1.8.0. Keep both temporary
32
+ paths until failures are diagnosed; remove them manually only after results have
33
+ been recorded. This bootstrap does not install the new `onrobot` package and
34
+ does not modify the repository-root application.
35
+
36
+ ## Full default gate and simulator prerequisite
37
+
38
+ The original required gate is unfiltered:
39
+
40
+ ```bash
41
+ REACHY_VENV_PY="$PEACHY_LEGACY_VENV_DIR/bin/python" \
42
+ "$PEACHY_LEGACY_VENV_DIR/bin/python" -m pytest tests -q
43
+ ```
44
+
45
+ The two default simulator tests use the repository's existing daemon fixture,
46
+ which launches
47
+ `reachy_mini_env/bin/python` unless `REACHY_VENV_PY` names another provisioned
48
+ SDK 1.8.0 simulator environment. The fixture starts:
49
+
50
+ ```text
51
+ python -m reachy_mini.daemon.app.main --sim --headless --no-media
52
+ ```
53
+
54
+ It treats port 8000 as ready, then requires the SDK client to receive initial
55
+ joint-position and head-pose state. Provision `reachy_mini_env` with the root
56
+ README workflow and confirm no daemon is already occupying port 8000 before
57
+ running the full gate or the focused diagnostic:
58
+
59
+ ```bash
60
+ REACHY_VENV_PY="$PWD/reachy_mini_env/bin/python" \
61
+ "$PEACHY_LEGACY_VENV_DIR/bin/python" -m pytest \
62
+ tests/e2e/test_m5_local_brain.py tests/e2e/test_sim_smoke.py -q -m e2e
63
+ ```
64
+
65
+ These tests require local socket binding and a working headless MuJoCo daemon.
66
+ Other opt-in E2E, model, voice, macOS-app, and robot markers retain the separate
67
+ authorization and hardware/model prerequisites documented in the root README.
68
+
69
+ ### macOS simulator blocker and acceptance decision
70
+
71
+ On the reviewed Apple Silicon host, both the Python and `mjpython` launchers
72
+ reported that SDK 1.8.0's daemon started successfully and bound port 8000, but a
73
+ 15-second `localhost_only` SDK probe never received initial joint/head state.
74
+ Consequently the unfiltered gate completed with `893 passed, 56 skipped, 2
75
+ errors`; only these setup nodes errored:
76
+
77
+ - `tests/e2e/test_m5_local_brain.py::test_local_brain_moves_head_and_emits_audio`
78
+ - `tests/e2e/test_sim_smoke.py::test_sim_moves_antennas`
79
+
80
+ The root README records the working macOS fallback as a non-headless daemon
81
+ started with `mjpython` in a real Terminal. That interactive GUI prerequisite is
82
+ outside a headless test worker. To complete the two nodes, provision the
83
+ documented worktree-local `reachy_mini_env`, start that GUI daemon in a real
84
+ Terminal, wait for actual joint/head state, and then run the focused command
85
+ above; the fixture will reuse the daemon already on port 8000.
86
+
87
+ Task 9 received explicit human acceptance of this amendment: the passing
88
+ `-m "not e2e"` baseline and the unfiltered `893 passed, 56 skipped, 2 errors`
89
+ result are accepted with the two simulator setup errors recorded as an external
90
+ macOS GUI-daemon prerequisite. The two focused E2E nodes remain required GUI
91
+ simulator follow-up; this record does not claim the full default gate passed.
docs/openai-realtime-smoke.md ADDED
@@ -0,0 +1,86 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # OpenAI Realtime adapter smoke
2
+
3
+ This is a paid, opt-in test. It builds one client and opens at most one WebSocket
4
+ session attempt—there is no retry loop. It limits each response to 64 output
5
+ tokens, disables the startup greeting and tools, feeds one short checked-in voice
6
+ clip, feeds it once more to exercise barge-in, and observes at most two
7
+ `response.created` events (maximum 128 output tokens). A test-local wrapper
8
+ counts those events directly at the handler boundary, so the bounded console
9
+ activity queue cannot lose them; a third event cancels its active response and
10
+ fails the smoke immediately. It tears the session down within the test's
11
+ 45-second outer timeout. The approved ceiling for one run is **$0.10**.
12
+
13
+ ## TLS trust on macOS Python.org installs
14
+
15
+ The guarded smoke sets `SSL_CERT_FILE` to the active `onrobot/.venv` certifi CA
16
+ bundle after the spend/key guards pass and before it constructs or starts the
17
+ Realtime handler. This is test-local setup: it does not change the production
18
+ handler or persist an environment setting after the pytest process exits.
19
+
20
+ This setup is necessary on affected Python.org macOS installations where the
21
+ configured framework CA path, such as
22
+ `/Library/Frameworks/Python.framework/Versions/3.12/etc/openssl/cert.pem`, is
23
+ missing. In that condition, Python's default SSL context can fail with
24
+ `CERTIFICATE_VERIFY_FAILED: unable to get local issuer certificate` even though
25
+ the venv's certifi bundle is valid.
26
+
27
+ As a system-level alternative, run the installer matching the Python.org version
28
+ once (for Python 3.12, open `/Applications/Python 3.12/Install Certificates.command`).
29
+ That repairs the framework interpreter's default certificate link. The manual
30
+ smoke still selects its venv certifi bundle explicitly so its trust source is
31
+ deterministic.
32
+
33
+ ## Realtime event ordering
34
+
35
+ Response audio can reach the playback sink before the server publishes
36
+ `conversation.item.input_audio_transcription.completed`. After first audio, the
37
+ smoke therefore waits up to 10 seconds for the event bus reason
38
+ `user_transcription_completed` instead of taking an immediate queue snapshot.
39
+ It retains every earlier reason; if completion is truly absent, the assertion
40
+ reports the missing target and the reasons observed before the deadline. This
41
+ bounded wait remains inside the unchanged 45-second outer pytest timeout and
42
+ does not weaken the later barge-in or teardown checks.
43
+
44
+ Before running, review the [current official Realtime pricing page](https://developers.openai.com/api/docs/pricing). Do not run if
45
+ the two short inputs plus at most 128 output tokens could exceed $0.10. Use a
46
+ restricted project key, never paste the key into a command or committed file,
47
+ and authorize the ceiling explicitly:
48
+
49
+ ```bash
50
+ export OPENAI_API_KEY='set in your private shell or key manager'
51
+ export RUN_OPENAI_REALTIME=1
52
+ export PEACHY_LIVE_SPEND_APPROVED=0.10
53
+ onrobot/.venv/bin/python -m pytest -o addopts= -m live_openai \
54
+ onrobot/tests/manual/test_openai_realtime_live.py -v --timeout=45
55
+ ```
56
+
57
+ Pass criteria:
58
+
59
+ - the checked-in user recording reaches completed transcription;
60
+ - a delegating spy around the real console resampler proves 24 kHz provider input
61
+ reaches the 48 kHz sink with exactly twice the sample count;
62
+ - before the second feed, real response audio is pending both in the fake player
63
+ and in the handler output queue;
64
+ - second-feed speech start invokes exactly one additional player clear, empties
65
+ both pending layers, and that clear count remains exact through the complete
66
+ second-feed interruption lifecycle;
67
+ - releasing the playback gate during active speech does not replay cancelled
68
+ response audio; provider-audio provenance is captured at the event/queue
69
+ boundary, so a delayed cancelled frame remains attributed to its original
70
+ response even after a later `response.created` and is not mistaken for
71
+ legitimate later-response audio;
72
+ - a lossless test-local guard observes one or two capped `response.created`
73
+ events, mechanically keeping maximum output at 128 tokens; and
74
+ - teardown leaves no active connection or tool-manager task and surfaces any
75
+ unexpected session/playback task exception (intentional cancellations and
76
+ orderly completion are allowed).
77
+
78
+ For a listening check, route the sink through a real Reachy Mini only after the
79
+ automated adapter gate passes. Previously queued speech must stop and must not
80
+ resume after the interruption.
81
+
82
+ Known spike boundary: [official WebSocket guidance](https://developers.openai.com/api/docs/guides/realtime-conversations#interruption-and-truncation) also recommends sending
83
+ `conversation.item.truncate` with the amount of audio actually played so server
84
+ conversation state excludes unheard audio. This spike proves audible cancellation
85
+ and late-delta suppression but does not yet maintain a playback clock/item
86
+ truncation offset; record that as a required follow-on if the adapter gate passes.
index.html CHANGED
@@ -1,19 +1,125 @@
1
  <!doctype html>
2
  <html>
3
- <head>
4
- <meta charset="utf-8" />
5
- <meta name="viewport" content="width=device-width" />
6
- <title>My static Space</title>
7
- <link rel="stylesheet" href="style.css" />
8
- </head>
9
- <body>
10
- <div class="card">
11
- <h1>Welcome to your static Space!</h1>
12
- <p>You can modify this app directly by editing <i>index.html</i> in the Files and versions tab.</p>
13
- <p>
14
- Also don't forget to check the
15
- <a href="https://huggingface.co/docs/hub/spaces" target="_blank">Spaces documentation</a>.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
16
  </p>
17
  </div>
18
- </body>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
19
  </html>
 
1
  <!doctype html>
2
  <html>
3
+
4
+ <head>
5
+ <meta charset="utf-8" />
6
+ <meta name="viewport" content="width=device-width, initial-scale=1" />
7
+ <title>Reachy Mini Conversation App</title>
8
+ <link rel="preconnect" href="https://fonts.googleapis.com">
9
+ <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
10
+ <link href="https://fonts.googleapis.com/css2?family=Space+Grotesk:wght@400;500;600;700&family=Manrope:wght@400;500;600&display=swap" rel="stylesheet">
11
+ <link rel="stylesheet" href="style.css" />
12
+ </head>
13
+
14
+ <body>
15
+ <header class="hero">
16
+ <div class="topline">
17
+ <div class="brand">
18
+ <span class="logo">🤖</span>
19
+ <span class="brand-name">Reachy Mini</span>
20
+ </div>
21
+ <div class="pill">Realtime voice · Vision aware · Expressive motion</div>
22
+ </div>
23
+ <div class="hero-grid">
24
+ <div class="hero-copy">
25
+ <p class="eyebrow">Conversation App</p>
26
+ <h1>Talk, see, and move together.</h1>
27
+ <p class="lede">
28
+ A friendly, camera-aware companion for Reachy Mini. Chat out loud, watch it follow faces, dance, or react with recorded emotions—all while streaming transcripts in a clean web UI.
29
+ </p>
30
+ <div class="hero-actions">
31
+ <a class="btn primary" href="#highlights">Explore features</a>
32
+ <a class="btn ghost" href="#story">See how it feels</a>
33
+ </div>
34
+ <div class="hero-badges">
35
+ <span>Low-latency voice loop</span>
36
+ <span>Camera insights on demand</span>
37
+ <span>Choreographed dances & emotions</span>
38
+ <span>Personality profiles via web UI</span>
39
+ </div>
40
+ </div>
41
+ <div class="hero-visual">
42
+ <div class="glass-card">
43
+ <img src="docs/assets/reachy_mini_dance.gif" alt="Reachy Mini dancing" class="hero-gif">
44
+ <p class="caption">Reachy Mini can move, dance, and emote while holding a natural conversation.</p>
45
+ </div>
46
+ </div>
47
+ </div>
48
+ </header>
49
+
50
+ <section id="highlights" class="section features">
51
+ <div class="section-header">
52
+ <p class="eyebrow">What’s inside</p>
53
+ <h2>All-in-one conversational layer for your robot</h2>
54
+ <p class="intro">
55
+ The app blends realtime speech, vision, and motion so Reachy Mini feels present..
56
  </p>
57
  </div>
58
+ <div class="feature-grid">
59
+ <div class="feature-card">
60
+ <span class="icon">🎤</span>
61
+ <h3>Natural voice chat</h3>
62
+ <p>Talk freely and get fast, high-quality replies powered by realtime models.</p>
63
+ </div>
64
+ <div class="feature-card">
65
+ <span class="icon">🎥</span>
66
+ <h3>Vision-aware replies</h3>
67
+ <p>Ask the camera tool to see what’s in front, track a face, or keep attention on whoever is speaking.</p>
68
+ </div>
69
+ <div class="feature-card">
70
+ <span class="icon">💃</span>
71
+ <h3>Expressive motion</h3>
72
+ <p>Queue dances, play recorded emotions while Reachy listens and talks.</p>
73
+ </div>
74
+ <div class="feature-card">
75
+ <span class="icon">🧠</span>
76
+ <h3>Personalities on demand</h3>
77
+ <p>Switch conversation styles through profiles and decide which tools (dance, camera, tracking) each persona can use.</p>
78
+ </div>
79
+ <div class="feature-card">
80
+ <span class="icon">🌐</span>
81
+ <h3>Ready for your setup</h3>
82
+ <p>Works with wired or wireless Reachy Mini, and can run vision locally or through the default cloud model.</p>
83
+ </div>
84
+ </div>
85
+ </section>
86
+
87
+ <section id="story" class="section story">
88
+ <div class="story-grid">
89
+ <div class="story-card">
90
+ <p class="eyebrow">How it feels</p>
91
+ <h3>From hello to helpful in seconds</h3>
92
+ <ul class="story-list">
93
+ <li><span>👋</span> Say “Hey Reachy” and start chatting—no extra setup in the moment.</li>
94
+ <li><span>👀</span> Ask what it sees; it can peek through the camera or keep focus on your face.</li>
95
+ <li><span>🎭</span> Trigger emotions or dance breaks to keep the conversation lively.</li>
96
+ <li><span>📝</span> Follow along with live transcripts in the web UI or run audio-only from the console.</li>
97
+ </ul>
98
+ </div>
99
+ <div class="story-card secondary">
100
+ <p class="eyebrow">Where it shines</p>
101
+ <h3>Great for demos, teaching, and playful exploration</h3>
102
+ <p class="story-text">
103
+ Show off how Reachy Mini listens, responds, and moves in sync. Whether you’re guiding a class, hosting a booth, or experimenting at home, the app keeps the robot expressive without juggling scripts or joystick controls.
104
+ </p>
105
+ <div class="chips">
106
+ <span class="chip">Live conversation</span>
107
+ <span class="chip">Face tracking</span>
108
+ <span class="chip">Camera tool</span>
109
+ <span class="chip">Dance library</span>
110
+ <span class="chip">Profiles & tools</span>
111
+ </div>
112
+ </div>
113
+ </div>
114
+ </section>
115
+
116
+ <footer class="footer">
117
+ <p>
118
+ Reachy Mini Conversation App by <a href="https://github.com/pollen-robotics" target="_blank" rel="noopener">Pollen Robotics</a>.
119
+ Explore more apps on <a href="https://huggingface.co/spaces/pollen-robotics/Reachy_Mini_Apps" target="_blank" rel="noopener">Hugging Face Spaces</a>.
120
+ </p>
121
+ </footer>
122
+
123
+ </body>
124
+
125
  </html>
pyproject.toml ADDED
@@ -0,0 +1,79 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [build-system]
2
+ requires = ["setuptools>=68"]
3
+ build-backend = "setuptools.build_meta"
4
+
5
+ [project]
6
+ name = "peachy"
7
+ version = "0.1.0"
8
+ description = "Peachy — on-robot conversational assistant for Reachy Mini"
9
+ readme = "README.md"
10
+ license = { file = "LICENSE" }
11
+ requires-python = ">=3.12"
12
+ dependencies = [
13
+ "huggingface-hub>=1.17.0",
14
+ "httpx>=0.28.1",
15
+ "python-dotenv",
16
+ "openai==2.28.0",
17
+ "reachy_mini_dances_library",
18
+ "reachy_mini_toolbox",
19
+ "reachy-mini==1.9.0",
20
+ "mcp>=1.27.1",
21
+ "numpy",
22
+ "scipy",
23
+ ]
24
+ keywords = ["reachy-mini-app"]
25
+
26
+ [project.scripts]
27
+ peachy = "peachy.main:main"
28
+
29
+ [project.entry-points."reachy_mini_apps"]
30
+ peachy = "peachy.main:Peachy"
31
+
32
+ [dependency-groups]
33
+ dev = [
34
+ "build",
35
+ "certifi",
36
+ "pytest",
37
+ "pytest-asyncio",
38
+ "pytest-timeout",
39
+ "ruff==0.12.0",
40
+ "mypy==1.18.2",
41
+ ]
42
+
43
+ [tool.setuptools]
44
+ package-dir = { "" = "src" }
45
+ include-package-data = false
46
+
47
+ [tool.setuptools.packages.find]
48
+ where = ["src"]
49
+
50
+ [tool.setuptools.package-data]
51
+ peachy = [
52
+ "static/*",
53
+ "static/**/*",
54
+ ".env.example",
55
+ "profiles/**/*.txt",
56
+ ]
57
+
58
+ [tool.setuptools.exclude-package-data]
59
+ "*" = ["**/__pycache__/*", "**/*.pyc"]
60
+
61
+ [tool.pytest.ini_options]
62
+ testpaths = ["tests"]
63
+ asyncio_mode = "auto"
64
+ addopts = "-m 'not live_openai'"
65
+ markers = [
66
+ "live_openai: opt-in paid OpenAI Realtime smoke; never CI",
67
+ ]
68
+
69
+ [tool.ruff]
70
+ line-length = 119
71
+ exclude = [".venv", "dist", "build", "**/__pycache__", "*.egg-info", ".mypy_cache", ".pytest_cache"]
72
+
73
+ [tool.mypy]
74
+ python_version = "3.12"
75
+ files = ["src/"]
76
+ ignore_missing_imports = true
77
+ strict = true
78
+ show_error_codes = true
79
+ warn_unused_ignores = true
src/peachy/.env.example ADDED
@@ -0,0 +1,36 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Optional realtime input transcription language.
2
+ # Defaults to "en". Set to a backend-supported code such as "zh" for Chinese.
3
+ # REALTIME_TRANSCRIPTION_LANGUAGE="en"
4
+
5
+ PEACHY_REALTIME_PROVIDER=openai
6
+ OPENAI_API_KEY=
7
+ OPENAI_REALTIME_MODEL=gpt-realtime
8
+
9
+ # Hugging Face connection mode: "deployed" or "local".
10
+ # Deployed mode uses the built-in Hugging Face server.
11
+ # Defaults to "deployed" when unset.
12
+ # HF_REALTIME_CONNECTION_MODE="deployed"
13
+
14
+ # Direct Hugging Face realtime endpoint for local/LAN backends.
15
+ # Accepts either the base URL (`ws://127.0.0.1:8765/v1`) or the full realtime URL
16
+ # (`ws://127.0.0.1:8765/v1/realtime`). Used when HF_REALTIME_CONNECTION_MODE=local.
17
+ # HF_REALTIME_WS_URL="ws://127.0.0.1:8765/v1/realtime"
18
+
19
+ # Hugging Face token for accessing datasets/models
20
+ HF_TOKEN=
21
+
22
+ # Optional built-in profile selector. Use a folder name from profiles/.
23
+ # If unset, the app uses the default profile unless startup settings were saved from the UI.
24
+ # REACHY_MINI_CUSTOM_PROFILE=mars_rover
25
+
26
+ # Optional external profile/tool directories
27
+ # REACHY_MINI_EXTERNAL_PROFILES_DIRECTORY=external_content/external_profiles
28
+ # REACHY_MINI_EXTERNAL_TOOLS_DIRECTORY=external_content/external_tools
29
+
30
+ # Optional: discover and auto-load all tools found in REACHY_MINI_EXTERNAL_TOOLS_DIRECTORY,
31
+ # even if they are not listed in the selected profile's tools.txt.
32
+ # This is convenient for downloaded tools used with built-in/default profiles.
33
+ # AUTOLOAD_EXTERNAL_TOOLS=1
34
+
35
+ # Minutes of inactivity before Reachy goes to sleep and the app stops. Default 1440 (one day), 0 disables.
36
+ # REACHY_MINI_APP_TIMEOUT_MINUTES=1440
src/peachy/__init__.py ADDED
@@ -0,0 +1,2 @@
 
 
 
1
+ """Nothing (for ruff)."""
2
+ # Modified by Peachy contributors in 2026 for namespace, packaging, and provider integration.
src/peachy/app_lifecycle.py ADDED
@@ -0,0 +1,85 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Helpers for app startup and shutdown lifecycle behavior."""
2
+ # Modified by Peachy contributors in 2026 for namespace, packaging, and provider integration.
3
+
4
+
5
+ import asyncio
6
+ import logging
7
+ import urllib.error
8
+ import urllib.request
9
+
10
+ import numpy as np
11
+ import numpy.typing as npt
12
+
13
+ from reachy_mini import ReachyMini
14
+ from reachy_mini.reachy_mini import SLEEP_HEAD_POSE
15
+ from reachy_mini.utils.interpolation import distance_between_poses
16
+ from peachy.tools.core_tools import ToolDependencies
17
+ from peachy.tools.go_to_sleep import GoToSleep
18
+
19
+
20
+ _STOP_CURRENT_APP_PATH = "/api/apps/stop-current-app"
21
+ _STOP_CURRENT_APP_TIMEOUT_S = 2.0
22
+ _SLEEP_HEAD_TRANSLATION_TOLERANCE_M = 0.05
23
+ _SLEEP_HEAD_ROTATION_TOLERANCE_RAD = 0.35
24
+
25
+
26
+ def request_stop_current_app(robot: ReachyMini, logger: logging.Logger) -> bool:
27
+ """Request the Reachy Mini daemon to stop the current app."""
28
+ stop_current_app_url = f"http://{robot.client.host}:{robot.client.port}{_STOP_CURRENT_APP_PATH}"
29
+ request = urllib.request.Request(stop_current_app_url, method="POST")
30
+ try:
31
+ with urllib.request.urlopen(request, timeout=_STOP_CURRENT_APP_TIMEOUT_S) as response:
32
+ response.read()
33
+ except urllib.error.URLError as e:
34
+ logger.error("Failed to request current app stop via %s: %s", stop_current_app_url, e)
35
+ return False
36
+
37
+ logger.info("Requested current app stop via %s", stop_current_app_url)
38
+ return True
39
+
40
+
41
+ def _is_sleep_head_pose(head_pose: npt.ArrayLike) -> bool:
42
+ try:
43
+ current_head_pose: npt.NDArray[np.float64] = np.asarray(head_pose, dtype=np.float64)
44
+ except (TypeError, ValueError):
45
+ return False
46
+
47
+ if current_head_pose.shape != (4, 4):
48
+ return False
49
+
50
+ pose_distances = distance_between_poses(current_head_pose, SLEEP_HEAD_POSE)
51
+ translation_distance = float(pose_distances[0])
52
+ rotation_angle = float(pose_distances[1])
53
+ return (
54
+ translation_distance <= _SLEEP_HEAD_TRANSLATION_TOLERANCE_M
55
+ and rotation_angle <= _SLEEP_HEAD_ROTATION_TOLERANCE_RAD
56
+ )
57
+
58
+
59
+ def wake_up_if_sleeping(robot: ReachyMini, logger: logging.Logger) -> bool:
60
+ """Run the SDK wake-up movement when Reachy starts from the sleep pose."""
61
+ try:
62
+ head_pose = robot.get_current_head_pose()
63
+ except Exception as e:
64
+ logger.warning("Could not read robot pose before startup wake-up check: %s", e)
65
+ return False
66
+
67
+ if not _is_sleep_head_pose(head_pose):
68
+ return False
69
+
70
+ logger.info("Robot is in sleep pose; running wake-up movement.")
71
+ try:
72
+ robot.wake_up()
73
+ except Exception as e:
74
+ logger.error("Failed to run wake-up movement: %s", e)
75
+ return False
76
+ return True
77
+
78
+
79
+ def run_go_to_sleep_tool(deps: ToolDependencies, logger: logging.Logger) -> dict[str, object]:
80
+ """Run the shared go_to_sleep tool from synchronous shutdown paths."""
81
+ try:
82
+ return asyncio.run(GoToSleep()(deps))
83
+ except Exception as e:
84
+ logger.error("Failed to run go_to_sleep tool during shutdown: %s", e)
85
+ return {"error": f"go_to_sleep failed: {type(e).__name__}: {e}"}
src/peachy/audio/__init__.py ADDED
@@ -0,0 +1 @@
 
 
1
+ # Modified by Peachy contributors in 2026 for namespace, packaging, and provider integration.
src/peachy/audio/resampling.py ADDED
@@ -0,0 +1,55 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Provider-neutral mono normalization and audio sample-rate conversion."""
2
+
3
+ from __future__ import annotations
4
+
5
+ from typing import cast
6
+
7
+ import numpy as np
8
+ from numpy.typing import NDArray
9
+ from scipy.signal import resample
10
+
11
+ from peachy.streaming import AudioArray, audio_to_float32
12
+
13
+
14
+ def audio_to_mono(audio: AudioArray) -> AudioArray:
15
+ """Return a contiguous mono array while preserving dtype."""
16
+ if audio.ndim == 1:
17
+ return cast(AudioArray, np.ascontiguousarray(audio))
18
+ if audio.ndim != 2:
19
+ raise ValueError("Audio must be one- or two-dimensional")
20
+ if audio.size == 0:
21
+ return cast(AudioArray, np.empty(0, dtype=audio.dtype))
22
+ channels_last = audio.T if audio.shape[1] > audio.shape[0] else audio
23
+ mono = channels_last[:, 0]
24
+ return cast(AudioArray, np.ascontiguousarray(mono))
25
+
26
+
27
+ def resample_audio(
28
+ audio: AudioArray,
29
+ source_rate: int,
30
+ target_rate: int,
31
+ ) -> NDArray[np.float32]:
32
+ """Return contiguous mono float32 audio at `target_rate`."""
33
+ if source_rate <= 0 or target_rate <= 0:
34
+ raise ValueError("Audio sample rates must be positive")
35
+ mono = audio_to_float32(audio_to_mono(audio))
36
+ if mono.size == 0:
37
+ return np.ascontiguousarray(mono, dtype=np.float32)
38
+ if source_rate == target_rate:
39
+ return np.ascontiguousarray(mono, dtype=np.float32)
40
+ target_samples = round(mono.shape[0] * target_rate / source_rate)
41
+ if target_samples == 0:
42
+ return np.empty(0, dtype=np.float32)
43
+ converted = resample(mono, target_samples)
44
+ return np.ascontiguousarray(converted, dtype=np.float32)
45
+
46
+
47
+ def pcm16_at_rate(
48
+ audio: AudioArray,
49
+ source_rate: int,
50
+ target_rate: int,
51
+ ) -> NDArray[np.int16]:
52
+ """Return clipped, contiguous mono signed PCM16 at `target_rate`."""
53
+ converted = resample_audio(audio, source_rate, target_rate)
54
+ clipped = np.clip(converted, -1.0, 1.0)
55
+ return np.ascontiguousarray(clipped * 32767.0, dtype=np.int16)
src/peachy/audio/startup_config.py ADDED
@@ -0,0 +1,66 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Startup configuration for the Reachy Mini audio processor."""
2
+ # Modified by Peachy contributors in 2026 for namespace, packaging, and provider integration.
3
+
4
+
5
+ from __future__ import annotations
6
+ import logging
7
+ from collections.abc import Sequence
8
+
9
+
10
+ AudioControlValue = float | int
11
+ AudioStartupParameter = tuple[str, tuple[AudioControlValue, ...]]
12
+ WRITE_SETTLE_SECONDS = 0.1
13
+
14
+ AUDIO_STARTUP_CONFIG: tuple[AudioStartupParameter, ...] = (
15
+ ("PP_AGCMAXGAIN", (10.0,)),
16
+ ("PP_MIN_NS", (0.8,)),
17
+ ("PP_MIN_NN", (0.8,)),
18
+ ("PP_GAMMA_E", (0.5,)),
19
+ ("PP_GAMMA_ETAIL", (0.5,)),
20
+ ("PP_NLATTENONOFF", (0,)),
21
+ ("PP_MGSCALE", (4.0, 1.0, 1.0)),
22
+ )
23
+
24
+
25
+ def apply_audio_startup_config(
26
+ robot: object,
27
+ *,
28
+ logger: logging.Logger | None = None,
29
+ verify: bool = True,
30
+ write_settle_seconds: float = WRITE_SETTLE_SECONDS,
31
+ ) -> bool:
32
+ """Apply the tuned XVF3800 audio configuration for the conversation app."""
33
+ log = logger or logging.getLogger(__name__)
34
+ audio = getattr(getattr(robot, "media", None), "audio", None)
35
+
36
+ if audio is None:
37
+ log.warning("Skipping Reachy audio startup config: robot media audio is unavailable.")
38
+ return False
39
+
40
+ apply_audio_config = getattr(audio, "apply_audio_config", None)
41
+ if not callable(apply_audio_config):
42
+ log.warning("Skipping Reachy audio startup config: SDK audio config API is unavailable.")
43
+ return False
44
+
45
+ try:
46
+ applied = bool(
47
+ apply_audio_config(
48
+ AUDIO_STARTUP_CONFIG,
49
+ verify=verify,
50
+ write_settle_seconds=write_settle_seconds,
51
+ )
52
+ )
53
+ except Exception as exc:
54
+ log.warning("Skipping Reachy audio startup config: SDK audio config failed: %s", exc)
55
+ return False
56
+
57
+ if applied:
58
+ log.info("Applied Reachy audio startup config: %s", _format_config(AUDIO_STARTUP_CONFIG))
59
+ else:
60
+ log.warning("Reachy audio startup config was not applied.")
61
+
62
+ return applied
63
+
64
+
65
+ def _format_config(config: Sequence[AudioStartupParameter]) -> str:
66
+ return ", ".join(f"{name}={' '.join(str(value) for value in values)}" for name, values in config)
src/peachy/config.py ADDED
@@ -0,0 +1,489 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Modified by Peachy contributors in 2026 for namespace, packaging, and provider integration.
2
+ import os
3
+ import sys
4
+ import logging
5
+ from pathlib import Path
6
+ from dataclasses import dataclass
7
+ from urllib.parse import urlsplit, parse_qsl, urlunsplit
8
+ from importlib.resources import files
9
+
10
+ from dotenv import find_dotenv, load_dotenv
11
+ from peachy.state import resolve_state_root
12
+
13
+
14
+ # Locked profile: set to a profile name (e.g., "astronomer") to lock the app
15
+ # to that profile and disable all profile switching. Leave as None for normal behavior.
16
+ LOCKED_PROFILE: str | None = None
17
+ PROJECT_ROOT = Path(__file__).parents[2].resolve()
18
+
19
+
20
+ def _is_source_checkout_root(root: Path) -> bool:
21
+ """Return whether the given root looks like this project's source checkout."""
22
+ return (root / "pyproject.toml").is_file() and (root / "src" / "peachy").is_dir()
23
+
24
+
25
+ def _packaged_profiles_directory() -> Path | None:
26
+ """Return Peachy's wheel-owned built-in profiles directory."""
27
+ try:
28
+ return Path(str(files("peachy").joinpath("profiles")))
29
+ except Exception:
30
+ return None
31
+
32
+
33
+ def _resolve_default_profiles_directory() -> Path:
34
+ """Resolve built-in profiles from source checkout or installed package data."""
35
+ source_profiles = PROJECT_ROOT / "src" / "peachy" / "profiles"
36
+ if _is_source_checkout_root(PROJECT_ROOT) and source_profiles.is_dir():
37
+ return source_profiles
38
+
39
+ packaged_profiles = _packaged_profiles_directory()
40
+ if packaged_profiles is not None and packaged_profiles.is_dir():
41
+ return packaged_profiles
42
+
43
+ return source_profiles
44
+
45
+
46
+ DEFAULT_PROFILES_DIRECTORY = _resolve_default_profiles_directory()
47
+
48
+ # UI-created profiles live under a writable instance dir
49
+ USER_PERSONALITIES_DIRNAME = "user_personalities"
50
+
51
+ # Qwen3-TTS CustomVoice speaker catalog from the deployed Hugging Face backend.
52
+ HF_AVAILABLE_VOICES: list[str] = [
53
+ "Aiden",
54
+ "Ryan",
55
+ "Dylan",
56
+ "Eric",
57
+ "Ono_Anna",
58
+ "Serena",
59
+ "Sohee",
60
+ "Uncle_Fu",
61
+ "Vivian",
62
+ ]
63
+
64
+ HF_BACKEND = "huggingface"
65
+ HF_REALTIME_CONNECTION_MODE_ENV = "HF_REALTIME_CONNECTION_MODE"
66
+ HF_REALTIME_WS_URL_ENV = "HF_REALTIME_WS_URL"
67
+ REALTIME_TRANSCRIPTION_LANGUAGE_ENV = "REALTIME_TRANSCRIPTION_LANGUAGE"
68
+ HF_LOCAL_CONNECTION_MODE = "local"
69
+ HF_DEPLOYED_CONNECTION_MODE = "deployed"
70
+ HF_REALTIME_SESSION_PROXY_URL = "https://pollen-robotics-reachy-mini-realtime-url.hf.space/session"
71
+
72
+
73
+ @dataclass(frozen=True)
74
+ class HFBackendDefaults:
75
+ """Defaults for the Hugging Face realtime backend."""
76
+
77
+ connection_mode: str = HF_DEPLOYED_CONNECTION_MODE
78
+ # App-managed Hugging Face Space proxy. The Space forwards to the current
79
+ # session allocator, so allocator changes do not require app releases.
80
+ # Users who need a custom target should use HF_REALTIME_CONNECTION_MODE=local
81
+ # with HF_REALTIME_WS_URL.
82
+ session_url: str = HF_REALTIME_SESSION_PROXY_URL
83
+ voice: str = "Aiden"
84
+ direct_port: int = 8765
85
+
86
+
87
+ HF_DEFAULTS = HFBackendDefaults()
88
+
89
+ logger = logging.getLogger(__name__)
90
+
91
+ # Removed backend selectors kept in stale robot .env files: warn but ignore them.
92
+ _OBSOLETE_BACKEND_ENV_NAMES = ("BACKEND_PROVIDER", "MODEL_NAME")
93
+
94
+
95
+ def _warn_on_obsolete_backend_env() -> None:
96
+ """Warn when removed multi-backend selectors are still set; Hugging Face is the only backend."""
97
+ present = [name for name in _OBSOLETE_BACKEND_ENV_NAMES if (os.getenv(name) or "").strip()]
98
+ if present:
99
+ logger.warning(
100
+ "Ignoring obsolete backend environment variable(s): %s. This app now uses the Hugging Face backend only.",
101
+ ", ".join(present),
102
+ )
103
+
104
+
105
+ def _env_flag(name: str, default: bool = False) -> bool:
106
+ """Parse a boolean environment flag.
107
+
108
+ Accepted truthy values: 1, true, yes, on
109
+ Accepted falsy values: 0, false, no, off
110
+ """
111
+ raw = os.getenv(name)
112
+ if raw is None:
113
+ return default
114
+
115
+ value = raw.strip().lower()
116
+ if value in {"1", "true", "yes", "on"}:
117
+ return True
118
+ if value in {"0", "false", "no", "off"}:
119
+ return False
120
+
121
+ logger.warning("Invalid boolean value for %s=%r, using default=%s", name, raw, default)
122
+ return default
123
+
124
+
125
+ APP_TIMEOUT_MINUTES_ENV = "REACHY_MINI_APP_TIMEOUT_MINUTES"
126
+ DEFAULT_APP_TIMEOUT_MINUTES = 1440.0
127
+
128
+
129
+ def resolve_app_timeout_minutes() -> float | None:
130
+ """Read the app inactivity timeout (minutes) from the environment; None means disabled."""
131
+ raw_value = os.getenv(APP_TIMEOUT_MINUTES_ENV, "").strip()
132
+ if not raw_value:
133
+ return DEFAULT_APP_TIMEOUT_MINUTES
134
+ try:
135
+ timeout_minutes = float(raw_value)
136
+ except ValueError:
137
+ logger.warning("Ignoring invalid %s=%r; using default.", APP_TIMEOUT_MINUTES_ENV, raw_value)
138
+ return DEFAULT_APP_TIMEOUT_MINUTES
139
+ return timeout_minutes if timeout_minutes > 0 else None
140
+
141
+
142
+ def _normalize_hf_connection_mode(value: str | None) -> str | None:
143
+ """Normalize the Hugging Face connection mode, if explicitly configured."""
144
+ candidate = (value or "").strip().lower()
145
+ if not candidate:
146
+ return None
147
+
148
+ if candidate not in {HF_LOCAL_CONNECTION_MODE, HF_DEPLOYED_CONNECTION_MODE}:
149
+ logger.warning(
150
+ "Invalid %s=%r. Expected local or deployed.",
151
+ HF_REALTIME_CONNECTION_MODE_ENV,
152
+ value,
153
+ )
154
+ return None
155
+ return candidate
156
+
157
+
158
+ def _normalize_transcription_language(value: str | None) -> str:
159
+ """Return the configured realtime transcription language."""
160
+ candidate = (value or "").strip()
161
+ return candidate or "en"
162
+
163
+
164
+ @dataclass(frozen=True)
165
+ class HFConnectionSelection:
166
+ """Resolved Hugging Face connection mode and target availability."""
167
+
168
+ mode: str
169
+ has_target: bool
170
+ session_url: str | None = None
171
+ direct_ws_url: str | None = None
172
+
173
+
174
+ @dataclass(frozen=True)
175
+ class HFRealtimeURLParts:
176
+ """Parsed Hugging Face realtime URL components used by UI and client setup."""
177
+
178
+ base_url: str
179
+ websocket_base_url: str
180
+ connect_query: dict[str, str]
181
+ host: str | None
182
+ port: int | None
183
+ has_realtime_path: bool
184
+
185
+
186
+ def parse_hf_realtime_url(realtime_url: str) -> HFRealtimeURLParts:
187
+ """Parse a Hugging Face realtime URL into OpenAI-compatible client endpoints."""
188
+ parsed = urlsplit(realtime_url)
189
+ scheme = parsed.scheme.lower()
190
+ if scheme not in {"ws", "wss", "http", "https"}:
191
+ raise ValueError(
192
+ "Expected Hugging Face realtime URL to start with ws://, wss://, http://, or https://, "
193
+ f"got: {realtime_url}"
194
+ )
195
+
196
+ path = parsed.path.rstrip("/")
197
+ has_realtime_path = path.endswith("/realtime")
198
+ if has_realtime_path:
199
+ base_path = path[: -len("/realtime")]
200
+ else:
201
+ base_path = path
202
+
203
+ connect_query = {key: value for key, value in parse_qsl(parsed.query, keep_blank_values=True) if key != "model"}
204
+ http_scheme = "https" if scheme in {"wss", "https"} else "http"
205
+ websocket_scheme = "wss" if scheme in {"wss", "https"} else "ws"
206
+ base_url = urlunsplit((http_scheme, parsed.netloc, base_path, "", ""))
207
+ websocket_base_url = urlunsplit((websocket_scheme, parsed.netloc, base_path, "", ""))
208
+ return HFRealtimeURLParts(
209
+ base_url=base_url,
210
+ websocket_base_url=websocket_base_url,
211
+ connect_query=connect_query,
212
+ host=parsed.hostname,
213
+ port=parsed.port or HF_DEFAULTS.direct_port,
214
+ has_realtime_path=has_realtime_path,
215
+ )
216
+
217
+
218
+ def parse_hf_direct_target(ws_url: str | None) -> tuple[str | None, int | None]:
219
+ """Extract host and port from a direct Hugging Face realtime URL."""
220
+ if not ws_url:
221
+ return None, None
222
+ try:
223
+ parsed = parse_hf_realtime_url(ws_url)
224
+ return parsed.host, parsed.port
225
+ except Exception:
226
+ return None, None
227
+
228
+
229
+ def build_hf_direct_ws_url(host: str, port: int) -> str:
230
+ """Build the direct Hugging Face realtime websocket URL used by the app."""
231
+ return f"ws://{host}:{port}/v1/realtime"
232
+
233
+
234
+ def _collect_profile_names(profiles_root: Path) -> set[str]:
235
+ """Return profile folder names from a profiles root directory."""
236
+ if not profiles_root.exists() or not profiles_root.is_dir():
237
+ return set()
238
+ return {p.name for p in profiles_root.iterdir() if p.is_dir()}
239
+
240
+
241
+ def _collect_tool_module_names(tools_root: Path) -> set[str]:
242
+ """Return tool module names from a tools directory."""
243
+ if not tools_root.exists() or not tools_root.is_dir():
244
+ return set()
245
+ ignored = {"__init__", "core_tools"}
246
+ return {p.stem for p in tools_root.glob("*.py") if p.is_file() and p.stem not in ignored}
247
+
248
+
249
+ def _raise_on_name_collisions(
250
+ *,
251
+ label: str,
252
+ external_root: Path,
253
+ internal_root: Path,
254
+ external_names: set[str],
255
+ internal_names: set[str],
256
+ ) -> None:
257
+ """Raise with a clear message when external/internal names collide."""
258
+ collisions = sorted(external_names & internal_names)
259
+ if not collisions:
260
+ return
261
+
262
+ raise RuntimeError(
263
+ f"Config.__init__(): Ambiguous {label} names found in both external and built-in libraries: {collisions}. "
264
+ f"External {label} root: {external_root}. Built-in {label} root: {internal_root}. "
265
+ f"Please rename the conflicting external {label}(s) to continue."
266
+ )
267
+
268
+
269
+ # Validate LOCKED_PROFILE at startup
270
+ if LOCKED_PROFILE is not None:
271
+ _profiles_dir = DEFAULT_PROFILES_DIRECTORY
272
+ _profile_path = _profiles_dir / LOCKED_PROFILE
273
+ _instructions_file = _profile_path / "instructions.txt"
274
+ if not _profile_path.is_dir():
275
+ logger.critical("LOCKED_PROFILE %r does not exist in %s", LOCKED_PROFILE, _profiles_dir)
276
+ sys.exit(1)
277
+ if not _instructions_file.is_file():
278
+ logger.critical("LOCKED_PROFILE %r has no instructions.txt", LOCKED_PROFILE)
279
+ sys.exit(1)
280
+
281
+ _skip_dotenv = _env_flag("REACHY_MINI_SKIP_DOTENV", default=False)
282
+
283
+ if _skip_dotenv:
284
+ logger.info("Skipping .env loading because REACHY_MINI_SKIP_DOTENV is set")
285
+ else:
286
+ # Locate .env file (search upward from current working directory)
287
+ dotenv_path = find_dotenv(usecwd=True)
288
+
289
+ if dotenv_path:
290
+ # Load .env and override environment variables
291
+ load_dotenv(dotenv_path=dotenv_path, override=True)
292
+ logger.info(f"Configuration loaded from {dotenv_path}")
293
+ else:
294
+ logger.warning("No .env file found, using environment variables")
295
+
296
+ _warn_on_obsolete_backend_env()
297
+
298
+
299
+ class Config:
300
+ """Configuration class for the conversation app."""
301
+
302
+ HF_REALTIME_CONNECTION_MODE = (
303
+ _normalize_hf_connection_mode(os.getenv(HF_REALTIME_CONNECTION_MODE_ENV)) or HF_DEFAULTS.connection_mode
304
+ )
305
+ # Deliberately ignore HF_REALTIME_SESSION_URL from the environment; the app-managed proxy is HF_DEFAULTS.session_url.
306
+ HF_REALTIME_SESSION_URL = HF_DEFAULTS.session_url
307
+ HF_REALTIME_WS_URL = os.getenv(HF_REALTIME_WS_URL_ENV)
308
+ REALTIME_TRANSCRIPTION_LANGUAGE = _normalize_transcription_language(os.getenv(REALTIME_TRANSCRIPTION_LANGUAGE_ENV))
309
+ HF_TOKEN = os.getenv("HF_TOKEN") # Optional, falls back to hf auth login if not set
310
+
311
+ logger.debug(
312
+ "HF mode: %s, HF session URL set: %s, HF direct URL set: %s",
313
+ HF_REALTIME_CONNECTION_MODE,
314
+ bool(HF_REALTIME_SESSION_URL and HF_REALTIME_SESSION_URL.strip()),
315
+ bool(HF_REALTIME_WS_URL and HF_REALTIME_WS_URL.strip()),
316
+ )
317
+
318
+ # Filesystem root containing profile directories, not a Python import path.
319
+ _profiles_directory_env = os.getenv("REACHY_MINI_EXTERNAL_PROFILES_DIRECTORY")
320
+ PROFILES_DIRECTORY = Path(_profiles_directory_env) if _profiles_directory_env else DEFAULT_PROFILES_DIRECTORY
321
+ INSTANCE_PATH: Path | None = None # set at startup; writable home for UI-created profiles
322
+ _tools_directory_env = os.getenv("REACHY_MINI_EXTERNAL_TOOLS_DIRECTORY")
323
+ TOOLS_DIRECTORY = Path(_tools_directory_env) if _tools_directory_env else None
324
+ AUTOLOAD_EXTERNAL_TOOLS = _env_flag("AUTOLOAD_EXTERNAL_TOOLS", default=False)
325
+ REACHY_MINI_CUSTOM_PROFILE = LOCKED_PROFILE or os.getenv("REACHY_MINI_CUSTOM_PROFILE")
326
+
327
+ logger.debug(f"Custom Profile: {REACHY_MINI_CUSTOM_PROFILE}")
328
+
329
+ def __init__(self) -> None:
330
+ """Initialize the configuration."""
331
+ if self.REACHY_MINI_CUSTOM_PROFILE and self.PROFILES_DIRECTORY != DEFAULT_PROFILES_DIRECTORY:
332
+ selected_profile_path = self.PROFILES_DIRECTORY / self.REACHY_MINI_CUSTOM_PROFILE
333
+ if not selected_profile_path.is_dir():
334
+ available_profiles = sorted(_collect_profile_names(self.PROFILES_DIRECTORY))
335
+ raise RuntimeError(
336
+ "Config.__init__(): Selected profile "
337
+ f"'{self.REACHY_MINI_CUSTOM_PROFILE}' was not found in external profiles root "
338
+ f"{self.PROFILES_DIRECTORY}. "
339
+ f"Available external profiles: {available_profiles}. "
340
+ "Either set 'REACHY_MINI_CUSTOM_PROFILE' to one of the available external profiles "
341
+ "or unset 'REACHY_MINI_EXTERNAL_PROFILES_DIRECTORY' to use built-in profiles."
342
+ )
343
+
344
+ if self.PROFILES_DIRECTORY != DEFAULT_PROFILES_DIRECTORY:
345
+ external_profiles = _collect_profile_names(self.PROFILES_DIRECTORY)
346
+ internal_profiles = _collect_profile_names(DEFAULT_PROFILES_DIRECTORY)
347
+ _raise_on_name_collisions(
348
+ label="profile",
349
+ external_root=self.PROFILES_DIRECTORY,
350
+ internal_root=DEFAULT_PROFILES_DIRECTORY,
351
+ external_names=external_profiles,
352
+ internal_names=internal_profiles,
353
+ )
354
+
355
+ if self.TOOLS_DIRECTORY is not None:
356
+ builtin_tools_root = Path(__file__).parent / "tools"
357
+ external_tools = _collect_tool_module_names(self.TOOLS_DIRECTORY)
358
+ internal_tools = _collect_tool_module_names(builtin_tools_root)
359
+ _raise_on_name_collisions(
360
+ label="tool",
361
+ external_root=self.TOOLS_DIRECTORY,
362
+ internal_root=builtin_tools_root,
363
+ external_names=external_tools,
364
+ internal_names=internal_tools,
365
+ )
366
+
367
+ if self.PROFILES_DIRECTORY != DEFAULT_PROFILES_DIRECTORY:
368
+ logger.warning(
369
+ "Environment variable 'REACHY_MINI_EXTERNAL_PROFILES_DIRECTORY' is set. "
370
+ "Profiles (instructions.txt, ...) will be loaded from %s.",
371
+ self.PROFILES_DIRECTORY,
372
+ )
373
+ else:
374
+ logger.info(
375
+ "'REACHY_MINI_EXTERNAL_PROFILES_DIRECTORY' is not set. Using built-in profiles from %s.",
376
+ DEFAULT_PROFILES_DIRECTORY,
377
+ )
378
+
379
+ if self.TOOLS_DIRECTORY is not None:
380
+ logger.warning(
381
+ "Environment variable 'REACHY_MINI_EXTERNAL_TOOLS_DIRECTORY' is set. "
382
+ "External tools will be loaded from %s.",
383
+ self.TOOLS_DIRECTORY,
384
+ )
385
+ else:
386
+ logger.info("'REACHY_MINI_EXTERNAL_TOOLS_DIRECTORY' is not set. Using built-in shared tools only.")
387
+
388
+ def user_personalities_root(self) -> Path:
389
+ """Writable root for UI-created profiles."""
390
+ base = self.INSTANCE_PATH if self.INSTANCE_PATH is not None else resolve_state_root()
391
+ return Path(base) / USER_PERSONALITIES_DIRNAME
392
+
393
+ def resolve_profile_dir(self, profile: str) -> Path:
394
+ """On-disk directory for a profile selection."""
395
+ head, _, tail = profile.partition("/")
396
+ if head == USER_PERSONALITIES_DIRNAME and tail:
397
+ return self.user_personalities_root() / tail
398
+ return self.PROFILES_DIRECTORY / profile
399
+
400
+
401
+ config = Config()
402
+
403
+
404
+ def refresh_runtime_config_from_env() -> None:
405
+ """Refresh mutable runtime config fields from the current environment."""
406
+ _warn_on_obsolete_backend_env()
407
+ config.HF_REALTIME_CONNECTION_MODE = (
408
+ _normalize_hf_connection_mode(os.getenv(HF_REALTIME_CONNECTION_MODE_ENV)) or HF_DEFAULTS.connection_mode
409
+ )
410
+ # Deliberately ignore HF_REALTIME_SESSION_URL from the environment; the app-managed proxy is HF_DEFAULTS.session_url.
411
+ config.HF_REALTIME_SESSION_URL = HF_DEFAULTS.session_url
412
+ config.HF_REALTIME_WS_URL = os.getenv(HF_REALTIME_WS_URL_ENV)
413
+ config.REALTIME_TRANSCRIPTION_LANGUAGE = _normalize_transcription_language(
414
+ os.getenv(REALTIME_TRANSCRIPTION_LANGUAGE_ENV)
415
+ )
416
+ config.HF_TOKEN = os.getenv("HF_TOKEN")
417
+ config.REACHY_MINI_CUSTOM_PROFILE = LOCKED_PROFILE or os.getenv("REACHY_MINI_CUSTOM_PROFILE")
418
+
419
+
420
+ def get_available_voices() -> list[str]:
421
+ """Return the curated Hugging Face voice list."""
422
+ return list(HF_AVAILABLE_VOICES)
423
+
424
+
425
+ def get_default_voice() -> str:
426
+ """Return the default Hugging Face voice."""
427
+ return HF_DEFAULTS.voice
428
+
429
+
430
+ def get_hf_session_url() -> str | None:
431
+ """Return the built-in Hugging Face session proxy URL, if any."""
432
+ value = (getattr(config, "HF_REALTIME_SESSION_URL", None) or "").strip()
433
+ return value or None
434
+
435
+
436
+ def get_hf_direct_ws_url() -> str | None:
437
+ """Return the configured direct Hugging Face realtime URL, if any."""
438
+ value = (getattr(config, "HF_REALTIME_WS_URL", None) or "").strip()
439
+ return value or None
440
+
441
+
442
+ def get_hf_connection_selection() -> HFConnectionSelection:
443
+ """Resolve the selected Hugging Face connection mode and whether it is usable."""
444
+ session_url = get_hf_session_url()
445
+ direct_ws_url = get_hf_direct_ws_url()
446
+ mode = _normalize_hf_connection_mode(getattr(config, "HF_REALTIME_CONNECTION_MODE", None))
447
+ if mode is None:
448
+ raise RuntimeError(f"{HF_REALTIME_CONNECTION_MODE_ENV} must be set to local or deployed.")
449
+
450
+ target = direct_ws_url if mode == HF_LOCAL_CONNECTION_MODE else session_url
451
+
452
+ return HFConnectionSelection(
453
+ mode=mode,
454
+ has_target=bool(target),
455
+ session_url=session_url,
456
+ direct_ws_url=direct_ws_url,
457
+ )
458
+
459
+
460
+ def has_hf_realtime_target() -> bool:
461
+ """Return whether Hugging Face has a target for the selected mode."""
462
+ return get_hf_connection_selection().has_target
463
+
464
+
465
+ def set_instance_path(instance_path: str | Path | None) -> None:
466
+ """Record the app instance dir so UI-created profiles persist outside package data."""
467
+ config.INSTANCE_PATH = Path(instance_path) if instance_path else None
468
+
469
+
470
+ def set_custom_profile(profile: str | None) -> None:
471
+ """Update the selected custom profile at runtime and expose it via env.
472
+
473
+ This ensures modules that read `config` and code that inspects the
474
+ environment see a consistent value.
475
+ """
476
+ if LOCKED_PROFILE is not None:
477
+ return
478
+ try:
479
+ config.REACHY_MINI_CUSTOM_PROFILE = profile
480
+ except Exception as e:
481
+ logger.warning("Failed to update config profile: %s", e)
482
+ try:
483
+ if profile:
484
+ os.environ["REACHY_MINI_CUSTOM_PROFILE"] = profile
485
+ else:
486
+ # Remove to reflect default
487
+ os.environ.pop("REACHY_MINI_CUSTOM_PROFILE", None)
488
+ except Exception as e:
489
+ logger.warning("Failed to sync profile to environment: %s", e)
src/peachy/console.py ADDED
@@ -0,0 +1,1005 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Bidirectional local audio stream with optional web settings UI.
2
+
3
+ If the selected backend is missing its required API key, a settings page is
4
+ served via the Reachy Mini Apps settings server so users can configure it.
5
+ """
6
+ # Modified by Peachy contributors in 2026 for namespace, packaging, and provider integration.
7
+
8
+
9
+ import os
10
+ import time
11
+ import asyncio
12
+ import logging
13
+ import tempfile
14
+ import threading
15
+ from typing import List, Optional
16
+ from pathlib import Path
17
+ from collections.abc import Callable, AsyncGenerator
18
+
19
+ from reachy_mini import ReachyMini
20
+ from reachy_mini.media.media_manager import MediaBackend
21
+ from peachy.config import (
22
+ LOCKED_PROFILE,
23
+ HF_REALTIME_WS_URL_ENV,
24
+ HF_LOCAL_CONNECTION_MODE,
25
+ HF_DEPLOYED_CONNECTION_MODE,
26
+ HF_REALTIME_CONNECTION_MODE_ENV,
27
+ config,
28
+ get_default_voice,
29
+ get_hf_session_url,
30
+ build_hf_direct_ws_url,
31
+ parse_hf_direct_target,
32
+ get_hf_connection_selection,
33
+ refresh_runtime_config_from_env,
34
+ )
35
+ from peachy.streaming import AdditionalOutputs
36
+ from peachy.audio.resampling import resample_audio
37
+ from peachy.startup_settings import read_startup_settings, write_startup_settings
38
+ from peachy.tools.core_tools import initialize_tools
39
+ from peachy.personality_routes import mount_personality_routes
40
+ from peachy.audio.startup_config import apply_audio_startup_config
41
+ from peachy.conversation_handler import ConversationHandler
42
+ from peachy.providers import (
43
+ OPENAI_DEFAULT_MODEL,
44
+ OPENAI_MODEL_ENV,
45
+ OPENAI_PROVIDER,
46
+ PROVIDER_ENV,
47
+ ProviderDefinition,
48
+ available_provider_names,
49
+ get_provider_definition,
50
+ redact_configured_credentials,
51
+ redact_secrets,
52
+ resolve_provider_name,
53
+ )
54
+
55
+
56
+ try:
57
+ # FastAPI is provided by the Reachy Mini Apps runtime
58
+ from fastapi import FastAPI, Request, Response
59
+ from pydantic import BaseModel
60
+ from fastapi.responses import FileResponse, JSONResponse, StreamingResponse
61
+ from starlette.staticfiles import StaticFiles
62
+ except Exception: # pragma: no cover - only loaded when settings_app is used
63
+ FastAPI = object # type: ignore
64
+ FileResponse = object # type: ignore
65
+ JSONResponse = object # type: ignore
66
+ StreamingResponse = object # type: ignore
67
+ StaticFiles = object # type: ignore
68
+ BaseModel = object # type: ignore
69
+ Request = object # type: ignore
70
+
71
+ logger = logging.getLogger(__name__)
72
+
73
+ _SSE_KEEPALIVE_INTERVAL_SECONDS = 15.0 # below typical 60s proxy idle timeout
74
+ SETTINGS_API_PREFIX = "/api/v1"
75
+
76
+
77
+ def _detach_framework_root_routes(app: "FastAPI") -> None:
78
+ """Strip framework routes that would shadow the settings UI."""
79
+ routes = getattr(app, "router", None)
80
+ routes = getattr(routes, "routes", None) if routes else getattr(app, "routes", None)
81
+ if routes is None:
82
+ return
83
+ survivors = []
84
+ for route in routes:
85
+ path = getattr(route, "path", None)
86
+ is_catch_all = isinstance(path, str) and path.startswith("/{") and path.endswith(":path}")
87
+ if path in ("/", "/static") or is_catch_all:
88
+ logger.debug("detaching framework-provided route %r (%s)", path, type(route).__name__)
89
+ continue
90
+ survivors.append(route)
91
+ routes[:] = survivors
92
+
93
+
94
+ class ConversationEventBus:
95
+ """Thread-safe fan-out bus that delivers conversation activity events to SSE subscribers."""
96
+
97
+ MAX_QUEUE_SIZE = 64
98
+
99
+ def __init__(self) -> None:
100
+ """Initialize the bus."""
101
+ # Each entry is (loop, queue); loop captured at subscribe() time for cross-thread delivery.
102
+ self._subscribers: list[tuple[asyncio.AbstractEventLoop, "asyncio.Queue[str]"]] = []
103
+ self._lock = threading.Lock()
104
+
105
+ def subscribe(self) -> tuple["asyncio.Queue[str]", Callable[[], None]]:
106
+ """Return a per-subscriber queue and its unsubscribe callback. Must be called from a coroutine."""
107
+ loop = asyncio.get_running_loop()
108
+ queue: "asyncio.Queue[str]" = asyncio.Queue(maxsize=self.MAX_QUEUE_SIZE)
109
+ entry = (loop, queue)
110
+ with self._lock:
111
+ self._subscribers.append(entry)
112
+
113
+ def unsubscribe() -> None:
114
+ with self._lock:
115
+ try:
116
+ self._subscribers.remove(entry)
117
+ except ValueError:
118
+ pass
119
+
120
+ return queue, unsubscribe
121
+
122
+ def publish(self, event: str) -> None:
123
+ """Broadcast to all subscribers via call_soon_threadsafe. Drops events for full or closed queues."""
124
+ with self._lock:
125
+ snapshot = list(self._subscribers)
126
+ for loop, queue in snapshot:
127
+ try:
128
+ loop.call_soon_threadsafe(self._enqueue_safely, queue, event)
129
+ except RuntimeError:
130
+ pass # loop closed; subscriber will clean itself up on disconnect
131
+
132
+ @staticmethod
133
+ def _enqueue_safely(queue: "asyncio.Queue[str]", event: str) -> None:
134
+ try:
135
+ queue.put_nowait(event)
136
+ except asyncio.QueueFull:
137
+ logger.debug("conversation event dropped (subscriber queue full): %s", event)
138
+
139
+
140
+ async def _conversation_events_stream(
141
+ bus: ConversationEventBus,
142
+ request: "Request",
143
+ ) -> "AsyncGenerator[str, None]":
144
+ """Yield SSE lines for one subscriber; emits keep-alive comments during idle periods."""
145
+ queue, unsubscribe = bus.subscribe()
146
+ try:
147
+ yield "retry: 2000\n\n"
148
+ yield "event: ready\ndata: connected\n\n"
149
+
150
+ while True:
151
+ if await request.is_disconnected():
152
+ return
153
+ try:
154
+ event = await asyncio.wait_for(queue.get(), timeout=_SSE_KEEPALIVE_INTERVAL_SECONDS)
155
+ except asyncio.TimeoutError:
156
+ yield ": keep-alive\n\n"
157
+ continue
158
+ yield f"event: activity\ndata: {event}\n\n"
159
+ finally:
160
+ unsubscribe()
161
+
162
+
163
+ LOCAL_PLAYER_BACKEND = (
164
+ getattr(MediaBackend, "LOCAL", None)
165
+ or getattr(MediaBackend, "GSTREAMER", None)
166
+ or getattr(MediaBackend, "DEFAULT", None)
167
+ )
168
+
169
+ HandlerFactory = Callable[[Optional[str]], ConversationHandler]
170
+ ProviderResolver = Callable[[], ProviderDefinition]
171
+
172
+ LEGACY_STARTUP_ENV_NAMES = (
173
+ "REACHY_MINI_CUSTOM_PROFILE",
174
+ "REACHY_MINI_VOICE_OVERRIDE",
175
+ )
176
+ BACKEND_RETRY_DELAY_SECONDS = 5.0
177
+
178
+
179
+ class LocalStream:
180
+ """LocalStream using Reachy Mini's recorder/player."""
181
+
182
+ def __init__(
183
+ self,
184
+ handler: ConversationHandler,
185
+ robot: ReachyMini,
186
+ *,
187
+ settings_app: Optional[FastAPI] = None,
188
+ instance_path: Optional[str] = None,
189
+ handler_factory: HandlerFactory | None = None,
190
+ provider_resolver: ProviderResolver = get_provider_definition,
191
+ startup_voice: Optional[str] = None,
192
+ ):
193
+ """Initialize the stream with a realtime handler and pipelines.
194
+
195
+ - ``settings_app``: the Reachy Mini Apps FastAPI to attach settings endpoints.
196
+ - ``instance_path``: directory where per-instance ``.env`` should be stored.
197
+ - ``handler_factory``: builds a fresh handler for the currently selected backend.
198
+ - ``provider_resolver``: returns the current provider definition and readiness.
199
+ """
200
+ self._robot = robot
201
+ self._stop_event = asyncio.Event()
202
+ self._restart_requested = asyncio.Event()
203
+ self._tasks: List[asyncio.Task[None]] = []
204
+ self._handler_factory = handler_factory
205
+ self._provider_resolver = provider_resolver
206
+ self._voice_override = startup_voice
207
+ self._settings_app: Optional[FastAPI] = settings_app
208
+ self._instance_path: Optional[str] = instance_path
209
+ self._settings_initialized = False
210
+ self._asyncio_loop = None
211
+ self._mic_muted = False # mic starts live; the UI toggles it via the settings API
212
+ self._backend_connection_state = "not_started"
213
+ self._backend_error: str | None = None
214
+ self._backend_retry_delay = BACKEND_RETRY_DELAY_SECONDS
215
+ # One bus for the stream's lifetime: the conversation events route
216
+ # closes over it, so handler rebuilds must keep publishing into it.
217
+ self._event_bus = ConversationEventBus()
218
+ self._install_handler(handler)
219
+
220
+ def _install_handler(self, handler: ConversationHandler) -> None:
221
+ """Set the active handler and wire LocalStream-owned helpers into it."""
222
+ self.handler = handler
223
+ self.handler._clear_queue = self.clear_audio_queue
224
+ self._attach_event_bus_to_handler()
225
+
226
+ def _attach_event_bus_to_handler(self) -> None:
227
+ """Wire the event bus as the handler's activity observer, if supported."""
228
+ setter = getattr(self.handler, "set_activity_observer", None)
229
+ if callable(setter):
230
+ setter(self._event_bus.publish)
231
+
232
+ def seconds_since_activity(self) -> float:
233
+ """Seconds since the live handler last saw conversation activity."""
234
+ return time.monotonic() - self.handler.last_activity_time
235
+
236
+ def _read_env_lines(self, env_path: Path) -> list[str]:
237
+ """Load env file contents or a template as a list of lines."""
238
+ inst = env_path.parent
239
+ try:
240
+ if env_path.exists():
241
+ try:
242
+ return env_path.read_text(encoding="utf-8").splitlines()
243
+ except Exception:
244
+ return []
245
+ template_text = None
246
+ ex = inst / ".env.example"
247
+ if ex.exists():
248
+ try:
249
+ template_text = ex.read_text(encoding="utf-8")
250
+ except Exception:
251
+ template_text = None
252
+ if template_text is None:
253
+ try:
254
+ cwd_example = Path.cwd() / ".env.example"
255
+ if cwd_example.exists():
256
+ template_text = cwd_example.read_text(encoding="utf-8")
257
+ except Exception:
258
+ template_text = None
259
+ if template_text is None:
260
+ packaged = Path(__file__).parent / ".env.example"
261
+ if packaged.exists():
262
+ try:
263
+ template_text = packaged.read_text(encoding="utf-8")
264
+ except Exception:
265
+ template_text = None
266
+ return template_text.splitlines() if template_text else []
267
+ except Exception:
268
+ return []
269
+
270
+ def _backend_connected(self) -> bool:
271
+ """Return whether the active handler currently has a realtime connection."""
272
+ try:
273
+ handler_state = vars(self.handler)
274
+ except TypeError:
275
+ handler_state = {}
276
+ return handler_state.get("connection") is not None
277
+
278
+ def _can_rebuild_handler(self) -> bool:
279
+ """Return whether LocalStream can construct handlers for backend changes."""
280
+ return self._handler_factory is not None
281
+
282
+ def _build_handler_for_current_backend(self) -> ConversationHandler:
283
+ """Create and install a fresh handler for the current runtime backend config."""
284
+ if self._handler_factory is None:
285
+ return self.handler
286
+ handler = self._handler_factory(self._voice_override)
287
+ self._install_handler(handler)
288
+ return handler
289
+
290
+ async def _shutdown_active_handler(self) -> None:
291
+ """Best-effort shutdown for the currently active handler."""
292
+ try:
293
+ await self.handler.shutdown()
294
+ except asyncio.CancelledError:
295
+ raise
296
+ except Exception as e:
297
+ logger.debug(
298
+ "Active handler shutdown ignored during restart: %s",
299
+ redact_configured_credentials(e),
300
+ )
301
+
302
+ def _mark_restart_requested(self, reason: str) -> None:
303
+ """Request a backend restart from a synchronous route handler."""
304
+ logger.info("Backend restart requested: %s", reason)
305
+ self._set_backend_connection_state("connecting")
306
+ loop = self._asyncio_loop
307
+ if loop is not None and loop.is_running():
308
+ asyncio.run_coroutine_threadsafe(self.request_backend_restart(reason), loop)
309
+ return
310
+ self._restart_requested.set()
311
+
312
+ async def request_backend_restart(self, reason: str) -> None:
313
+ """Ask the startup loop to rebuild the backend and stop the current handler."""
314
+ self._set_backend_connection_state("connecting")
315
+ self._restart_requested.set()
316
+ self.clear_audio_queue()
317
+ await self._shutdown_active_handler()
318
+
319
+ async def _sleep_or_restart_requested(self, delay: float) -> None:
320
+ """Sleep for a retry interval, waking early if a restart is requested."""
321
+ if self._restart_requested.is_set():
322
+ return
323
+ try:
324
+ await asyncio.wait_for(self._restart_requested.wait(), timeout=delay)
325
+ except asyncio.TimeoutError:
326
+ pass
327
+
328
+ async def _wait_for_restart_or_stop(self) -> None:
329
+ """Wait until configuration/manual restart or app shutdown."""
330
+ while not self._stop_event.is_set() and not self._restart_requested.is_set():
331
+ await asyncio.sleep(0.1)
332
+
333
+ @staticmethod
334
+ def _format_backend_error(error: BaseException | str) -> str:
335
+ """Return a compact user-facing backend error string."""
336
+ if isinstance(error, str):
337
+ return redact_secrets(error)
338
+ message = str(error).strip()
339
+ rendered = f"{type(error).__name__}: {message}" if message else type(error).__name__
340
+ return redact_secrets(rendered)
341
+
342
+ def _set_backend_connection_state(self, state: str, error: BaseException | str | None = None) -> None:
343
+ """Update backend connection status exposed through the settings UI."""
344
+ self._backend_connection_state = state
345
+ if error is not None:
346
+ self._backend_error = self._format_backend_error(error)
347
+ elif state != "disconnected":
348
+ self._backend_error = None
349
+
350
+ def _backend_connection_status(self) -> dict[str, object]:
351
+ """Return the backend connection state exposed in the settings API."""
352
+ connected = self._backend_connected()
353
+ state = "connected" if connected else self._backend_connection_state
354
+ return {
355
+ "backend_connected": connected,
356
+ "backend_connection_state": state,
357
+ "backend_error": None if connected else self._backend_error,
358
+ }
359
+
360
+ @staticmethod
361
+ def _apply_runtime_env_values(
362
+ updates: dict[str, str],
363
+ removals: tuple[str, ...] = (),
364
+ ) -> None:
365
+ """Apply runtime environment updates, rolling back on refresh failure."""
366
+ affected_names = set(updates).union(removals)
367
+ previous_values = {name: os.environ.get(name) for name in affected_names}
368
+ try:
369
+ for env_name in removals:
370
+ os.environ.pop(env_name, None)
371
+ for env_name, value in updates.items():
372
+ os.environ[env_name] = value
373
+ refresh_runtime_config_from_env()
374
+ except Exception:
375
+ for env_name, previous in previous_values.items():
376
+ if previous is None:
377
+ os.environ.pop(env_name, None)
378
+ else:
379
+ os.environ[env_name] = previous
380
+ try:
381
+ refresh_runtime_config_from_env()
382
+ except Exception:
383
+ pass
384
+ raise
385
+
386
+ @staticmethod
387
+ def _write_env_file_atomically(env_path: Path, text: str) -> None:
388
+ """Securely replace an env file without exposing a partial credential write."""
389
+ file_descriptor, temporary_name = tempfile.mkstemp(
390
+ prefix=f".{env_path.name}.",
391
+ suffix=".tmp",
392
+ dir=env_path.parent,
393
+ )
394
+ os.close(file_descriptor)
395
+ temporary_path = Path(temporary_name)
396
+ try:
397
+ temporary_path.write_text(text, encoding="utf-8")
398
+ temporary_path.chmod(0o600)
399
+ temporary_path.replace(env_path)
400
+ finally:
401
+ temporary_path.unlink(missing_ok=True)
402
+
403
+ def _persist_env_values(
404
+ self,
405
+ updates: dict[str, str],
406
+ *,
407
+ removals: tuple[str, ...] = (),
408
+ ) -> None:
409
+ """Persist environment updates/removals transactionally, then refresh runtime config."""
410
+ normalized_updates = {name: (value or "").strip() for name, value in updates.items()}
411
+ normalized_updates = {name: value for name, value in normalized_updates.items() if value}
412
+ normalized_removals = tuple(
413
+ sorted(
414
+ {
415
+ name.strip()
416
+ for name in removals
417
+ if name and name.strip() and name.strip() not in normalized_updates
418
+ }
419
+ )
420
+ )
421
+ if not normalized_updates and not normalized_removals:
422
+ return
423
+
424
+ if not self._instance_path:
425
+ self._apply_runtime_env_values(normalized_updates, normalized_removals)
426
+ return
427
+
428
+ inst = Path(self._instance_path)
429
+ env_path = inst / ".env"
430
+ original_exists = env_path.exists()
431
+ original_text = env_path.read_text(encoding="utf-8") if original_exists else None
432
+ lines = original_text.splitlines() if original_text is not None else self._read_env_lines(env_path)
433
+ lines = [
434
+ line
435
+ for line in lines
436
+ if not any(line.strip().startswith(f"{env_name}=") for env_name in normalized_removals)
437
+ ]
438
+ for env_name, value in normalized_updates.items():
439
+ replaced = False
440
+ for i, line in enumerate(lines):
441
+ if line.strip().startswith(f"{env_name}="):
442
+ lines[i] = f"{env_name}={value}"
443
+ replaced = True
444
+ break
445
+ if not replaced:
446
+ lines.append(f"{env_name}={value}")
447
+ final_text = "\n".join(lines) + "\n"
448
+
449
+ self._write_env_file_atomically(env_path, final_text)
450
+ try:
451
+ self._apply_runtime_env_values(normalized_updates, normalized_removals)
452
+ except Exception:
453
+ try:
454
+ if original_exists and original_text is not None:
455
+ self._write_env_file_atomically(env_path, original_text)
456
+ else:
457
+ env_path.unlink(missing_ok=True)
458
+ except Exception:
459
+ logger.error("Failed to roll back provider configuration file after runtime update failure")
460
+ raise
461
+ changed_names = sorted((*normalized_updates, *normalized_removals))
462
+ logger.info("Persisted %s to %s", ", ".join(changed_names), env_path)
463
+
464
+ def _remove_persisted_env_values(self, env_names: tuple[str, ...]) -> None:
465
+ """Remove keys from the instance `.env` without mutating the current runtime."""
466
+ normalized_names = tuple(sorted({name.strip() for name in env_names if name and name.strip()}))
467
+ if not normalized_names or not self._instance_path:
468
+ return
469
+
470
+ env_path = Path(self._instance_path) / ".env"
471
+ if not env_path.exists():
472
+ return
473
+
474
+ try:
475
+ lines = env_path.read_text(encoding="utf-8").splitlines()
476
+ filtered_lines = [
477
+ line
478
+ for line in lines
479
+ if not any(line.strip().startswith(f"{env_name}=") for env_name in normalized_names)
480
+ ]
481
+ if filtered_lines == lines:
482
+ return
483
+
484
+ final_text = "\n".join(filtered_lines)
485
+ if final_text:
486
+ final_text += "\n"
487
+ env_path.write_text(final_text, encoding="utf-8")
488
+ env_path.chmod(0o600)
489
+ logger.info("Removed %s from %s", ", ".join(normalized_names), env_path)
490
+ except Exception as e:
491
+ logger.warning("Failed to remove %s: %s", ", ".join(normalized_names), e)
492
+
493
+ def _persist_hf_direct_connection(self, host: str, port: int) -> None:
494
+ """Persist a direct Hugging Face websocket target."""
495
+ self._persist_env_values(
496
+ {
497
+ HF_REALTIME_CONNECTION_MODE_ENV: HF_LOCAL_CONNECTION_MODE,
498
+ HF_REALTIME_WS_URL_ENV: build_hf_direct_ws_url(host, port),
499
+ }
500
+ )
501
+
502
+ def _persist_hf_allocator_connection(self) -> None:
503
+ """Persist the deployed Hugging Face allocator mode."""
504
+ self._persist_env_values(
505
+ {HF_REALTIME_CONNECTION_MODE_ENV: HF_DEPLOYED_CONNECTION_MODE},
506
+ removals=("HF_REALTIME_SESSION_URL",),
507
+ )
508
+
509
+ def _persist_personality(self, profile: Optional[str], voice_override: Optional[str] = None) -> None:
510
+ """Persist startup profile and voice in instance-local UI settings."""
511
+ if LOCKED_PROFILE is not None:
512
+ return
513
+ selection = (profile or "").strip() or None
514
+ normalized_voice_override = (voice_override or "").strip() or None
515
+ try:
516
+ from peachy.config import set_custom_profile
517
+
518
+ set_custom_profile(selection)
519
+ except Exception:
520
+ pass
521
+
522
+ if not self._instance_path:
523
+ return
524
+ try:
525
+ write_startup_settings(
526
+ self._instance_path,
527
+ profile=selection,
528
+ voice=normalized_voice_override,
529
+ )
530
+ self._remove_persisted_env_values(LEGACY_STARTUP_ENV_NAMES)
531
+ logger.info("Persisted startup personality settings to %s", Path(self._instance_path))
532
+ except Exception as e:
533
+ logger.warning("Failed to persist startup personality settings: %s", e)
534
+
535
+ def _read_persisted_personality(self) -> Optional[str]:
536
+ """Read the saved startup personality from instance-local UI settings."""
537
+ return read_startup_settings(self._instance_path).profile
538
+
539
+ async def apply_personality(self, profile: Optional[str]) -> str:
540
+ """Apply a personality by updating config and restarting the active backend."""
541
+ try:
542
+ from peachy.config import set_custom_profile
543
+ from peachy.prompts import get_session_voice, get_session_instructions
544
+
545
+ previous_profile = getattr(config, "REACHY_MINI_CUSTOM_PROFILE", None)
546
+ set_custom_profile(profile)
547
+ try:
548
+ get_session_instructions()
549
+ get_session_voice(default=get_default_voice())
550
+ except Exception:
551
+ set_custom_profile(previous_profile)
552
+ raise
553
+ except Exception as e:
554
+ safe_error = redact_secrets(e)
555
+ logger.error("Error applying personality '%s': %s", profile, safe_error)
556
+ return f"Failed to apply personality: {safe_error}"
557
+
558
+ # Rebuild the tool registry
559
+ initialize_tools(force=True)
560
+ await self.request_backend_restart("personality_changed")
561
+ return "Applied personality and restarting backend."
562
+
563
+ async def get_available_voices(self) -> list[str]:
564
+ """Return voices exposed by the active provider handler."""
565
+ try:
566
+ return list(await self.handler.get_available_voices())
567
+ except asyncio.CancelledError:
568
+ raise
569
+ except Exception as e:
570
+ logger.error("Failed to read voices from active provider: %s", redact_configured_credentials(e))
571
+ return []
572
+
573
+ def get_current_voice(self) -> str:
574
+ """Return the active provider handler's current voice."""
575
+ try:
576
+ handler_voice = self.handler.get_current_voice()
577
+ if isinstance(handler_voice, str):
578
+ return handler_voice
579
+ except Exception as e:
580
+ logger.error("Failed to read current voice from active provider: %s", redact_configured_credentials(e))
581
+
582
+ # Preserve the pre-provider fallback for incomplete test/external handlers.
583
+ if self._voice_override:
584
+ return self._voice_override
585
+ try:
586
+ from peachy.prompts import get_session_voice
587
+
588
+ return get_session_voice(default=get_default_voice())
589
+ except Exception:
590
+ return get_default_voice()
591
+
592
+ async def change_voice(self, voice: str) -> str:
593
+ """Change the voice through the active handler without rebuilding the backend."""
594
+ try:
595
+ status = await self.handler.change_voice(voice)
596
+ except asyncio.CancelledError:
597
+ raise
598
+ except Exception as e:
599
+ safe_error = redact_configured_credentials(e)
600
+ logger.error("Error changing voice to %r: %s", voice, safe_error)
601
+ return f"Failed to change voice: {safe_error}"
602
+
603
+ try:
604
+ current_voice = self.handler.get_current_voice()
605
+ if isinstance(current_voice, str) and current_voice.strip():
606
+ self._voice_override = current_voice
607
+ except Exception as e:
608
+ logger.debug(
609
+ "Could not sync LocalStream voice override after voice change: %s",
610
+ redact_configured_credentials(e),
611
+ )
612
+ if self._voice_override:
613
+ self._persist_voice_override(self._voice_override)
614
+ return status
615
+
616
+ def _persist_voice_override(self, voice: str) -> None:
617
+ """Persist the chosen voice as the startup voice, keeping the startup profile."""
618
+ if not self._instance_path:
619
+ return
620
+ try:
621
+ existing = read_startup_settings(self._instance_path)
622
+ write_startup_settings(self._instance_path, profile=existing.profile, voice=voice)
623
+ except Exception as e:
624
+ logger.warning("Failed to persist startup voice: %s", e)
625
+
626
+ def _init_settings_ui_if_needed(self) -> None:
627
+ """Attach minimal settings UI to the settings app.
628
+
629
+ Always mounts the UI when a settings_app is provided so that users
630
+ see a confirmation message even if the API key is already configured.
631
+ """
632
+ if self._settings_initialized:
633
+ return
634
+ if self._settings_app is None:
635
+ return
636
+ settings_app = self._settings_app
637
+
638
+ static_dir = Path(__file__).parent / "static"
639
+ index_file = static_dir / "index.html"
640
+ logger.info("Serving settings UI from %s", static_dir)
641
+
642
+ # Framework pre-registers GET / and /static; strip them so our routes aren't shadowed.
643
+ _detach_framework_root_routes(settings_app)
644
+
645
+ if hasattr(settings_app, "mount"):
646
+ try:
647
+ settings_app.mount("/static", StaticFiles(directory=str(static_dir)), name="static")
648
+ except Exception:
649
+ pass
650
+
651
+ class BackendPayload(BaseModel):
652
+ provider: Optional[str] = None
653
+ openai_api_key: Optional[str] = None
654
+ openai_model: Optional[str] = None
655
+ hf_mode: Optional[str] = None
656
+ hf_host: Optional[str] = None
657
+ hf_port: Optional[int] = None
658
+
659
+ def _status_payload() -> dict[str, object]:
660
+ definition = self._provider_resolver()
661
+ readiness = definition.readiness()
662
+ backend_connection = self._backend_connection_status()
663
+ payload = {
664
+ "provider": definition.name,
665
+ "available_providers": list(available_provider_names()),
666
+ "provider_state": readiness.state,
667
+ "provider_message": readiness.message,
668
+ "can_proceed": readiness.state == "ready",
669
+ "requires_restart": not self._can_rebuild_handler(),
670
+ **definition.redacted_status(),
671
+ **backend_connection,
672
+ }
673
+ return {
674
+ name: redact_secrets(value) if isinstance(value, str) else value
675
+ for name, value in payload.items()
676
+ }
677
+
678
+ # GET / -> index.html
679
+ @settings_app.get("/")
680
+ def _root() -> FileResponse:
681
+ return FileResponse(str(index_file))
682
+
683
+ # GET /favicon.ico -> optional, avoid noisy 404s on some browsers
684
+ @settings_app.get("/favicon.ico")
685
+ def _favicon() -> Response:
686
+ return Response(status_code=204)
687
+
688
+ @settings_app.get(f"{SETTINGS_API_PREFIX}/status")
689
+ def _status() -> JSONResponse:
690
+ return JSONResponse(_status_payload())
691
+
692
+ @settings_app.post(f"{SETTINGS_API_PREFIX}/reconnect")
693
+ def _reconnect() -> JSONResponse:
694
+ self._mark_restart_requested("manual_reconnect")
695
+ return JSONResponse({"ok": True, "message": "Reconnecting selected provider."})
696
+
697
+ event_bus = self._event_bus
698
+
699
+ @settings_app.get(f"{SETTINGS_API_PREFIX}/conversation_events")
700
+ async def _conversation_events(request: Request) -> StreamingResponse:
701
+ return StreamingResponse(
702
+ _conversation_events_stream(event_bus, request),
703
+ media_type="text/event-stream",
704
+ headers={
705
+ "Cache-Control": "no-cache",
706
+ "X-Accel-Buffering": "no", # disable proxy buffering (e.g. nginx)
707
+ "Connection": "keep-alive",
708
+ },
709
+ )
710
+
711
+ class MicPayload(BaseModel):
712
+ muted: bool
713
+
714
+ @settings_app.get(f"{SETTINGS_API_PREFIX}/mic")
715
+ def _mic_state() -> JSONResponse:
716
+ return JSONResponse({"muted": self._mic_muted})
717
+
718
+ @settings_app.post(f"{SETTINGS_API_PREFIX}/mic")
719
+ def _set_mic(payload: MicPayload) -> JSONResponse:
720
+ self._mic_muted = bool(payload.muted)
721
+ logger.info("Microphone %s via web UI", "muted" if self._mic_muted else "unmuted")
722
+ return JSONResponse({"muted": self._mic_muted})
723
+
724
+ @settings_app.post(f"{SETTINGS_API_PREFIX}/backend_config")
725
+ def _set_backend(payload: BackendPayload) -> JSONResponse:
726
+ selected = resolve_provider_name(payload.provider)
727
+ if selected not in available_provider_names():
728
+ return JSONResponse({"ok": False, "error": "invalid_provider"}, status_code=400)
729
+
730
+ updates = {PROVIDER_ENV: selected}
731
+ removals: tuple[str, ...] = ()
732
+ if selected == OPENAI_PROVIDER:
733
+ supplied_key = (payload.openai_api_key or "").strip()
734
+ existing_key = (os.getenv("OPENAI_API_KEY") or "").strip()
735
+ if not supplied_key and not existing_key:
736
+ return JSONResponse({"ok": False, "error": "empty_key"}, status_code=400)
737
+ if supplied_key:
738
+ updates["OPENAI_API_KEY"] = supplied_key
739
+ model = (payload.openai_model or os.getenv(OPENAI_MODEL_ENV) or OPENAI_DEFAULT_MODEL).strip()
740
+ updates[OPENAI_MODEL_ENV] = model or OPENAI_DEFAULT_MODEL
741
+ else:
742
+ hf_selection = get_hf_connection_selection()
743
+ hf_mode = (payload.hf_mode or hf_selection.mode).strip().lower()
744
+ if hf_mode == HF_LOCAL_CONNECTION_MODE:
745
+ existing_host, existing_port = parse_hf_direct_target(hf_selection.direct_ws_url)
746
+ host = (payload.hf_host or "").strip() or existing_host or ""
747
+ if not host:
748
+ return JSONResponse({"ok": False, "error": "empty_hf_host"}, status_code=400)
749
+ if "://" in host or "/" in host or "?" in host or "#" in host:
750
+ return JSONResponse({"ok": False, "error": "invalid_hf_host"}, status_code=400)
751
+
752
+ port = payload.hf_port if payload.hf_port is not None else existing_port or 8765
753
+ if port < 1 or port > 65535:
754
+ return JSONResponse({"ok": False, "error": "invalid_hf_port"}, status_code=400)
755
+
756
+ updates[HF_REALTIME_CONNECTION_MODE_ENV] = HF_LOCAL_CONNECTION_MODE
757
+ updates[HF_REALTIME_WS_URL_ENV] = build_hf_direct_ws_url(host, port)
758
+ elif hf_mode == HF_DEPLOYED_CONNECTION_MODE:
759
+ if not bool(get_hf_session_url()):
760
+ return JSONResponse({"ok": False, "error": "missing_hf_session_url"}, status_code=400)
761
+ updates[HF_REALTIME_CONNECTION_MODE_ENV] = HF_DEPLOYED_CONNECTION_MODE
762
+ removals = ("HF_REALTIME_SESSION_URL",)
763
+ else:
764
+ return JSONResponse({"ok": False, "error": "invalid_hf_mode"}, status_code=400)
765
+
766
+ try:
767
+ self._persist_env_values(updates, removals=removals)
768
+ except Exception as e:
769
+ logger.warning("Failed to persist provider configuration (%s)", type(e).__name__)
770
+ return JSONResponse({"ok": False, "error": "persistence_failed"}, status_code=500)
771
+ if self._can_rebuild_handler():
772
+ self._mark_restart_requested("provider_config_changed")
773
+ message = "Connection saved. Reconnecting backend."
774
+ else:
775
+ message = "Connection saved. Restart Reachy Mini Conversation from the desktop app to apply it."
776
+ return JSONResponse(
777
+ {
778
+ "ok": True,
779
+ "message": message,
780
+ **_status_payload(),
781
+ }
782
+ )
783
+
784
+ try:
785
+ mount_personality_routes(
786
+ self._settings_app,
787
+ self.handler,
788
+ lambda: self._asyncio_loop,
789
+ persist_personality=self._persist_personality,
790
+ get_persisted_personality=self._read_persisted_personality,
791
+ apply_personality=self.apply_personality,
792
+ get_voices=self.get_available_voices,
793
+ get_current_voice=self.get_current_voice,
794
+ change_voice=self.change_voice,
795
+ api_prefix=SETTINGS_API_PREFIX,
796
+ )
797
+ except Exception:
798
+ logger.exception("Failed to mount personality routes; the personality UI will be unavailable")
799
+
800
+ self._settings_initialized = True
801
+
802
+ async def _run_handler_startup_loop(self) -> None:
803
+ """Start the realtime handler and keep settings UI alive after backend failures."""
804
+ while not self._stop_event.is_set():
805
+ definition = self._provider_resolver()
806
+ readiness = definition.readiness()
807
+ if readiness.state != "ready":
808
+ self._set_backend_connection_state(readiness.state, readiness.message)
809
+ await self._wait_for_restart_or_stop()
810
+ if self._stop_event.is_set():
811
+ return
812
+
813
+ if self._restart_requested.is_set():
814
+ await self._shutdown_active_handler()
815
+ self._restart_requested.clear()
816
+ if not self._can_rebuild_handler():
817
+ self._set_backend_connection_state("restart_required")
818
+ await self._wait_for_restart_or_stop()
819
+ continue
820
+ try:
821
+ self._build_handler_for_current_backend()
822
+ except Exception as exc:
823
+ self._set_backend_connection_state("configuration_error", exc)
824
+ await self._wait_for_restart_or_stop()
825
+ continue
826
+ definition = self._provider_resolver()
827
+ readiness = definition.readiness()
828
+ if readiness.state != "ready":
829
+ continue
830
+
831
+ self._set_backend_connection_state("connecting")
832
+ try:
833
+ await self.handler.start_up()
834
+ except asyncio.CancelledError:
835
+ raise
836
+ except Exception as exc:
837
+ self._set_backend_connection_state("disconnected", exc)
838
+ logger.warning("Provider activation stopped: %s", redact_secrets(exc))
839
+ else:
840
+ if self._stop_event.is_set():
841
+ return
842
+ self._set_backend_connection_state("disconnected")
843
+
844
+ if definition.auto_reconnect:
845
+ await self._sleep_or_restart_requested(self._backend_retry_delay)
846
+ else:
847
+ await self._wait_for_restart_or_stop()
848
+
849
+ def launch(self) -> None:
850
+ """Start the recorder/player and run the async processing loops.
851
+
852
+ If the selected backend is missing its required key, expose a tiny
853
+ settings UI via the Reachy Mini settings server to collect it before
854
+ starting streams.
855
+ """
856
+ self._stop_event.clear()
857
+
858
+ # Try to load an existing instance .env first (covers subsequent runs)
859
+ if self._instance_path:
860
+ try:
861
+ from dotenv import load_dotenv
862
+
863
+ env_path = Path(self._instance_path) / ".env"
864
+ if env_path.exists():
865
+ load_dotenv(dotenv_path=str(env_path), override=True)
866
+ refresh_runtime_config_from_env()
867
+ except Exception:
868
+ pass # Instance .env loading is optional; continue with defaults
869
+
870
+ # Always expose settings UI if a settings app is available
871
+ # (do this AFTER loading the instance .env so status endpoint sees the right value)
872
+ self._init_settings_ui_if_needed()
873
+
874
+ # Media starts even while the provider waits for configuration so the UI stays responsive.
875
+ self._robot.media.start_recording()
876
+ self._robot.media.start_playing()
877
+
878
+ async def runner() -> None:
879
+ # Capture loop for cross-thread personality actions
880
+ loop = asyncio.get_running_loop()
881
+ self._asyncio_loop = loop # type: ignore[assignment]
882
+ # Connect the backend first so it overlaps the warmup and audio config below.
883
+ handler_task = asyncio.create_task(self._run_handler_startup_loop(), name="realtime-handler")
884
+ self._tasks = [handler_task]
885
+ await asyncio.sleep(1) # give the pipelines time to start
886
+ await asyncio.to_thread(apply_audio_startup_config, self._robot, logger=logger)
887
+ self._tasks += [
888
+ asyncio.create_task(self.record_loop(), name="stream-record-loop"),
889
+ asyncio.create_task(self.play_loop(), name="stream-play-loop"),
890
+ ]
891
+ try:
892
+ await asyncio.gather(*self._tasks)
893
+ except asyncio.CancelledError:
894
+ logger.info("Tasks cancelled during shutdown")
895
+ finally:
896
+ # Ensure handler connection is closed
897
+ await self._shutdown_active_handler()
898
+
899
+ asyncio.run(runner())
900
+
901
+ def close(self) -> None:
902
+ """Stop the stream and underlying media pipelines.
903
+
904
+ This method:
905
+ - Stops audio recording and playback first
906
+ - Sets the stop event to signal async loops to terminate
907
+ - Cancels all pending async tasks (openai-handler, record-loop, play-loop)
908
+ """
909
+ logger.info("Stopping LocalStream...")
910
+
911
+ # Stop media pipelines FIRST before cancelling async tasks
912
+ # This ensures clean shutdown before PortAudio cleanup
913
+ try:
914
+ self._robot.media.stop_recording()
915
+ except Exception as e:
916
+ logger.debug(f"Error stopping recording (may already be stopped): {e}")
917
+
918
+ try:
919
+ self._robot.media.stop_playing()
920
+ except Exception as e:
921
+ logger.debug(f"Error stopping playback (may already be stopped): {e}")
922
+
923
+ # close() runs on watcher threads, loop-owned state must change on the loop.
924
+ loop = self._asyncio_loop
925
+ if loop is None or not loop.is_running():
926
+ self._stop_event.set()
927
+ return
928
+ loop.call_soon_threadsafe(self._stop_event.set)
929
+ for task in self._tasks:
930
+ if not task.done():
931
+ loop.call_soon_threadsafe(task.cancel)
932
+
933
+ def clear_audio_queue(self) -> None:
934
+ """Flush queued playback audio immediately on user barge-in.
935
+
936
+ Calls the SDK's ``clear_player()`` — now a first-class flush on both
937
+ the local GStreamer and WebRTC backends (the WebRTC one also tells the
938
+ daemon to drop audio already queued for the speaker). Falls back to the
939
+ deprecated ``clear_output_buffer()`` only for older SDKs.
940
+ """
941
+ logger.info("User intervention: flushing player queue")
942
+ audio = getattr(self._robot.media, "audio", None)
943
+ if audio is not None:
944
+ if hasattr(audio, "clear_player") and callable(audio.clear_player):
945
+ audio.clear_player()
946
+ elif hasattr(audio, "clear_output_buffer") and callable(audio.clear_output_buffer):
947
+ # Older SDK without clear_player(); best-effort.
948
+ audio.clear_output_buffer()
949
+ # Drain the handler's pending output in place — do NOT replace the
950
+ # queue object, since emit() may be awaiting it (wait_for_item).
951
+ self._drain_output_queue()
952
+
953
+ def _drain_output_queue(self) -> None:
954
+ """Empty the handler's output queue in place without replacing it."""
955
+ queue = getattr(self.handler, "output_queue", None)
956
+ if queue is None:
957
+ return
958
+ while not queue.empty():
959
+ try:
960
+ queue.get_nowait()
961
+ except asyncio.QueueEmpty:
962
+ break
963
+
964
+ async def record_loop(self) -> None:
965
+ """Read mic frames from the recorder and forward them to the handler."""
966
+ input_sample_rate = self._robot.media.get_input_audio_samplerate()
967
+ logger.debug(f"Audio recording started at {input_sample_rate} Hz")
968
+
969
+ while not self._stop_event.is_set():
970
+ audio_frame = self._robot.media.get_audio_sample()
971
+ if audio_frame is not None and not self._mic_muted:
972
+ await self.handler.receive((input_sample_rate, audio_frame))
973
+ await asyncio.sleep(0) # avoid busy loop
974
+
975
+ async def play_loop(self) -> None:
976
+ """Fetch outputs from the handler: log text and play audio frames."""
977
+ while not self._stop_event.is_set():
978
+ handler = self.handler
979
+ try:
980
+ handler_output = await asyncio.wait_for(handler.emit(), timeout=0.5)
981
+ except asyncio.TimeoutError:
982
+ continue
983
+
984
+ if isinstance(handler_output, AdditionalOutputs):
985
+ for msg in handler_output.args:
986
+ content = msg.get("content", "")
987
+ if isinstance(content, str):
988
+ logger.info(
989
+ "role=%s content=%s",
990
+ msg.get("role"),
991
+ content if len(content) < 500 else content[:500] + "…",
992
+ )
993
+
994
+ elif isinstance(handler_output, tuple):
995
+ source_rate, audio_data = handler_output
996
+ if audio_data.size == 0:
997
+ continue
998
+ output_rate = self._robot.media.get_output_audio_samplerate()
999
+ audio_frame = resample_audio(audio_data, source_rate, output_rate)
1000
+ self._robot.media.push_audio_sample(audio_frame)
1001
+
1002
+ else:
1003
+ logger.debug("Ignoring output type=%s", type(handler_output).__name__)
1004
+
1005
+ await asyncio.sleep(0) # yield to event loop
src/peachy/conversation_handler.py ADDED
@@ -0,0 +1,133 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Modified by Peachy contributors in 2026 for namespace, packaging, and provider integration.
2
+ from __future__ import annotations
3
+ import time
4
+ import asyncio
5
+ import logging
6
+ from abc import ABC, abstractmethod
7
+ from typing import ClassVar, TypeAlias
8
+ from collections.abc import Callable
9
+
10
+ from peachy.streaming import AdditionalOutputs, AsyncStreamHandler, AudioArray, wait_for_item
11
+ from peachy.idle_policy import start_idle_tool_call
12
+ from peachy.tools.core_tools import ToolDependencies, get_tool_specs
13
+ from peachy.tools.background_tool_manager import BackgroundToolManager
14
+
15
+
16
+ logger = logging.getLogger(__name__)
17
+
18
+
19
+ AudioFrame: TypeAlias = tuple[int, AudioArray]
20
+ HandlerOutput: TypeAlias = AudioFrame | AdditionalOutputs | None
21
+ QueueItem: TypeAlias = AudioFrame | AdditionalOutputs
22
+
23
+
24
+ class ConversationHandler(AsyncStreamHandler, ABC):
25
+ """Shared app handler contract and idle behavior for realtime conversation backends."""
26
+
27
+ IDLE_BEHAVIOR_THRESHOLD_S: ClassVar[float] = 180.0
28
+
29
+ deps: ToolDependencies
30
+ tool_manager: BackgroundToolManager
31
+ output_queue: asyncio.Queue[QueueItem]
32
+ last_activity_time: float
33
+ last_idle_behavior_time: float
34
+ _activity_observer: Callable[[str], None] | None = None
35
+
36
+ def __init__(self) -> None:
37
+ """Initialize the stream handler and shared idle/activity tracking."""
38
+ super().__init__()
39
+ self.last_activity_time = time.monotonic()
40
+ self.last_idle_behavior_time = self.last_activity_time
41
+
42
+ def set_activity_observer(self, observer: Callable[[str], None] | None) -> None:
43
+ """Attach or detach an activity observer. Pass None to clear."""
44
+ self._activity_observer = observer
45
+
46
+ def _mark_activity(self, reason: str) -> None:
47
+ """Record non-idle conversation activity for the idle timer."""
48
+ self.last_activity_time = time.monotonic()
49
+ logger.debug("last activity time updated to %s (%s)", self.last_activity_time, reason)
50
+ if self._activity_observer is not None:
51
+ try:
52
+ self._activity_observer(reason)
53
+ except Exception:
54
+ logger.debug("activity observer raised (ignored)", exc_info=True)
55
+
56
+ def _idle_behavior_ready(self) -> bool:
57
+ """Return whether idle behavior may run now. Backends can add guards."""
58
+ return True
59
+
60
+ async def emit(self) -> HandlerOutput:
61
+ """Emit the next queued output, triggering local idle behavior when due."""
62
+ now = time.monotonic()
63
+ idle_duration = now - self.last_activity_time
64
+ idle_behavior_duration = now - self.last_idle_behavior_time
65
+ if (
66
+ idle_duration > self.IDLE_BEHAVIOR_THRESHOLD_S
67
+ and idle_behavior_duration > self.IDLE_BEHAVIOR_THRESHOLD_S
68
+ and self._idle_behavior_ready()
69
+ and self.deps.movement_manager.is_idle()
70
+ ):
71
+ try:
72
+ await self.send_idle_signal(idle_duration)
73
+ except Exception as e:
74
+ logger.warning("Idle tool skipped (connection closed?): %s", e)
75
+ return None
76
+ self.last_idle_behavior_time = now
77
+ handler_output = await wait_for_item(self.output_queue)
78
+ return handler_output
79
+
80
+ async def send_idle_signal(self, idle_duration: float) -> None:
81
+ """Run a locally selected idle tool without sending an idle turn to the model."""
82
+ if not self._is_connected():
83
+ logger.debug("No active session; cannot run idle tool")
84
+ return
85
+
86
+ available_tool_names = {spec["name"] for spec in get_tool_specs()}
87
+ await start_idle_tool_call(
88
+ deps=self.deps,
89
+ tool_manager=self.tool_manager,
90
+ output_queue=self.output_queue,
91
+ available_tool_names=available_tool_names,
92
+ idle_duration=idle_duration,
93
+ )
94
+
95
+ @abstractmethod
96
+ def _is_connected(self) -> bool:
97
+ """Return whether the backend session/connection is currently open."""
98
+ ...
99
+
100
+ @abstractmethod
101
+ async def start_up(self) -> None:
102
+ """Start the realtime handler."""
103
+ ...
104
+
105
+ @abstractmethod
106
+ async def shutdown(self) -> None:
107
+ """Shut down the realtime handler."""
108
+ ...
109
+
110
+ @abstractmethod
111
+ async def receive(self, frame: AudioFrame) -> None:
112
+ """Receive an input audio frame."""
113
+ ...
114
+
115
+ @abstractmethod
116
+ async def apply_personality(self, profile: str | None) -> str:
117
+ """Apply a personality profile."""
118
+ ...
119
+
120
+ @abstractmethod
121
+ async def get_available_voices(self) -> list[str]:
122
+ """Return voices available for the active backend."""
123
+ ...
124
+
125
+ @abstractmethod
126
+ def get_current_voice(self) -> str:
127
+ """Return the current voice."""
128
+ ...
129
+
130
+ @abstractmethod
131
+ async def change_voice(self, voice: str) -> str:
132
+ """Change the current voice."""
133
+ ...
src/peachy/dance_emotion_moves.py ADDED
@@ -0,0 +1,156 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Dance and emotion moves for the movement queue system.
2
+
3
+ This module implements dance moves and emotions as Move objects that can be queued
4
+ and executed sequentially by the MovementManager.
5
+ """
6
+ # Modified by Peachy contributors in 2026 for namespace, packaging, and provider integration.
7
+
8
+
9
+ from __future__ import annotations
10
+ import logging
11
+ from typing import Tuple
12
+
13
+ import numpy as np
14
+ from numpy.typing import NDArray
15
+
16
+ from reachy_mini.motion.move import Move
17
+ from reachy_mini.motion.recorded_move import RecordedMoves
18
+ from reachy_mini_dances_library.dance_move import DanceMove
19
+
20
+
21
+ logger = logging.getLogger(__name__)
22
+
23
+
24
+ class DanceQueueMove(Move): # type: ignore
25
+ """Wrapper for dance moves to work with the movement queue system."""
26
+
27
+ def __init__(self, move_name: str):
28
+ """Initialize a DanceQueueMove."""
29
+ self.dance_move = DanceMove(move_name)
30
+ self.move_name = move_name
31
+
32
+ @property
33
+ def duration(self) -> float:
34
+ """Duration property required by official Move interface."""
35
+ return float(self.dance_move.duration)
36
+
37
+ def evaluate(self, t: float) -> tuple[NDArray[np.float64] | None, NDArray[np.float64] | None, float | None]:
38
+ """Evaluate dance move at time t."""
39
+ try:
40
+ # Get the pose from the dance move
41
+ head_pose, antennas, body_yaw = self.dance_move.evaluate(t)
42
+
43
+ # Convert to numpy array if antennas is tuple and return in official Move format
44
+ if isinstance(antennas, tuple):
45
+ antennas = np.array([antennas[0], antennas[1]])
46
+
47
+ return (head_pose, antennas, body_yaw)
48
+
49
+ except Exception as e:
50
+ logger.error(f"Error evaluating dance move '{self.move_name}' at t={t}: {e}")
51
+ # Return neutral pose on error
52
+ from reachy_mini.utils import create_head_pose
53
+
54
+ neutral_head_pose = create_head_pose(0, 0, 0, 0, 0, 0, degrees=True)
55
+ return (neutral_head_pose, np.array([0.0, 0.0], dtype=np.float64), 0.0)
56
+
57
+
58
+ class EmotionQueueMove(Move): # type: ignore
59
+ """Wrapper for emotion moves to work with the movement queue system."""
60
+
61
+ def __init__(self, emotion_name: str, recorded_moves: RecordedMoves):
62
+ """Initialize an EmotionQueueMove."""
63
+ self.emotion_move = recorded_moves.get(emotion_name)
64
+ self.emotion_name = emotion_name
65
+
66
+ @property
67
+ def duration(self) -> float:
68
+ """Duration property required by official Move interface."""
69
+ return float(self.emotion_move.duration)
70
+
71
+ def evaluate(self, t: float) -> tuple[NDArray[np.float64] | None, NDArray[np.float64] | None, float | None]:
72
+ """Evaluate emotion move at time t."""
73
+ try:
74
+ # Get the pose from the emotion move
75
+ head_pose, antennas, body_yaw = self.emotion_move.evaluate(t)
76
+
77
+ # Convert to numpy array if antennas is tuple and return in official Move format
78
+ if isinstance(antennas, tuple):
79
+ antennas = np.array([antennas[0], antennas[1]])
80
+
81
+ return (head_pose, antennas, body_yaw)
82
+
83
+ except Exception as e:
84
+ logger.error(f"Error evaluating emotion '{self.emotion_name}' at t={t}: {e}")
85
+ # Return neutral pose on error
86
+ from reachy_mini.utils import create_head_pose
87
+
88
+ neutral_head_pose = create_head_pose(0, 0, 0, 0, 0, 0, degrees=True)
89
+ return (neutral_head_pose, np.array([0.0, 0.0], dtype=np.float64), 0.0)
90
+
91
+
92
+ class GotoQueueMove(Move): # type: ignore
93
+ """Wrapper for goto moves to work with the movement queue system."""
94
+
95
+ def __init__(
96
+ self,
97
+ target_head_pose: NDArray[np.float32],
98
+ start_head_pose: NDArray[np.float32] | None = None,
99
+ target_antennas: Tuple[float, float] = (0, 0),
100
+ start_antennas: Tuple[float, float] | None = None,
101
+ target_body_yaw: float = 0,
102
+ start_body_yaw: float | None = None,
103
+ duration: float = 1.0,
104
+ ):
105
+ """Initialize a GotoQueueMove."""
106
+ self._duration = duration
107
+ self.target_head_pose = target_head_pose
108
+ self.start_head_pose = start_head_pose
109
+ self.target_antennas = target_antennas
110
+ self.start_antennas = start_antennas or (0, 0)
111
+ self.target_body_yaw = target_body_yaw
112
+ self.start_body_yaw = start_body_yaw or 0
113
+
114
+ @property
115
+ def duration(self) -> float:
116
+ """Duration property required by official Move interface."""
117
+ return self._duration
118
+
119
+ def evaluate(self, t: float) -> tuple[NDArray[np.float64] | None, NDArray[np.float64] | None, float | None]:
120
+ """Evaluate goto move at time t using linear interpolation."""
121
+ try:
122
+ from reachy_mini.utils import create_head_pose
123
+ from reachy_mini.utils.interpolation import linear_pose_interpolation
124
+
125
+ # Clamp t to [0, 1] for interpolation
126
+ t_clamped = max(0, min(1, t / self.duration))
127
+
128
+ # Use start pose if available, otherwise neutral
129
+ if self.start_head_pose is not None:
130
+ start_pose = self.start_head_pose
131
+ else:
132
+ start_pose = create_head_pose(0, 0, 0, 0, 0, 0, degrees=True)
133
+
134
+ # Interpolate head pose
135
+ head_pose = linear_pose_interpolation(start_pose, self.target_head_pose, t_clamped)
136
+
137
+ # Interpolate antennas - return as numpy array
138
+ antennas = np.array(
139
+ [
140
+ self.start_antennas[0] + (self.target_antennas[0] - self.start_antennas[0]) * t_clamped,
141
+ self.start_antennas[1] + (self.target_antennas[1] - self.start_antennas[1]) * t_clamped,
142
+ ],
143
+ dtype=np.float64,
144
+ )
145
+
146
+ # Interpolate body yaw
147
+ body_yaw = self.start_body_yaw + (self.target_body_yaw - self.start_body_yaw) * t_clamped
148
+
149
+ return (head_pose, antennas, body_yaw)
150
+
151
+ except Exception as e:
152
+ logger.error(f"Error evaluating goto move at t={t}: {e}")
153
+ # Return target pose on error - convert to float64
154
+ target_head_pose_f64 = self.target_head_pose.astype(np.float64)
155
+ target_antennas_array = np.array([self.target_antennas[0], self.target_antennas[1]], dtype=np.float64)
156
+ return (target_head_pose_f64, target_antennas_array, self.target_body_yaw)
src/peachy/huggingface_realtime.py ADDED
@@ -0,0 +1,1133 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Modified by Peachy contributors in 2026 for namespace, packaging, and provider integration.
2
+ import json
3
+ import time
4
+ import uuid
5
+ import base64
6
+ import random
7
+ import asyncio
8
+ import logging
9
+ from typing import Any, Final, Optional
10
+ from urllib.parse import urlsplit, urlunsplit
11
+
12
+ import httpx
13
+ import numpy as np
14
+ from openai import AsyncOpenAI
15
+ from pydantic import Field, BaseModel
16
+ from typing_extensions import Literal, TypedDict
17
+ from openai.types.realtime import (
18
+ AudioTranscriptionParam,
19
+ RealtimeAudioConfigParam,
20
+ RealtimeToolsConfigParam,
21
+ RealtimeFunctionToolParam,
22
+ RealtimeAudioConfigInputParam,
23
+ RealtimeAudioConfigOutputParam,
24
+ RealtimeSessionCreateRequestParam,
25
+ )
26
+ from websockets.exceptions import ConnectionClosedError
27
+ from openai.resources.realtime.realtime import AsyncRealtimeConnection
28
+ from openai.types.realtime.realtime_audio_input_turn_detection_param import ServerVad
29
+
30
+ from peachy.tools import core_tools
31
+ from peachy.config import (
32
+ HF_LOCAL_CONNECTION_MODE,
33
+ config,
34
+ get_default_voice,
35
+ get_available_voices,
36
+ get_hf_direct_ws_url,
37
+ parse_hf_realtime_url,
38
+ get_hf_connection_selection,
39
+ )
40
+ from peachy.prompts import (
41
+ get_session_voice,
42
+ get_session_instructions,
43
+ get_session_greeting_prompt,
44
+ )
45
+ from peachy.streaming import AdditionalOutputs, audio_to_int16
46
+ from peachy.tools.core_tools import (
47
+ ToolSpec,
48
+ ToolDependencies,
49
+ get_tool_specs,
50
+ )
51
+ from peachy.conversation_handler import AudioFrame, QueueItem, ConversationHandler
52
+ from peachy.providers import redact_secrets
53
+ from peachy.tools.background_tool_manager import (
54
+ ToolCallRoutine,
55
+ ToolNotification,
56
+ BackgroundToolManager,
57
+ )
58
+
59
+
60
+ logger = logging.getLogger(__name__)
61
+
62
+ _RESPONSE_DONE_TIMEOUT: Final[float] = 30.0
63
+ _RESPONSE_REJECTION_RETRY_DELAY: Final[float] = 0.5
64
+
65
+
66
+ def _safe_endpoint_for_log(endpoint: str) -> str:
67
+ """Render an endpoint without userinfo, query parameters, or fragments."""
68
+ parsed = urlsplit(endpoint)
69
+ host = parsed.hostname or ""
70
+ if ":" in host and not host.startswith("["):
71
+ host = f"[{host}]"
72
+ netloc = f"{host}:{parsed.port}" if parsed.port is not None else host
73
+ return urlunsplit((parsed.scheme, netloc, parsed.path, "", ""))
74
+
75
+
76
+ class InputTranscriptChunksByItem(BaseModel):
77
+ """Current item_id and its accumulated deltas. Only one item at a time."""
78
+
79
+ item_id: str | None = None
80
+ deltas: list[str] = Field(default_factory=list)
81
+
82
+
83
+ def to_realtime_tools_config(tool_specs: list[ToolSpec]) -> RealtimeToolsConfigParam:
84
+ """Convert app tool specs to the OpenAI-compatible realtime session shape."""
85
+ realtime_tools: RealtimeToolsConfigParam = []
86
+ for spec in tool_specs:
87
+ realtime_tools.append(
88
+ RealtimeFunctionToolParam(
89
+ type="function",
90
+ name=spec["name"],
91
+ description=spec["description"],
92
+ parameters=spec["parameters"],
93
+ )
94
+ )
95
+ return realtime_tools
96
+
97
+
98
+ class HFNativeRateAudioPCM(TypedDict):
99
+ """Hugging Face extension for native-rate PCM audio."""
100
+
101
+ type: Literal["audio/pcm"]
102
+ rate: None
103
+
104
+
105
+ def _native_rate_audio_pcm() -> HFNativeRateAudioPCM:
106
+ """Return the Hugging Face native-rate PCM config."""
107
+ return {"type": "audio/pcm", "rate": None}
108
+
109
+
110
+ def _build_openai_compatible_client_from_realtime_url(
111
+ realtime_url: str,
112
+ bearer_token: str | None,
113
+ ) -> tuple[AsyncOpenAI, dict[str, str]]:
114
+ """Build an OpenAI-compatible realtime client from a direct websocket/base URL."""
115
+ parsed = parse_hf_realtime_url(realtime_url)
116
+ client = AsyncOpenAI(
117
+ api_key=bearer_token or "DUMMY",
118
+ base_url=parsed.base_url,
119
+ websocket_base_url=parsed.websocket_base_url,
120
+ )
121
+ return client, parsed.connect_query
122
+
123
+
124
+ class HuggingFaceRealtimeHandler(ConversationHandler):
125
+ """Realtime stream handler for the Hugging Face OpenAI-compatible endpoint."""
126
+
127
+ SAMPLE_RATE = 16000
128
+
129
+ def __init__(
130
+ self,
131
+ deps: ToolDependencies,
132
+ instance_path: Optional[str] = None,
133
+ startup_voice: Optional[str] = None,
134
+ ):
135
+ """Initialize the handler."""
136
+ super().__init__()
137
+
138
+ self.deps = deps
139
+
140
+ self.client: AsyncOpenAI
141
+ self.connection: AsyncRealtimeConnection | None = None
142
+ self.output_queue: asyncio.Queue[QueueItem] = asyncio.Queue()
143
+
144
+ self.instance_path = instance_path
145
+ self._voice_override: str | None = self._normalize_startup_voice(startup_voice)
146
+ self._realtime_connect_query: dict[str, str] = {}
147
+
148
+ # Debouncing for partial transcripts
149
+ self.partial_transcript_task: asyncio.Task[None] | None = None
150
+ self.partial_debounce_delay = 0.5 # seconds
151
+ self.input_transcript_chunks_by_item = InputTranscriptChunksByItem()
152
+
153
+ # Internal lifecycle flags
154
+ self._connected_event: asyncio.Event = asyncio.Event()
155
+
156
+ # Background tool manager
157
+ self.tool_manager = BackgroundToolManager()
158
+
159
+ # Response-in-progress guard: the Realtime API only allows one active
160
+ # response per conversation at a time. A dedicated worker task
161
+ # (_response_sender_loop) dequeues and sends one request at a time
162
+ self._pending_responses: asyncio.Queue[dict[str, Any]] = asyncio.Queue()
163
+ self._response_done_event: asyncio.Event = asyncio.Event()
164
+ self._response_done_event.set()
165
+ self._response_started_or_rejected_event: asyncio.Event = asyncio.Event()
166
+ self._last_response_rejected: bool = False
167
+ self._turn_user_done_at: float | None = None
168
+ self._turn_response_created_at: float | None = None
169
+ self._turn_first_audio_at: float | None = None
170
+ self._startup_greeting_sent = False
171
+ self._in_flight_tool_calls: set[str] = set()
172
+ self._tool_batch_needs_response = False
173
+ self._restart_tasks: set[asyncio.Task[None]] = set()
174
+ self._restart_lock = asyncio.Lock()
175
+ self._shutdown_started = False
176
+
177
+ @staticmethod
178
+ def _sanitize_tool_result_for_model(tool_name: str, tool_result: dict[str, Any]) -> dict[str, Any]:
179
+ """Remove bulky transport-only fields before echoing tool output back to the model."""
180
+ if tool_name == "camera" and "b64_im" in tool_result:
181
+ sanitized = dict(tool_result)
182
+ sanitized.pop("b64_im", None)
183
+ sanitized["image_attached"] = True
184
+ return sanitized
185
+ return tool_result
186
+
187
+ def _normalize_startup_voice(self, voice: str | None) -> str | None:
188
+ """Return a valid persisted startup voice, or None."""
189
+ return self._resolve_backend_voice(voice, source="persisted startup voice")
190
+
191
+ def _restart_task_done(self, task: asyncio.Task[None]) -> None:
192
+ """Retrieve and sanitize background restart failures."""
193
+ self._restart_tasks.discard(task)
194
+ if task.cancelled():
195
+ return
196
+ try:
197
+ task.result()
198
+ except Exception as exc:
199
+ logger.warning("Realtime session restart failed: %s", redact_secrets(exc))
200
+
201
+ async def _cancel_restart_tasks(self) -> None:
202
+ """Cancel and retrieve retained restart tasks during handler shutdown."""
203
+ restart_tasks = tuple(self._restart_tasks)
204
+ for task in restart_tasks:
205
+ task.cancel()
206
+ if restart_tasks:
207
+ await asyncio.gather(*restart_tasks, return_exceptions=True)
208
+ self._restart_tasks.clear()
209
+
210
+ async def _wait_for_response_done_before_tool_result(self) -> bool:
211
+ """Return whether the function-call response finished before sending tool output."""
212
+ if self._response_done_event.is_set():
213
+ return True
214
+
215
+ try:
216
+ await asyncio.wait_for(
217
+ self._response_done_event.wait(),
218
+ timeout=_RESPONSE_DONE_TIMEOUT,
219
+ )
220
+ return True
221
+ except asyncio.TimeoutError:
222
+ return False
223
+
224
+ def _resolve_backend_voice(
225
+ self,
226
+ voice: str | None,
227
+ *,
228
+ source: str,
229
+ fallback: str | None = None,
230
+ ) -> str | None:
231
+ """Return a backend-supported voice, optionally falling back when unsupported."""
232
+ available_voices = get_available_voices()
233
+ voice_value = (voice or "").strip()
234
+ if not voice_value:
235
+ return fallback
236
+
237
+ voice_by_lowercase = {candidate.lower(): candidate for candidate in available_voices}
238
+ normalized_voice = voice_by_lowercase.get(voice_value.lower())
239
+ if normalized_voice is not None:
240
+ return normalized_voice
241
+
242
+ if voice:
243
+ logger.warning(
244
+ "Ignoring unsupported %s %r; expected one of %s",
245
+ source,
246
+ voice,
247
+ available_voices,
248
+ )
249
+ return fallback
250
+
251
+ def _get_session_config(self, tool_specs: list[ToolSpec]) -> RealtimeSessionCreateRequestParam:
252
+ """Return the Hugging Face OpenAI-compatible session config."""
253
+ return RealtimeSessionCreateRequestParam(
254
+ type="realtime",
255
+ instructions=get_session_instructions(self.instance_path),
256
+ audio=RealtimeAudioConfigParam(
257
+ input=RealtimeAudioConfigInputParam(
258
+ # The OpenAI SDK type only includes 24 kHz PCM, but the HF
259
+ # compatible server uses rate=None for native 16 kHz mode.
260
+ format=_native_rate_audio_pcm(), # type: ignore[typeddict-item]
261
+ transcription=AudioTranscriptionParam(
262
+ model="gpt-4o-transcribe",
263
+ language=config.REALTIME_TRANSCRIPTION_LANGUAGE,
264
+ ),
265
+ turn_detection=ServerVad(type="server_vad", interrupt_response=True),
266
+ ),
267
+ output=RealtimeAudioConfigOutputParam(
268
+ format=_native_rate_audio_pcm(), # type: ignore[typeddict-item]
269
+ voice=self.get_current_voice(),
270
+ ),
271
+ ),
272
+ tools=to_realtime_tools_config(tool_specs),
273
+ tool_choice="auto",
274
+ )
275
+
276
+ def _is_connected(self) -> bool:
277
+ """Return whether the realtime connection is open."""
278
+ return self.connection is not None
279
+
280
+ def _idle_behavior_ready(self) -> bool:
281
+ """Hold idle behavior while a model response is still active."""
282
+ return self._response_done_event.is_set()
283
+
284
+ async def _cancel_partial_transcript_task(self) -> None:
285
+ if self.partial_transcript_task and not self.partial_transcript_task.done():
286
+ self.partial_transcript_task.cancel()
287
+ try:
288
+ await self.partial_transcript_task
289
+ except asyncio.CancelledError:
290
+ pass
291
+
292
+ async def change_voice(self, voice: str) -> str:
293
+ """Change only the voice, updating the active session when possible."""
294
+ default_voice = get_default_voice()
295
+ resolved_voice = (
296
+ self._resolve_backend_voice(voice, source="requested voice", fallback=default_voice) or default_voice
297
+ )
298
+ self._voice_override = resolved_voice
299
+ if self.connection is not None:
300
+ try:
301
+ await self.connection.session.update(
302
+ session=RealtimeSessionCreateRequestParam(
303
+ type="realtime",
304
+ audio=RealtimeAudioConfigParam(
305
+ output=RealtimeAudioConfigOutputParam(
306
+ voice=resolved_voice,
307
+ ),
308
+ ),
309
+ ),
310
+ )
311
+ return f"Voice changed to {resolved_voice}."
312
+ except Exception as e:
313
+ logger.warning("Failed to update live session for voice change: %s", redact_secrets(e))
314
+ return "Voice change failed. Will take effect on next connection."
315
+ return "Voice changed. Will take effect on next connection."
316
+
317
+ def get_current_voice(self) -> str:
318
+ """Return the voice currently selected for this handler."""
319
+ default_voice = get_default_voice()
320
+ voice = self._voice_override or get_session_voice(default=default_voice)
321
+ return self._resolve_backend_voice(voice, source="session voice", fallback=default_voice) or default_voice
322
+
323
+ async def apply_personality(self, profile: str | None) -> str:
324
+ """Apply a new personality (profile) at runtime if possible.
325
+
326
+ - Updates the global config's selected profile for subsequent calls.
327
+ - If a realtime connection is active, sends a session.update with the
328
+ freshly resolved instructions so the change takes effect immediately.
329
+
330
+ Returns a short status message for UI feedback.
331
+ """
332
+ try:
333
+ # Update the in-process config value and env
334
+ from peachy.config import config as _config
335
+ from peachy.config import set_custom_profile
336
+
337
+ set_custom_profile(profile)
338
+ logger.info(
339
+ "Set custom profile to %r (config=%r)", profile, getattr(_config, "REACHY_MINI_CUSTOM_PROFILE", None)
340
+ )
341
+
342
+ try:
343
+ instructions = get_session_instructions(self.instance_path)
344
+ voice = self.get_current_voice()
345
+ except Exception as e:
346
+ safe_error = redact_secrets(e)
347
+ logger.error("Failed to resolve personality content: %s", safe_error)
348
+ return f"Failed to apply personality: {safe_error}"
349
+
350
+ # Rebuild the tool registry
351
+ core_tools.initialize_tools(force=True)
352
+
353
+ # Attempt a live update first, then force a full restart to ensure it sticks
354
+ if self.connection is not None:
355
+ try:
356
+ await self.connection.session.update(
357
+ session=RealtimeSessionCreateRequestParam(
358
+ type="realtime",
359
+ instructions=instructions,
360
+ audio=RealtimeAudioConfigParam(
361
+ output=RealtimeAudioConfigOutputParam(
362
+ voice=voice,
363
+ ),
364
+ ),
365
+ ),
366
+ )
367
+ logger.info("Applied personality via live update: %s", profile or "built-in default")
368
+ except Exception as e:
369
+ logger.warning("Live update failed; will restart session: %s", redact_secrets(e))
370
+
371
+ # Force a real restart to guarantee the new instructions/voice
372
+ try:
373
+ await self._restart_session()
374
+ return "Applied personality and restarted realtime session."
375
+ except Exception as e:
376
+ logger.warning("Failed to restart session after apply: %s", redact_secrets(e))
377
+ return "Applied personality. Will take effect on next connection."
378
+ else:
379
+ logger.info(
380
+ "Applied personality recorded: %s (no live connection; will apply on next session)",
381
+ profile or "built-in default",
382
+ )
383
+ return "Applied personality. Will take effect on next connection."
384
+ except Exception as e:
385
+ safe_error = redact_secrets(e)
386
+ logger.error("Error applying personality '%s': %s", profile, safe_error)
387
+ return f"Failed to apply personality: {safe_error}"
388
+
389
+ async def _emit_debounced_partial(self, transcript: str, item_id: str, sequence_counter: int) -> None:
390
+ """Emit partial transcript after debounce delay."""
391
+ try:
392
+ await asyncio.sleep(self.partial_debounce_delay)
393
+
394
+ input_transcript = self.input_transcript_chunks_by_item
395
+ if input_transcript.item_id == item_id and len(input_transcript.deltas) - 1 == sequence_counter:
396
+ await self.output_queue.put(AdditionalOutputs({"role": "user_partial", "content": transcript}))
397
+ logger.debug(f"Debounced partial emitted: {transcript}")
398
+ except asyncio.CancelledError:
399
+ logger.debug("Debounced partial cancelled")
400
+ raise
401
+
402
+ def _record_partial_transcript_delta(
403
+ self,
404
+ input_transcript: InputTranscriptChunksByItem,
405
+ item_id: str,
406
+ delta: str,
407
+ ) -> None:
408
+ """Record a Hugging Face partial transcript snapshot."""
409
+ input_transcript.item_id = item_id
410
+ input_transcript.deltas = [delta]
411
+
412
+ async def start_up(self) -> None:
413
+ """Start the handler with minimal retries on unexpected websocket closure."""
414
+ self.client = await self._build_realtime_client()
415
+
416
+ max_attempts = 3
417
+ for attempt in range(1, max_attempts + 1):
418
+ try:
419
+ await self._run_realtime_session()
420
+ # Normal exit from the session, stop retrying
421
+ return
422
+ except ConnectionClosedError as e:
423
+ # Abrupt close (e.g., "no close frame received or sent") → retry
424
+ logger.warning(
425
+ "Realtime websocket closed unexpectedly (attempt %d/%d): %s",
426
+ attempt,
427
+ max_attempts,
428
+ redact_secrets(e),
429
+ )
430
+ if attempt < max_attempts:
431
+ self.client = await self._build_realtime_client()
432
+ # exponential backoff with jitter
433
+ base_delay = 2 ** (attempt - 1) # 1s, 2s, 4s, 8s, etc.
434
+ jitter = random.uniform(0, 0.5)
435
+ delay = base_delay + jitter
436
+ logger.info("Retrying in %.1f seconds...", delay)
437
+ await asyncio.sleep(delay)
438
+ continue
439
+ raise
440
+ finally:
441
+ # never keep a stale reference
442
+ self.connection = None
443
+ try:
444
+ self._connected_event.clear()
445
+ except Exception:
446
+ pass
447
+
448
+ async def _restart_session(self) -> None:
449
+ """Force-close the current session and start a fresh one in background.
450
+
451
+ Does not block the caller while the new session is establishing.
452
+ """
453
+ try:
454
+ async with self._restart_lock:
455
+ if self._shutdown_started:
456
+ return
457
+
458
+ if self.connection is not None:
459
+ try:
460
+ await self.connection.close()
461
+ except Exception:
462
+ pass
463
+ finally:
464
+ self.connection = None
465
+
466
+ # Ensure we have a client (start_up must have run once)
467
+ if getattr(self, "client", None) is None:
468
+ logger.warning("Cannot restart: realtime client not initialized yet.")
469
+ return
470
+
471
+ # Build before registering the session task, then re-check the
472
+ # shutdown gate because client construction may have suspended.
473
+ try:
474
+ self._connected_event.clear()
475
+ except Exception:
476
+ pass
477
+ client = await self._build_realtime_client()
478
+ if self._shutdown_started:
479
+ return
480
+ self.client = client
481
+ restart_task = asyncio.create_task(
482
+ self._run_realtime_session(),
483
+ name="realtime-session-restart",
484
+ )
485
+ self._restart_tasks.add(restart_task)
486
+ restart_task.add_done_callback(self._restart_task_done)
487
+ try:
488
+ await asyncio.wait_for(self._connected_event.wait(), timeout=5.0)
489
+ if not self._shutdown_started:
490
+ logger.info("Realtime session restarted and connected.")
491
+ except asyncio.TimeoutError:
492
+ if not self._shutdown_started:
493
+ logger.warning("Realtime session restart timed out; continuing in background.")
494
+ except asyncio.CancelledError:
495
+ raise
496
+ except Exception as e:
497
+ logger.warning("_restart_session failed: %s", redact_secrets(e))
498
+
499
+ async def _safe_response_create(self, **kwargs: Any) -> None:
500
+ """Enqueue a response.create() kwargs for the sender worker _response_sender_loop().
501
+
502
+ This method never blocks the caller.
503
+ """
504
+ await self._pending_responses.put(kwargs)
505
+
506
+ async def _send_startup_greeting_prompt(self) -> None:
507
+ """Prompt the model to open the conversation once the session is ready."""
508
+ if self._startup_greeting_sent or not self.connection:
509
+ return
510
+
511
+ greeting_prompt = get_session_greeting_prompt().strip()
512
+ if not greeting_prompt:
513
+ self._startup_greeting_sent = True
514
+ return
515
+
516
+ try:
517
+ await self.connection.conversation.item.create(
518
+ item={
519
+ "type": "message",
520
+ "role": "user",
521
+ "content": [
522
+ {
523
+ "type": "input_text",
524
+ "text": greeting_prompt,
525
+ },
526
+ ],
527
+ },
528
+ )
529
+ self._startup_greeting_sent = True
530
+ self._mark_activity("startup_greeting_prompt")
531
+ await self._safe_response_create()
532
+ logger.info("Queued startup greeting prompt")
533
+ except Exception as e:
534
+ logger.warning("Failed to queue startup greeting prompt: %s", redact_secrets(e))
535
+
536
+ async def _response_sender_loop(self) -> None:
537
+ """Dedicated worker that sends ``response.create()`` calls serially.
538
+
539
+ This logic was designed to comply with the response.create() docstring specification for event ordering:
540
+ https://github.com/openai/openai-python/blob/3e0c05b84a2056870abf3bd6a5e7849020209cc3/src/openai/resources/realtime/realtime.py#L649C1-L651C30
541
+
542
+ For each queued request the worker:
543
+ 1. Waits until no response is active (_response_done_event).
544
+ 2. Sends response.create().
545
+ 3. Waits until the receiver observes response.created or a rejection.
546
+ 4. Waits for the response cycle to complete (response.done).
547
+ 5. If the server rejected with active_response, retries from step 1.
548
+ """
549
+ while self.connection:
550
+ try:
551
+ kwargs = await self._pending_responses.get()
552
+ except asyncio.CancelledError:
553
+ return
554
+
555
+ # Parallel tool calls enqueue duplicate empty requests; coalesce to one.
556
+ while not kwargs and not self._pending_responses.empty():
557
+ try:
558
+ self._pending_responses.get_nowait()
559
+ except asyncio.QueueEmpty:
560
+ break
561
+
562
+ sent = False
563
+ max_retries = 5
564
+ attempts = 0
565
+ while not sent and self.connection and attempts < max_retries:
566
+ try:
567
+ await asyncio.wait_for(
568
+ self._response_done_event.wait(),
569
+ timeout=_RESPONSE_DONE_TIMEOUT,
570
+ )
571
+ except asyncio.TimeoutError:
572
+ logger.debug("Timed out waiting for previous response to finish; forcing ahead")
573
+ self._response_done_event.set()
574
+
575
+ if not self.connection:
576
+ break
577
+
578
+ self._last_response_rejected = False
579
+ self._response_started_or_rejected_event.clear()
580
+ try:
581
+ await self.connection.response.create(**kwargs)
582
+ except Exception as e:
583
+ logger.debug("_response_sender_loop: send failed: %s", redact_secrets(e))
584
+ self._response_done_event.set()
585
+ break
586
+
587
+ try:
588
+ await asyncio.wait_for(
589
+ self._response_started_or_rejected_event.wait(),
590
+ timeout=_RESPONSE_DONE_TIMEOUT,
591
+ )
592
+ except asyncio.TimeoutError:
593
+ logger.debug("Timed out waiting for response.created or response rejection")
594
+
595
+ # Check if the receiver loop observed an asynchronous rejection.
596
+ if self._last_response_rejected:
597
+ attempts += 1
598
+ if attempts >= max_retries:
599
+ logger.debug("response.create rejected %d times; giving up", attempts)
600
+ break
601
+ logger.debug("response.create was rejected; retrying (%d/%d)", attempts, max_retries)
602
+ await asyncio.sleep(_RESPONSE_REJECTION_RETRY_DELAY)
603
+ continue
604
+
605
+ try:
606
+ await asyncio.wait_for(
607
+ self._response_done_event.wait(),
608
+ timeout=_RESPONSE_DONE_TIMEOUT,
609
+ )
610
+ except asyncio.TimeoutError:
611
+ logger.debug("Timed out waiting for response.done; assuming response completed")
612
+ self._response_done_event.set()
613
+ break
614
+
615
+ sent = True
616
+
617
+ async def _handle_tool_result(self, completed_tool: ToolNotification) -> None:
618
+ """Process the result of a tool call."""
619
+ if completed_tool.error is not None:
620
+ logger.error(
621
+ "Tool '%s' (id=%s) failed with error: %s",
622
+ completed_tool.tool_name,
623
+ completed_tool.id,
624
+ completed_tool.error,
625
+ )
626
+ tool_result = {"error": completed_tool.error}
627
+ tool_result_for_model = tool_result
628
+ elif completed_tool.result is not None:
629
+ tool_result = completed_tool.result
630
+ tool_result_for_model = (
631
+ self._sanitize_tool_result_for_model(completed_tool.tool_name, tool_result)
632
+ if isinstance(tool_result, dict)
633
+ else tool_result
634
+ )
635
+ logger.info(
636
+ "Tool '%s' (id=%s) executed successfully.",
637
+ completed_tool.tool_name,
638
+ completed_tool.id,
639
+ )
640
+ logger.debug("Tool '%s' model-visible result: %s", completed_tool.tool_name, tool_result_for_model)
641
+ else:
642
+ logger.warning(
643
+ "Tool '%s' (id=%s) returned no result and no error", completed_tool.tool_name, completed_tool.id
644
+ )
645
+ tool_result = {"error": "No result returned from tool execution"}
646
+ tool_result_for_model = tool_result
647
+
648
+ # Connection may have closed while tool was running
649
+ if not self.connection:
650
+ logger.warning(
651
+ "Connection closed during tool '%s' (id=%s) execution; cannot send result back",
652
+ completed_tool.tool_name,
653
+ completed_tool.id,
654
+ )
655
+ return
656
+
657
+ try:
658
+ send_result_to_model = not completed_tool.is_idle_tool_call
659
+ if send_result_to_model:
660
+ self._mark_activity("tool_result_ready")
661
+ model_result_submitted = False
662
+ if send_result_to_model and isinstance(completed_tool.id, str):
663
+ if not await self._wait_for_response_done_before_tool_result():
664
+ send_result_to_model = False
665
+ if not send_result_to_model:
666
+ logger.warning(
667
+ "Dropping realtime model result for tool '%s' (id=%s) because response.done was not observed",
668
+ completed_tool.tool_name,
669
+ completed_tool.id,
670
+ )
671
+ elif not self.connection:
672
+ logger.warning(
673
+ "Connection closed before sending tool '%s' (id=%s) result back",
674
+ completed_tool.tool_name,
675
+ completed_tool.id,
676
+ )
677
+ return
678
+ else:
679
+ await self.connection.conversation.item.create(
680
+ item={
681
+ "type": "function_call_output",
682
+ "call_id": completed_tool.id,
683
+ "output": json.dumps(tool_result_for_model),
684
+ },
685
+ )
686
+ model_result_submitted = True
687
+
688
+ await self.output_queue.put(
689
+ AdditionalOutputs(
690
+ {
691
+ "role": "assistant",
692
+ "content": json.dumps(tool_result_for_model),
693
+ },
694
+ ),
695
+ )
696
+
697
+ if model_result_submitted and completed_tool.tool_name == "camera" and "b64_im" in tool_result:
698
+ # use raw base64, don't json.dumps (which adds quotes)
699
+ b64_im = tool_result["b64_im"]
700
+ if not isinstance(b64_im, str):
701
+ logger.warning("Unexpected type for b64_im: %s", type(b64_im))
702
+ b64_im = str(b64_im)
703
+ image_width = tool_result.get("image_width")
704
+ image_height = tool_result.get("image_height")
705
+ jpeg_bytes_value = tool_result.get("jpeg_bytes")
706
+ jpeg_bytes = jpeg_bytes_value if isinstance(jpeg_bytes_value, int) else (len(b64_im) * 3) // 4
707
+ await self.connection.conversation.item.create(
708
+ item={
709
+ "type": "message",
710
+ "role": "user",
711
+ "content": [
712
+ {
713
+ "type": "input_image",
714
+ "image_url": f"data:image/jpeg;base64,{b64_im}",
715
+ },
716
+ ],
717
+ },
718
+ )
719
+ if isinstance(image_width, int) and isinstance(image_height, int):
720
+ logger.info(
721
+ "Added camera image to conversation frame=%sx%s jpeg_bytes=%s",
722
+ image_width,
723
+ image_height,
724
+ jpeg_bytes,
725
+ )
726
+ else:
727
+ logger.info(
728
+ "Added camera image to conversation jpeg_bytes=%s",
729
+ jpeg_bytes,
730
+ )
731
+
732
+ if isinstance(completed_tool.id, str):
733
+ self._in_flight_tool_calls.discard(completed_tool.id)
734
+
735
+ tool = core_tools.ALL_TOOLS.get(completed_tool.tool_name)
736
+ # Always surface errors, skip the spoken follow-up for tools that opt out.
737
+ if model_result_submitted and (completed_tool.error is not None or tool is None or tool.needs_response):
738
+ self._tool_batch_needs_response = True
739
+
740
+ # Parallel tool calls in one turn: respond once every result is in, not per tool.
741
+ if self._tool_batch_needs_response and not self._in_flight_tool_calls:
742
+ self._tool_batch_needs_response = False
743
+ await self._safe_response_create()
744
+
745
+ except ConnectionClosedError:
746
+ logger.warning("Connection closed while sending tool result")
747
+ self.connection = None
748
+ self._response_done_event.set()
749
+
750
+ async def _run_realtime_session(self) -> None:
751
+ """Establish and manage a single realtime session."""
752
+ tool_specs = get_tool_specs()
753
+ logger.info(
754
+ "Tools to be used in conversation: %s",
755
+ [tool["name"] for tool in tool_specs],
756
+ )
757
+ connect_kwargs: dict[str, Any] = {}
758
+ if self._realtime_connect_query:
759
+ connect_kwargs["extra_query"] = self._realtime_connect_query
760
+ async with self.client.realtime.connect(**connect_kwargs) as conn:
761
+ try:
762
+ session_config = self._get_session_config(tool_specs)
763
+ await conn.session.update(session=session_config)
764
+ logger.info(
765
+ "Realtime session initialized with profile=%r voice=%r",
766
+ getattr(config, "REACHY_MINI_CUSTOM_PROFILE", None),
767
+ self.get_current_voice(),
768
+ )
769
+ except Exception as exc:
770
+ logger.error(
771
+ "Realtime session.update failed; aborting startup: %s",
772
+ redact_secrets(exc),
773
+ )
774
+ raise
775
+
776
+ logger.info("Realtime session updated successfully")
777
+
778
+ # Reset the partial-transcript accumulator for each new session
779
+ self.input_transcript_chunks_by_item = InputTranscriptChunksByItem()
780
+
781
+ # Manage events received from the realtime server.
782
+ self.connection = conn
783
+ try:
784
+ self._connected_event.set()
785
+ except Exception:
786
+ pass
787
+
788
+ response_sender_task: asyncio.Task[None] | None = None
789
+ try:
790
+ # Start the background tool manager
791
+ self.tool_manager.start_up(tool_callbacks=[self._handle_tool_result])
792
+
793
+ # Start the response sender worker
794
+ response_sender_task = asyncio.create_task(self._response_sender_loop(), name="response-sender")
795
+ await self._send_startup_greeting_prompt()
796
+
797
+ async for event in self.connection:
798
+ logger.debug("Realtime event: %s", event.type)
799
+ if event.type == "input_audio_buffer.speech_started":
800
+ self._mark_activity("user_speech_started")
801
+ self._turn_user_done_at = None
802
+ self._turn_response_created_at = None
803
+ self._turn_first_audio_at = None
804
+ if self._clear_queue:
805
+ self._clear_queue()
806
+ self.deps.movement_manager.set_listening(True)
807
+ logger.debug("User speech started")
808
+
809
+ if event.type == "input_audio_buffer.speech_stopped":
810
+ self._mark_activity("user_speech_stopped")
811
+ self.deps.movement_manager.set_listening(False)
812
+ logger.debug("User speech stopped - server will auto-commit with VAD")
813
+
814
+ if event.type == "response.output_audio.done":
815
+ self.deps.movement_manager.set_speaking(False)
816
+ logger.debug("response completed")
817
+
818
+ if event.type == "response.output_text.delta":
819
+ logger.debug("response text delta")
820
+
821
+ if event.type == "response.output_text.done":
822
+ logger.debug("response text done: %s", event.text)
823
+
824
+ if event.type == "response.created":
825
+ self._mark_activity("response_created")
826
+ self.deps.movement_manager.set_speaking(True)
827
+ self._response_done_event.clear()
828
+ self._response_started_or_rejected_event.set()
829
+ if self._turn_user_done_at is not None and self._turn_response_created_at is None:
830
+ self._turn_response_created_at = time.perf_counter()
831
+ delta_ms = (self._turn_response_created_at - self._turn_user_done_at) * 1000
832
+ logger.info("Turn latency: response.created %.0f ms after user transcript", delta_ms)
833
+ logger.debug("Response created (active)")
834
+
835
+ if event.type == "response.done":
836
+ # Doesn't mean the audio is done playing
837
+ # Resume tracking for responses that emit no audio (text-only / tool-only).
838
+ self.deps.movement_manager.set_speaking(False)
839
+ self._response_done_event.set()
840
+ self._response_started_or_rejected_event.set()
841
+ logger.debug("Response done")
842
+
843
+ if event.type == "conversation.item.input_audio_transcription.delta":
844
+ self._mark_activity("user_transcription_delta")
845
+ logger.debug(f"User partial transcript: {event.delta}")
846
+
847
+ item_id = event.item_id
848
+ delta = event.delta or ""
849
+
850
+ input_transcript = self.input_transcript_chunks_by_item
851
+ self._record_partial_transcript_delta(input_transcript, item_id, delta)
852
+
853
+ current_partial = "".join(input_transcript.deltas)
854
+ sequence_counter = len(input_transcript.deltas) - 1
855
+
856
+ await self._cancel_partial_transcript_task()
857
+
858
+ # Start new debounce timer with the last delta
859
+ self.partial_transcript_task = asyncio.create_task(
860
+ self._emit_debounced_partial(current_partial, item_id, sequence_counter)
861
+ )
862
+
863
+ # Handle completed transcription (user finished speaking)
864
+ if event.type == "conversation.item.input_audio_transcription.completed":
865
+ self._mark_activity("user_transcription_completed")
866
+ raw_transcript = event.transcript or ""
867
+ transcript = raw_transcript.strip()
868
+ logger.debug("User transcript: %s", raw_transcript)
869
+ self.deps.movement_manager.set_listening(False)
870
+
871
+ await self._cancel_partial_transcript_task()
872
+
873
+ if not transcript:
874
+ logger.debug("Ignoring empty user transcript")
875
+ continue
876
+
877
+ self._turn_user_done_at = time.perf_counter()
878
+ self._turn_response_created_at = None
879
+ self._turn_first_audio_at = None
880
+ self._in_flight_tool_calls.clear()
881
+ self._tool_batch_needs_response = False
882
+
883
+ await self.output_queue.put(AdditionalOutputs({"role": "user", "content": transcript}))
884
+
885
+ # Handle assistant transcription
886
+ if event.type == "response.output_audio_transcript.done":
887
+ self._mark_activity("assistant_transcript_done")
888
+ logger.debug(f"Assistant transcript: {event.transcript}")
889
+ await self.output_queue.put(
890
+ AdditionalOutputs({"role": "assistant", "content": event.transcript})
891
+ )
892
+
893
+ # Handle audio delta
894
+ if event.type == "response.output_audio.delta":
895
+ decoded_pcm_bytes = base64.b64decode(event.delta)
896
+ decoded_pcm = np.frombuffer(decoded_pcm_bytes, dtype=np.int16).reshape(1, -1)
897
+ self._mark_activity("assistant_audio_delta")
898
+ if self._turn_user_done_at is not None and self._turn_first_audio_at is None:
899
+ self._turn_first_audio_at = time.perf_counter()
900
+ delta_ms = (self._turn_first_audio_at - self._turn_user_done_at) * 1000
901
+ logger.info("Turn latency: first audio delta %.0f ms after user transcript", delta_ms)
902
+ await self.output_queue.put(
903
+ (
904
+ self.SAMPLE_RATE,
905
+ decoded_pcm,
906
+ ),
907
+ )
908
+ # ---- tool-calling plumbing ----
909
+ if event.type == "response.function_call_arguments.done":
910
+ self._mark_activity("tool_call_received")
911
+ tool_name = getattr(event, "name", None)
912
+ args_json_str = getattr(event, "arguments", None)
913
+ call_id: str = str(getattr(event, "call_id", uuid.uuid4()))
914
+
915
+ logger.info(
916
+ "Tool call received — tool_name=%r, call_id=%s, args=%s",
917
+ tool_name,
918
+ call_id,
919
+ args_json_str,
920
+ )
921
+
922
+ if not isinstance(tool_name, str) or not isinstance(args_json_str, str):
923
+ logger.error(
924
+ "Invalid tool call: tool_name=%s (type=%s), args=%s (type=%s), call_id=%s",
925
+ tool_name,
926
+ type(tool_name).__name__,
927
+ args_json_str,
928
+ type(args_json_str).__name__,
929
+ call_id,
930
+ )
931
+ continue
932
+
933
+ self._in_flight_tool_calls.add(call_id)
934
+ background_tool = await self.tool_manager.start_tool(
935
+ call_id=call_id,
936
+ tool_call_routine=ToolCallRoutine(
937
+ tool_name=tool_name,
938
+ args_json_str=args_json_str,
939
+ deps=self.deps,
940
+ ),
941
+ is_idle_tool_call=False,
942
+ )
943
+
944
+ await self.output_queue.put(
945
+ AdditionalOutputs(
946
+ {
947
+ "role": "assistant",
948
+ "content": f"🛠️ Used tool {tool_name} with args {args_json_str}. The tool is now running. Tool ID: {background_tool.tool_id}",
949
+ },
950
+ ),
951
+ )
952
+ logger.info(
953
+ "Started background tool: %s (id=%s, call_id=%s)",
954
+ tool_name,
955
+ background_tool.tool_id,
956
+ call_id,
957
+ )
958
+
959
+ # server error
960
+ if event.type == "error":
961
+ err = getattr(event, "error", None)
962
+ msg = getattr(err, "message", str(err) if err else "unknown error")
963
+ code = getattr(err, "code", "") or getattr(err, "type", "")
964
+
965
+ if code == "conversation_already_has_active_response":
966
+ # response.create was rejected. The sender worker
967
+ # is waiting on _response_done_event; when the active
968
+ # response finishes it will wake up and see this flag.
969
+ self._last_response_rejected = True
970
+ self._response_started_or_rejected_event.set()
971
+ logger.debug("response.create rejected; worker will retry after active response finishes")
972
+ else:
973
+ self._response_started_or_rejected_event.set()
974
+ logger.error(
975
+ "Realtime error [%s]: %s (raw=%s)",
976
+ code,
977
+ redact_secrets(msg),
978
+ redact_secrets(err),
979
+ )
980
+
981
+ if code == "input_audio_buffer_commit_empty":
982
+ self.deps.movement_manager.set_listening(False)
983
+
984
+ # Only show user-facing errors, not internal state errors.
985
+ if code not in (
986
+ "input_audio_buffer_commit_empty",
987
+ "conversation_already_has_active_response",
988
+ ):
989
+ await self.output_queue.put(
990
+ AdditionalOutputs(
991
+ {
992
+ "role": "assistant",
993
+ "content": f"[error] {redact_secrets(msg)}",
994
+ }
995
+ )
996
+ )
997
+ finally:
998
+ # Stop the response sender worker.
999
+ if response_sender_task is not None:
1000
+ response_sender_task.cancel()
1001
+ try:
1002
+ await response_sender_task
1003
+ except asyncio.CancelledError:
1004
+ pass
1005
+
1006
+ # Stop background tool manager tasks (listener + cleanup) in all paths.
1007
+ await self.tool_manager.shutdown()
1008
+
1009
+ # Microphone receive
1010
+ async def receive(self, frame: AudioFrame) -> None:
1011
+ """Receive audio frame from the microphone and send it to the realtime server.
1012
+
1013
+ Handles both mono and stereo audio formats, converting to the expected
1014
+ mono format for the realtime API.
1015
+
1016
+ Args:
1017
+ frame: A tuple containing (sample_rate, audio_data).
1018
+
1019
+ """
1020
+ if not self.connection:
1021
+ return
1022
+
1023
+ _, audio_frame = frame
1024
+ if audio_frame.size == 0:
1025
+ return
1026
+
1027
+ # Reshape if needed
1028
+ if audio_frame.ndim == 2:
1029
+ # channels-last convention
1030
+ if audio_frame.shape[1] > audio_frame.shape[0]:
1031
+ audio_frame = audio_frame.T
1032
+ # Multiple channels -> Mono channel
1033
+ if audio_frame.shape[1] > 1:
1034
+ audio_frame = audio_frame[:, 0]
1035
+
1036
+ # Cast if needed
1037
+ audio_frame = audio_to_int16(audio_frame)
1038
+
1039
+ # Send to the realtime input buffer (guard against races during reconnect).
1040
+ try:
1041
+ audio_message = base64.b64encode(audio_frame.tobytes()).decode("utf-8")
1042
+ await self.connection.input_audio_buffer.append(audio=audio_message)
1043
+ except Exception as e:
1044
+ logger.debug("Dropping audio frame: connection not ready (%s)", redact_secrets(e))
1045
+ return
1046
+
1047
+ async def shutdown(self) -> None:
1048
+ """Shutdown the handler."""
1049
+ self._shutdown_started = True
1050
+ self._connected_event.set()
1051
+
1052
+ # Wait for any restart still constructing its client to observe the
1053
+ # shutdown gate before taking the complete registered-task snapshot.
1054
+ async with self._restart_lock:
1055
+ pass
1056
+
1057
+ # Unblock the response sender worker so it can exit
1058
+ self._response_done_event.set()
1059
+
1060
+ await self._cancel_restart_tasks()
1061
+
1062
+ # Stop background tool manager tasks (listener + cleanup)
1063
+ await self.tool_manager.shutdown()
1064
+
1065
+ await self._cancel_partial_transcript_task()
1066
+
1067
+ if self.connection:
1068
+ try:
1069
+ await self.connection.close()
1070
+ except ConnectionClosedError as e:
1071
+ logger.debug("Connection already closed during shutdown: %s", redact_secrets(e))
1072
+ except Exception as e:
1073
+ logger.debug("connection.close() ignored: %s", redact_secrets(e))
1074
+ finally:
1075
+ self.connection = None
1076
+
1077
+ # Clear any remaining items in the output queue
1078
+ while not self.output_queue.empty():
1079
+ try:
1080
+ self.output_queue.get_nowait()
1081
+ except asyncio.QueueEmpty:
1082
+ break
1083
+
1084
+ async def get_available_voices(self) -> list[str]:
1085
+ """Return the available Hugging Face voices."""
1086
+ return get_available_voices()
1087
+
1088
+ async def _build_realtime_client(self) -> AsyncOpenAI:
1089
+ """Build the Hugging Face OpenAI-compatible realtime client."""
1090
+ bearer_token = (config.HF_TOKEN or "").strip()
1091
+ connection_selection = get_hf_connection_selection()
1092
+ direct_realtime_url = get_hf_direct_ws_url()
1093
+ if connection_selection.mode == HF_LOCAL_CONNECTION_MODE:
1094
+ if not direct_realtime_url:
1095
+ raise RuntimeError("HF_REALTIME_WS_URL must be set when HF_REALTIME_CONNECTION_MODE=local")
1096
+ client, connect_query = _build_openai_compatible_client_from_realtime_url(
1097
+ direct_realtime_url,
1098
+ bearer_token,
1099
+ )
1100
+ self._realtime_connect_query = connect_query
1101
+ logger.info(
1102
+ "Using direct Hugging Face realtime endpoint %s",
1103
+ redact_secrets(_safe_endpoint_for_log(direct_realtime_url)),
1104
+ )
1105
+ return client
1106
+
1107
+ session_url = connection_selection.session_url
1108
+ if not session_url:
1109
+ raise RuntimeError("Built-in Hugging Face session proxy URL is unavailable")
1110
+ if direct_realtime_url:
1111
+ logger.info("HF_REALTIME_CONNECTION_MODE=deployed; ignoring HF_REALTIME_WS_URL.")
1112
+
1113
+ allocator_headers = {"Authorization": f"Bearer {bearer_token}"} if bearer_token else None
1114
+ async with httpx.AsyncClient(timeout=10.0) as http_client:
1115
+ response = await http_client.post(session_url, headers=allocator_headers)
1116
+ response.raise_for_status()
1117
+ payload = response.json()
1118
+
1119
+ connect_url = payload.get("connect_url")
1120
+ if not isinstance(connect_url, str) or not connect_url:
1121
+ raise RuntimeError(f"Session allocator response did not contain a valid connect_url: {payload!r}")
1122
+
1123
+ parsed_connect_url = parse_hf_realtime_url(connect_url)
1124
+ if not parsed_connect_url.has_realtime_path:
1125
+ raise ValueError(f"Expected realtime connect URL ending with /realtime, got: {connect_url}")
1126
+
1127
+ logger.info("Allocated realtime session %s", payload.get("session_id") or "<unknown>")
1128
+ client, connect_query = _build_openai_compatible_client_from_realtime_url(
1129
+ connect_url,
1130
+ bearer_token,
1131
+ )
1132
+ self._realtime_connect_query = connect_query
1133
+ return client
src/peachy/idle_policy.py ADDED
@@ -0,0 +1,140 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Local idle tool selection and dispatch.
2
+
3
+ Default idle weights among active, registered tools:
4
+ - IdleDoNothing: 60%
5
+ - Dance: 16%
6
+ - PlayEmotion: 16%
7
+ - MoveHead: 8%
8
+
9
+ Profiles or sessions can disable tools. When one or more idle tools are unavailable, selection is renormalized across the
10
+ remaining active candidates.
11
+ """
12
+ # Modified by Peachy contributors in 2026 for namespace, packaging, and provider integration.
13
+
14
+
15
+ from __future__ import annotations
16
+ import json
17
+ import uuid
18
+ import random
19
+ import asyncio
20
+ import logging
21
+ from typing import Any, Final
22
+ from dataclasses import dataclass
23
+ from collections.abc import Mapping, Callable, Iterable
24
+
25
+ from peachy.tools import core_tools
26
+ from peachy.streaming import AdditionalOutputs
27
+ from peachy.tools.dance import Dance
28
+ from peachy.tools.move_head import MoveHead
29
+ from peachy.tools.play_emotion import PlayEmotion
30
+ from peachy.tools.idle_do_nothing import IdleDoNothing
31
+ from peachy.tools.background_tool_manager import (
32
+ BackgroundTool,
33
+ ToolCallRoutine,
34
+ BackgroundToolManager,
35
+ )
36
+
37
+
38
+ logger = logging.getLogger(__name__)
39
+
40
+
41
+ @dataclass(frozen=True)
42
+ class IdleToolCandidate:
43
+ """Weighted idle tool candidate and argument factory."""
44
+
45
+ tool_type: type[core_tools.Tool]
46
+ weight: float
47
+ args_factory: Callable[[], dict[str, Any]]
48
+
49
+
50
+ def _no_args() -> dict[str, Any]:
51
+ return {}
52
+
53
+
54
+ def _idle_do_nothing_args() -> dict[str, Any]:
55
+ return {"reason": "random idle policy selected stillness"}
56
+
57
+
58
+ def _move_head_args() -> dict[str, Any]:
59
+ return {"direction": random.choice(tuple(MoveHead.DELTAS))}
60
+
61
+
62
+ _IDLE_TOOL_CANDIDATES: Final[tuple[IdleToolCandidate, ...]] = (
63
+ IdleToolCandidate(IdleDoNothing, 0.60, _idle_do_nothing_args),
64
+ IdleToolCandidate(Dance, 0.16, _no_args),
65
+ IdleToolCandidate(PlayEmotion, 0.16, _no_args),
66
+ IdleToolCandidate(MoveHead, 0.08, _move_head_args),
67
+ )
68
+
69
+
70
+ def choose_idle_tool_call(
71
+ available_tool_names: Iterable[str],
72
+ *,
73
+ tool_registry: Mapping[str, core_tools.Tool] | None = None,
74
+ ) -> tuple[str, dict[str, Any]] | None:
75
+ """Choose a weighted idle tool call from the tools available to the session."""
76
+ available = set(available_tool_names)
77
+ registry = core_tools.ALL_TOOLS if tool_registry is None else tool_registry
78
+ candidates = [
79
+ (tool.name, candidate)
80
+ for candidate in _IDLE_TOOL_CANDIDATES
81
+ for tool in registry.values()
82
+ if tool.name in available and isinstance(tool, candidate.tool_type)
83
+ ]
84
+ if not candidates:
85
+ return None
86
+
87
+ selected_name, selected_candidate = random.choices(
88
+ candidates,
89
+ weights=[candidate.weight for _, candidate in candidates],
90
+ k=1,
91
+ )[0]
92
+ return selected_name, selected_candidate.args_factory()
93
+
94
+
95
+ async def start_idle_tool_call(
96
+ *,
97
+ deps: core_tools.ToolDependencies,
98
+ tool_manager: BackgroundToolManager,
99
+ output_queue: asyncio.Queue[Any],
100
+ available_tool_names: Iterable[str],
101
+ idle_duration: float,
102
+ ) -> BackgroundTool | None:
103
+ """Start a locally selected idle tool and publish the console notification."""
104
+ selected_tool = choose_idle_tool_call(available_tool_names)
105
+ if selected_tool is None:
106
+ logger.warning("No idle tools are available; idle action skipped")
107
+ return None
108
+
109
+ tool_name, arguments = selected_tool
110
+ args_json_str = json.dumps(arguments)
111
+ call_id = f"idle-{uuid.uuid4()}"
112
+ background_tool = await tool_manager.start_tool(
113
+ call_id=call_id,
114
+ tool_call_routine=ToolCallRoutine(
115
+ tool_name=tool_name,
116
+ args_json_str=args_json_str,
117
+ deps=deps,
118
+ ),
119
+ is_idle_tool_call=True,
120
+ )
121
+ await output_queue.put(
122
+ AdditionalOutputs(
123
+ {
124
+ "role": "assistant",
125
+ "content": (
126
+ f"🛠️ Idle tool {tool_name} with args {args_json_str}. "
127
+ f"The tool is now running. Tool ID: {background_tool.tool_id}"
128
+ ),
129
+ },
130
+ ),
131
+ )
132
+ logger.info(
133
+ "Started local idle tool after %.1fs idle: %s (id=%s, call_id=%s, args=%s)",
134
+ idle_duration,
135
+ tool_name,
136
+ background_tool.tool_id,
137
+ call_id,
138
+ args_json_str,
139
+ )
140
+ return background_tool
src/peachy/main.py ADDED
@@ -0,0 +1,372 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Entrypoint for the Reachy Mini conversation app."""
2
+ # Modified by Peachy contributors in 2026 for namespace, packaging, and provider integration.
3
+
4
+
5
+ from __future__ import annotations
6
+ import sys
7
+ import time
8
+ import asyncio
9
+ import logging
10
+ import argparse
11
+ import threading
12
+ from typing import TYPE_CHECKING, Any, Optional
13
+ from pathlib import Path
14
+ from collections.abc import Callable, Awaitable
15
+
16
+ from fastapi import FastAPI, Request, Response
17
+
18
+ from reachy_mini import ReachyMini, ReachyMiniApp
19
+ from peachy import app_lifecycle
20
+ from peachy.state import resolve_state_root
21
+ from peachy.utils import (
22
+ parse_args,
23
+ setup_logger,
24
+ log_connection_troubleshooting,
25
+ )
26
+
27
+
28
+ if TYPE_CHECKING:
29
+ from peachy.console import LocalStream
30
+ from peachy.conversation_handler import ConversationHandler
31
+ from peachy.providers import ProviderDefinition
32
+ from peachy.tools.core_tools import ToolDependencies
33
+
34
+
35
+ def _build_selected_handler(
36
+ deps: "ToolDependencies",
37
+ instance_path: str | None,
38
+ startup_voice: str | None,
39
+ ) -> tuple["ProviderDefinition", "ConversationHandler"]:
40
+ """Build the handler selected by the explicit provider registry."""
41
+ from peachy.providers import get_provider_definition
42
+
43
+ definition = get_provider_definition()
44
+ return definition, definition.build_handler(deps, instance_path, startup_voice)
45
+
46
+
47
+ def _start_inactivity_timeout_thread(
48
+ timeout_minutes: float,
49
+ stream_manager: LocalStream,
50
+ logger: logging.Logger,
51
+ app_stop_event: threading.Event | None,
52
+ go_to_sleep: Callable[[], dict[str, Any]] | None = None,
53
+ ) -> threading.Thread:
54
+ """Start a daemon that puts the app to sleep after inactivity."""
55
+ timeout_seconds = timeout_minutes * 60.0
56
+
57
+ def poll_inactivity_timeout() -> None:
58
+ logger.info("App inactivity timeout enabled: %.1f minutes.", timeout_minutes)
59
+ while app_stop_event is None or not app_stop_event.is_set():
60
+ elapsed = stream_manager.seconds_since_activity()
61
+ if elapsed >= timeout_seconds:
62
+ logger.info("No activity for %.1f minutes; going to sleep.", elapsed / 60.0)
63
+ try:
64
+ if go_to_sleep is not None:
65
+ go_to_sleep()
66
+ else:
67
+ stream_manager.close()
68
+ except Exception as e:
69
+ logger.error("Error while going to sleep after inactivity timeout: %s", e)
70
+ try:
71
+ stream_manager.close()
72
+ except Exception as close_error:
73
+ logger.error("Error while closing stream manager after inactivity timeout: %s", close_error)
74
+ return
75
+ time.sleep(1.0)
76
+
77
+ thread = threading.Thread(target=poll_inactivity_timeout, daemon=True)
78
+ thread.start()
79
+ return thread
80
+
81
+
82
+ def main() -> None:
83
+ """Entrypoint for the Reachy Mini conversation app."""
84
+ args, _ = parse_args()
85
+ state_root = resolve_state_root()
86
+ if args.command == "tool-spaces":
87
+ from peachy.tool_spaces import handle_tool_spaces_command
88
+
89
+ logger = setup_logger(args.debug)
90
+ try:
91
+ raise SystemExit(handle_tool_spaces_command(args, instance_path=state_root))
92
+ except Exception as exc:
93
+ logger.error("tool-spaces command failed: %s", exc)
94
+ raise SystemExit(1) from exc
95
+ run(args, instance_path=str(state_root))
96
+
97
+
98
+ def run(
99
+ args: argparse.Namespace,
100
+ robot: ReachyMini = None,
101
+ app_stop_event: Optional[threading.Event] = None,
102
+ settings_app: Optional[FastAPI] = None,
103
+ instance_path: Optional[str] = None,
104
+ ) -> None:
105
+ """Run the Reachy Mini conversation app."""
106
+ # Putting these dependencies here makes the dashboard faster to load when the conversation app is installed
107
+ from peachy.moves import MovementManager
108
+ from peachy.config import (
109
+ set_instance_path,
110
+ resolve_app_timeout_minutes,
111
+ refresh_runtime_config_from_env,
112
+ )
113
+ from peachy.startup_settings import (
114
+ StartupSettings,
115
+ load_startup_settings_into_runtime,
116
+ )
117
+
118
+ logger = setup_logger(args.debug)
119
+ logger.info("Starting Reachy Mini Conversation App")
120
+ set_instance_path(instance_path)
121
+ startup_settings = StartupSettings()
122
+
123
+ if instance_path is not None:
124
+ try:
125
+ from dotenv import load_dotenv
126
+
127
+ env_path = Path(instance_path) / ".env"
128
+ if env_path.exists():
129
+ load_dotenv(dotenv_path=str(env_path), override=True)
130
+ refresh_runtime_config_from_env()
131
+ logger.info("Loaded instance configuration from %s", env_path)
132
+ except Exception as e:
133
+ logger.warning("Failed to load instance configuration: %s", e)
134
+
135
+ try:
136
+ startup_settings = load_startup_settings_into_runtime(instance_path)
137
+ except Exception as e:
138
+ logger.warning("Failed to load startup settings: %s", e)
139
+
140
+ from peachy.console import LocalStream
141
+ from peachy.providers import get_provider_definition
142
+ from peachy.tools.core_tools import ToolDependencies, initialize_tools
143
+
144
+ if robot is None:
145
+ try:
146
+ robot_kwargs = {}
147
+ if args.robot_name is not None:
148
+ robot_kwargs["robot_name"] = args.robot_name
149
+
150
+ logger.info("Initializing ReachyMini (SDK will auto-detect appropriate backend)")
151
+ robot = ReachyMini(**robot_kwargs)
152
+
153
+ except TimeoutError as e:
154
+ logger.error(f"Connection timeout: Failed to connect to Reachy Mini daemon. Details: {e}")
155
+ log_connection_troubleshooting(logger, args.robot_name)
156
+ sys.exit(1)
157
+
158
+ except ConnectionError as e:
159
+ logger.error(f"Connection failed: Unable to establish connection to Reachy Mini. Details: {e}")
160
+ log_connection_troubleshooting(logger, args.robot_name)
161
+ sys.exit(1)
162
+
163
+ except Exception as e:
164
+ logger.error(f"Unexpected error during robot initialization: {type(e).__name__}: {e}")
165
+ logger.error("Please check your configuration and try again.")
166
+ sys.exit(1)
167
+
168
+ app_lifecycle.wake_up_if_sleeping(robot, logger)
169
+
170
+ movement_manager = MovementManager(current_robot=robot)
171
+
172
+ deps = ToolDependencies(
173
+ reachy_mini=robot,
174
+ movement_manager=movement_manager,
175
+ instance_path=instance_path,
176
+ camera_enabled=not args.no_camera,
177
+ )
178
+
179
+ def current_provider() -> ProviderDefinition:
180
+ return get_provider_definition()
181
+
182
+ def build_handler(startup_voice: Optional[str] = None) -> ConversationHandler:
183
+ """Build the currently selected realtime handler."""
184
+ definition, handler = _build_selected_handler(deps, instance_path, startup_voice)
185
+ logger.info("Selected realtime provider: %s", definition.name)
186
+ return handler
187
+
188
+ handler = build_handler(startup_settings.voice)
189
+
190
+ stream_manager: LocalStream | None = None
191
+ own_ui_server = None
192
+
193
+ effective_settings_app = settings_app
194
+ if args.ui and settings_app is None:
195
+ effective_settings_app = FastAPI()
196
+
197
+ @effective_settings_app.middleware("http")
198
+ async def _no_cache(request: Request, call_next: Callable[[Request], Awaitable[Response]]) -> Response:
199
+ """Serve everything no-store so browsers don't keep stale UI modules."""
200
+ response = await call_next(request)
201
+ response.headers["Cache-Control"] = "no-store"
202
+ return response
203
+
204
+ stream_manager = LocalStream(
205
+ handler,
206
+ robot,
207
+ settings_app=effective_settings_app,
208
+ instance_path=instance_path,
209
+ handler_factory=build_handler,
210
+ provider_resolver=current_provider,
211
+ startup_voice=startup_settings.voice,
212
+ )
213
+
214
+ # The page is served immediately, so the API must be live before the slow startup work below.
215
+ if effective_settings_app is not None:
216
+ stream_manager._init_settings_ui_if_needed()
217
+
218
+ go_to_sleep_lock = threading.Lock()
219
+ go_to_sleep_requested = threading.Event()
220
+
221
+ def go_to_sleep_and_stop_app() -> dict[str, Any]:
222
+ """Put Reachy to sleep, then stop the current app."""
223
+ if not go_to_sleep_lock.acquire(blocking=False):
224
+ return {"status": "already_requested"}
225
+
226
+ try:
227
+ if go_to_sleep_requested.is_set():
228
+ return {"status": "already_requested"}
229
+ go_to_sleep_requested.set()
230
+
231
+ logger.info("Going to sleep before stopping conversation app.")
232
+ sleep_error: str | None = None
233
+
234
+ try:
235
+ robot.disable_wobbling()
236
+ except Exception as e:
237
+ logger.debug("Error disabling wobbling before sleep: %s", e)
238
+
239
+ movement_manager.stop(reset_to_neutral=False)
240
+
241
+ try:
242
+ robot.goto_sleep()
243
+ except Exception as e:
244
+ sleep_error = f"{type(e).__name__}: {e}"
245
+ logger.error("Failed to move Reachy Mini to sleep pose: %s", e)
246
+
247
+ stop_current_app_requested = False
248
+ if app_stop_event is None or not app_stop_event.is_set():
249
+ stop_current_app_requested = app_lifecycle.request_stop_current_app(robot, logger)
250
+ local_stop_requested = True
251
+ if app_stop_event is not None:
252
+ app_stop_event.set()
253
+ else:
254
+ try:
255
+ stream_manager.close()
256
+ except Exception as e:
257
+ local_stop_requested = False
258
+ logger.error("Error while closing stream manager after go_to_sleep: %s", e)
259
+
260
+ result: dict[str, Any] = {
261
+ "status": "sleeping" if sleep_error is None else "stop_requested",
262
+ "stop_current_app_requested": stop_current_app_requested,
263
+ "local_stop_requested": local_stop_requested,
264
+ }
265
+ if sleep_error is not None:
266
+ result["error"] = f"go_to_sleep movement failed: {sleep_error}"
267
+ return result
268
+ finally:
269
+ go_to_sleep_lock.release()
270
+
271
+ deps.go_to_sleep = go_to_sleep_and_stop_app
272
+
273
+ def run_go_to_sleep_tool() -> dict[str, Any]:
274
+ return app_lifecycle.run_go_to_sleep_tool(deps, logger)
275
+
276
+ if args.ui and settings_app is None and effective_settings_app is not None:
277
+ import uvicorn
278
+
279
+ own_ui_server = uvicorn.Server(
280
+ uvicorn.Config(effective_settings_app, host="0.0.0.0", port=7860, log_level="warning")
281
+ )
282
+ threading.Thread(target=own_ui_server.run, daemon=True, name="ui-server").start()
283
+ logger.info("Web UI available at http://localhost:7860")
284
+
285
+ try:
286
+ initialize_tools(instance_path=instance_path)
287
+ except Exception as e:
288
+ logger.error("Failed to initialize tools: %s", e)
289
+ sys.exit(1)
290
+
291
+ # Each async service → its own thread/loop
292
+ movement_manager.start()
293
+ # Audio-reactive head motion is driven by the daemon's wobbler, which
294
+ # taps the media pipeline at push_audio_sample. The console stream pushes
295
+ # assistant audio through that pipeline directly.
296
+ robot.enable_wobbling()
297
+
298
+ timeout_minutes = resolve_app_timeout_minutes()
299
+ if timeout_minutes is not None:
300
+ _start_inactivity_timeout_thread(timeout_minutes, stream_manager, logger, app_stop_event, run_go_to_sleep_tool)
301
+
302
+ def poll_stop_event() -> None:
303
+ """Poll the stop event to allow graceful shutdown."""
304
+ if app_stop_event is not None:
305
+ app_stop_event.wait()
306
+
307
+ logger.info("App stop event detected, shutting down...")
308
+ run_go_to_sleep_tool()
309
+ try:
310
+ stream_manager.close()
311
+ except Exception as e:
312
+ logger.error(f"Error while closing stream manager: {e}")
313
+
314
+ if app_stop_event:
315
+ threading.Thread(target=poll_stop_event, daemon=True).start()
316
+
317
+ try:
318
+ stream_manager.launch()
319
+ except KeyboardInterrupt:
320
+ logger.info("Keyboard interruption in main thread... closing server.")
321
+ finally:
322
+ if own_ui_server is not None:
323
+ own_ui_server.should_exit = True
324
+
325
+ sleep_result = run_go_to_sleep_tool()
326
+ if "error" in sleep_result:
327
+ movement_manager.stop(reset_to_neutral=False)
328
+ try:
329
+ robot.disable_wobbling()
330
+ except Exception as e:
331
+ logger.debug(f"Error disabling wobbling during shutdown: {e}")
332
+
333
+ # Ensure media is explicitly closed before disconnecting
334
+ try:
335
+ robot.media.close()
336
+ except Exception as e:
337
+ logger.debug(f"Error closing media during shutdown: {e}")
338
+
339
+ # prevent connection to keep alive some threads
340
+ robot.client.disconnect()
341
+ time.sleep(1)
342
+ logger.info("Shutdown complete.")
343
+
344
+
345
+ class Peachy(ReachyMiniApp): # type: ignore[misc]
346
+ """Reachy Mini Apps entry point for the conversation app."""
347
+
348
+ custom_app_url = "http://0.0.0.0:7860/"
349
+ dont_start_webserver = False
350
+
351
+ def run(self, reachy_mini: ReachyMini, stop_event: threading.Event) -> None:
352
+ """Run the Reachy Mini conversation app."""
353
+ asyncio.set_event_loop(asyncio.new_event_loop())
354
+
355
+ args, _ = parse_args()
356
+
357
+ state_root = resolve_state_root()
358
+ run(
359
+ args,
360
+ robot=reachy_mini,
361
+ app_stop_event=stop_event,
362
+ settings_app=self.settings_app,
363
+ instance_path=str(state_root),
364
+ )
365
+
366
+
367
+ if __name__ == "__main__":
368
+ app = Peachy()
369
+ try:
370
+ app.wrapped_run()
371
+ except KeyboardInterrupt:
372
+ app.stop()
src/peachy/mcp_client.py ADDED
@@ -0,0 +1,391 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Helpers for consuming remote MCP tools over HTTP(S).
2
+
3
+ This module validates remote endpoints, discovers tools, and maps calls/results
4
+ into the app's tool interface without mutating the local project environment or
5
+ downloading third-party Python code.
6
+ """
7
+ # Modified by Peachy contributors in 2026 for namespace, packaging, and provider integration.
8
+
9
+
10
+ from __future__ import annotations
11
+ import re
12
+ from typing import TYPE_CHECKING, Any, Mapping, Sequence, AsyncIterator
13
+ from datetime import timedelta
14
+ from contextlib import asynccontextmanager
15
+ from dataclasses import field, dataclass
16
+ from urllib.parse import urlparse
17
+
18
+
19
+ if TYPE_CHECKING:
20
+ from mcp import ClientSession
21
+ from mcp.types import Tool as McpTool
22
+ from mcp.types import CallToolResult as McpCallToolResult
23
+
24
+
25
+ _NAME_SEGMENT_PATTERN = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*$")
26
+ _NAME_NORMALIZER_PATTERN = re.compile(r"[^A-Za-z0-9_]+")
27
+ _LOCAL_HTTP_HOSTS = {"127.0.0.1", "localhost", "::1"}
28
+ _NAMESPACE_SEPARATOR = "__"
29
+
30
+
31
+ class McpClientError(RuntimeError):
32
+ """Base error for the MCP client."""
33
+
34
+
35
+ class McpDependencyError(McpClientError):
36
+ """Raised when a required MCP client dependency is not installed."""
37
+
38
+
39
+ class McpTransportError(McpClientError):
40
+ """Raised when discovery fails before a remote tool runs."""
41
+
42
+
43
+ class McpToolInvocationError(McpClientError):
44
+ """Raised when a remote tool call fails at the transport layer."""
45
+
46
+
47
+ class McpToolTimeoutError(McpToolInvocationError):
48
+ """Raised when a remote tool call exceeds the configured timeout."""
49
+
50
+
51
+ def _require_name_segment(label: str, value: str) -> str:
52
+ candidate = value.strip()
53
+ if _NAME_SEGMENT_PATTERN.fullmatch(candidate) is None:
54
+ raise ValueError(f"Invalid {label} '{value}'. Expected pattern '[A-Za-z_][A-Za-z0-9_]*'.")
55
+ return candidate
56
+
57
+
58
+ def apply_name_normalization(value: str) -> str:
59
+ """Replace non-identifier characters with underscores and collapse runs."""
60
+ normalized = _NAME_NORMALIZER_PATTERN.sub("_", value).strip("_")
61
+ return re.sub(r"_+", "_", normalized)
62
+
63
+
64
+ def _normalize_name_segment(label: str, value: str) -> str:
65
+ raw = value.strip()
66
+ if not raw:
67
+ raise ValueError(f"{label.capitalize()} cannot be empty.")
68
+
69
+ normalized = apply_name_normalization(raw)
70
+ if not normalized:
71
+ raise ValueError(f"{label.capitalize()} '{value}' cannot be normalized into a valid tool identifier.")
72
+ if normalized[0].isdigit():
73
+ normalized = f"tool_{normalized}"
74
+ return _require_name_segment(label, normalized)
75
+
76
+
77
+ def validate_http_mcp_url(url: str) -> str:
78
+ """Validate that the MCP endpoint uses HTTP(S)."""
79
+ parsed = urlparse(url)
80
+ if parsed.scheme not in {"http", "https"}:
81
+ raise ValueError(f"Unsupported MCP URL scheme '{parsed.scheme}'. Use http:// or https://.")
82
+ if not parsed.netloc:
83
+ raise ValueError(f"Invalid MCP URL '{url}'. Missing host.")
84
+
85
+ host = (parsed.hostname or "").lower()
86
+ if parsed.scheme == "http" and host not in _LOCAL_HTTP_HOSTS:
87
+ raise ValueError("Remote MCP servers must use HTTPS. Plain HTTP is only allowed for localhost.")
88
+ return url
89
+
90
+
91
+ def build_namespaced_tool_name(server_alias: str, tool_name: str) -> str:
92
+ """Build a local tool name for a remote MCP tool."""
93
+ alias = _require_name_segment("server alias", server_alias)
94
+ tool_segment = _normalize_name_segment("tool name", tool_name)
95
+ return f"{alias}{_NAMESPACE_SEPARATOR}{tool_segment}"
96
+
97
+
98
+ def _dump_content_block(block: object) -> dict[str, Any]:
99
+ if hasattr(block, "model_dump"):
100
+ dumped = block.model_dump(mode="json", by_alias=True, exclude_none=True)
101
+ if isinstance(dumped, dict):
102
+ return dumped
103
+ return {"type": getattr(block, "type", "unknown")}
104
+
105
+
106
+ def _join_text_content(content_blocks: list[dict[str, Any]]) -> str | None:
107
+ text_parts = [
108
+ block["text"] for block in content_blocks if block.get("type") == "text" and isinstance(block.get("text"), str)
109
+ ]
110
+ if not text_parts:
111
+ return None
112
+ return "\n\n".join(text_parts)
113
+
114
+
115
+ def _exception_contains_timeout(exc: BaseException) -> bool:
116
+ timeout_exception = _httpx_timeout_exception_type()
117
+ if isinstance(exc, timeout_exception):
118
+ return True
119
+
120
+ if "timed out" in str(exc).lower() or "deadline exceeded" in str(exc).lower():
121
+ return True
122
+
123
+ nested: list[BaseException] = []
124
+ grouped_exceptions = getattr(exc, "exceptions", None)
125
+ if isinstance(grouped_exceptions, tuple):
126
+ nested.extend(grouped_exceptions)
127
+ if exc.__cause__ is not None:
128
+ nested.append(exc.__cause__)
129
+ if exc.__context__ is not None:
130
+ nested.append(exc.__context__)
131
+
132
+ return any(_exception_contains_timeout(item) for item in nested)
133
+
134
+
135
+ def _load_mcp_sdk() -> tuple[type["ClientSession"], Any]:
136
+ try:
137
+ from mcp import ClientSession
138
+ from mcp.client.streamable_http import streamable_http_client
139
+ except ImportError as exc:
140
+ raise McpDependencyError(
141
+ "Remote MCP tools require the app's MCP client dependencies. Reinstall or update the app environment."
142
+ ) from exc
143
+ return ClientSession, streamable_http_client
144
+
145
+
146
+ def _load_httpx() -> Any:
147
+ try:
148
+ import httpx
149
+ except ImportError as exc:
150
+ raise McpDependencyError(
151
+ "Remote MCP tools require the app's HTTP client dependencies. Reinstall or update the app environment."
152
+ ) from exc
153
+ return httpx
154
+
155
+
156
+ def _httpx_timeout_exception_type() -> tuple[type[BaseException], ...]:
157
+ try:
158
+ timeout_exception = _load_httpx().TimeoutException
159
+ except McpDependencyError:
160
+ return (TimeoutError,)
161
+ return (TimeoutError, timeout_exception)
162
+
163
+
164
+ @dataclass(frozen=True)
165
+ class RemoteMcpServerConfig:
166
+ """Allowlisted MCP server configuration."""
167
+
168
+ alias: str
169
+ url: str
170
+ headers: Mapping[str, str] = field(default_factory=dict)
171
+ request_timeout_s: float = 10.0
172
+ tool_timeout_s: float = 30.0
173
+
174
+ def __post_init__(self) -> None:
175
+ """Validate configuration once the dataclass has been created."""
176
+ object.__setattr__(self, "alias", _require_name_segment("server alias", self.alias))
177
+ object.__setattr__(self, "url", validate_http_mcp_url(self.url))
178
+ object.__setattr__(self, "headers", {str(k): str(v) for k, v in self.headers.items()})
179
+ if self.request_timeout_s <= 0:
180
+ raise ValueError("request_timeout_s must be greater than zero.")
181
+ if self.tool_timeout_s <= 0:
182
+ raise ValueError("tool_timeout_s must be greater than zero.")
183
+
184
+
185
+ @dataclass(frozen=True)
186
+ class RemoteToolSpec:
187
+ """App-facing representation of a remote MCP tool."""
188
+
189
+ server_alias: str
190
+ remote_name: str
191
+ namespaced_name: str
192
+ description: str
193
+ parameters_schema: dict[str, Any]
194
+
195
+ @classmethod
196
+ def from_mcp_tool(cls, server_alias: str, tool: "McpTool") -> "RemoteToolSpec":
197
+ """Build an app-facing spec from an MCP SDK tool descriptor."""
198
+ description = (getattr(tool, "description", None) or "").strip()
199
+ parameters_schema = getattr(tool, "inputSchema", None)
200
+ if not isinstance(parameters_schema, dict):
201
+ parameters_schema = {"type": "object", "properties": {}, "required": []}
202
+
203
+ remote_name = str(getattr(tool, "name", "")).strip()
204
+ if not remote_name:
205
+ raise ValueError("Remote MCP tool is missing a name.")
206
+
207
+ return cls(
208
+ server_alias=server_alias,
209
+ remote_name=remote_name,
210
+ namespaced_name=build_namespaced_tool_name(server_alias, remote_name),
211
+ description=description or f"Remote MCP tool '{remote_name}' from server '{server_alias}'.",
212
+ parameters_schema=dict(parameters_schema),
213
+ )
214
+
215
+ def to_function_spec(self) -> dict[str, Any]:
216
+ """Translate to the app's function-calling shape."""
217
+ return {
218
+ "type": "function",
219
+ "name": self.namespaced_name,
220
+ "description": self.description,
221
+ "parameters": self.parameters_schema,
222
+ }
223
+
224
+
225
+ @dataclass(frozen=True)
226
+ class RemoteToolCallResponse:
227
+ """Mapped result for a remote MCP tool call."""
228
+
229
+ server_alias: str
230
+ remote_tool_name: str
231
+ namespaced_tool_name: str
232
+ status: str
233
+ content_blocks: list[dict[str, Any]]
234
+ text: str | None
235
+ structured_content: Any | None
236
+
237
+ @classmethod
238
+ def from_call_tool_result(
239
+ cls,
240
+ *,
241
+ server_alias: str,
242
+ remote_tool_name: str,
243
+ result: "McpCallToolResult",
244
+ ) -> "RemoteToolCallResponse":
245
+ """Convert an MCP SDK tool result into the app's result envelope."""
246
+ content_blocks = [_dump_content_block(block) for block in getattr(result, "content", [])]
247
+ return cls(
248
+ server_alias=server_alias,
249
+ remote_tool_name=remote_tool_name,
250
+ namespaced_tool_name=build_namespaced_tool_name(server_alias, remote_tool_name),
251
+ status="error" if bool(getattr(result, "isError", False)) else "ok",
252
+ content_blocks=content_blocks,
253
+ text=_join_text_content(content_blocks),
254
+ structured_content=getattr(result, "structuredContent", None),
255
+ )
256
+
257
+ def to_tool_result(self) -> dict[str, Any]:
258
+ """Return a dict shaped like the app's tool results."""
259
+ payload: dict[str, Any] = {
260
+ "status": self.status,
261
+ "server_alias": self.server_alias,
262
+ "remote_tool_name": self.remote_tool_name,
263
+ "namespaced_tool_name": self.namespaced_tool_name,
264
+ "content_blocks": self.content_blocks,
265
+ }
266
+ if self.text is not None:
267
+ payload["text"] = self.text
268
+ if self.structured_content is not None:
269
+ payload["structured_content"] = self.structured_content
270
+ return payload
271
+
272
+
273
+ class RemoteMcpToolClient:
274
+ """Minimal async client for allowlisted remote MCP tool servers."""
275
+
276
+ def __init__(self, server: RemoteMcpServerConfig, known_tools: Sequence[RemoteToolSpec] = ()) -> None:
277
+ """Store one allowlisted server configuration and an in-memory tool cache."""
278
+ self.server = server
279
+ self._tool_index = _index_remote_tools(list(known_tools))
280
+
281
+ async def list_tool_specs(self) -> list[RemoteToolSpec]:
282
+ """Discover remote tools and translate them into app-facing specs."""
283
+ try:
284
+ async with self._session() as session:
285
+ discovered = await self._list_all_tools(session)
286
+ except McpDependencyError:
287
+ raise
288
+ except Exception as exc:
289
+ raise McpTransportError(
290
+ f"Failed to discover MCP tools from '{self.server.alias}' at {self.server.url}: {exc}"
291
+ ) from exc
292
+
293
+ specs = [RemoteToolSpec.from_mcp_tool(self.server.alias, tool) for tool in discovered]
294
+ self._tool_index = _index_remote_tools(specs)
295
+ return specs
296
+
297
+ async def list_function_specs(self) -> list[dict[str, Any]]:
298
+ """Discover tools and translate them into function-calling specs."""
299
+ return [spec.to_function_spec() for spec in await self.list_tool_specs()]
300
+
301
+ async def call_tool(self, namespaced_tool_name: str, arguments: Mapping[str, Any] | None = None) -> dict[str, Any]:
302
+ """Invoke a remote MCP tool by its namespaced local ID."""
303
+ spec = await self._resolve_tool_spec(namespaced_tool_name)
304
+ timeout_exception = _httpx_timeout_exception_type()
305
+
306
+ try:
307
+ async with self._session() as session:
308
+ result = await session.call_tool(
309
+ spec.remote_name,
310
+ arguments=dict(arguments or {}),
311
+ read_timeout_seconds=timedelta(seconds=self.server.tool_timeout_s),
312
+ )
313
+ except McpDependencyError:
314
+ raise
315
+ except timeout_exception as exc:
316
+ raise McpToolTimeoutError(
317
+ f"Timed out calling MCP tool '{namespaced_tool_name}' from '{self.server.alias}'."
318
+ ) from exc
319
+ except Exception as exc:
320
+ if _exception_contains_timeout(exc):
321
+ raise McpToolTimeoutError(
322
+ f"Timed out calling MCP tool '{namespaced_tool_name}' from '{self.server.alias}'."
323
+ ) from exc
324
+ raise McpToolInvocationError(
325
+ f"Failed to call MCP tool '{namespaced_tool_name}' from '{self.server.alias}': {exc}"
326
+ ) from exc
327
+
328
+ return RemoteToolCallResponse.from_call_tool_result(
329
+ server_alias=self.server.alias,
330
+ remote_tool_name=spec.remote_name,
331
+ result=result,
332
+ ).to_tool_result()
333
+
334
+ async def _resolve_tool_spec(self, namespaced_tool_name: str) -> RemoteToolSpec:
335
+ spec = self._tool_index.get(namespaced_tool_name)
336
+ if spec is not None:
337
+ return spec
338
+
339
+ await self.list_tool_specs()
340
+ spec = self._tool_index.get(namespaced_tool_name)
341
+ if spec is None:
342
+ raise ValueError(f"Unknown remote MCP tool '{namespaced_tool_name}' for server '{self.server.alias}'.")
343
+ return spec
344
+
345
+ async def _list_all_tools(self, session: "ClientSession") -> list["McpTool"]:
346
+ tools: list[McpTool] = []
347
+ cursor: str | None = None
348
+ while True:
349
+ page = await session.list_tools(cursor=cursor)
350
+ tools.extend(page.tools)
351
+ cursor = getattr(page, "nextCursor", None)
352
+ if cursor is None:
353
+ return tools
354
+
355
+ @asynccontextmanager
356
+ async def _session(self) -> AsyncIterator["ClientSession"]:
357
+ client_session_cls, streamable_http_client = _load_mcp_sdk()
358
+ httpx = _load_httpx()
359
+ client_timeout = max(self.server.request_timeout_s, self.server.tool_timeout_s)
360
+
361
+ async with httpx.AsyncClient(
362
+ headers=self.server.headers,
363
+ follow_redirects=False,
364
+ timeout=client_timeout,
365
+ ) as http_client:
366
+ async with streamable_http_client(self.server.url, http_client=http_client) as transport:
367
+ read_stream, write_stream, _ = transport
368
+ async with client_session_cls(read_stream, write_stream) as session:
369
+ await session.initialize()
370
+ yield session
371
+
372
+
373
+ def _index_remote_tools(specs: list[RemoteToolSpec]) -> dict[str, RemoteToolSpec]:
374
+ index: dict[str, RemoteToolSpec] = {}
375
+ collisions: dict[str, list[str]] = {}
376
+
377
+ for spec in specs:
378
+ existing = index.get(spec.namespaced_name)
379
+ if existing is None:
380
+ index[spec.namespaced_name] = spec
381
+ continue
382
+
383
+ collisions.setdefault(spec.namespaced_name, [existing.remote_name]).append(spec.remote_name)
384
+
385
+ if collisions:
386
+ details = "; ".join(
387
+ f"{tool_name}: {sorted(remote_names)}" for tool_name, remote_names in sorted(collisions.items())
388
+ )
389
+ raise ValueError(f"Remote MCP tool names collide after local namespacing/normalization. Conflicts: {details}")
390
+
391
+ return index
src/peachy/memory.py ADDED
@@ -0,0 +1,213 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Modified by Peachy contributors in 2026 for namespace, packaging, and provider integration.
2
+ from __future__ import annotations
3
+ import os
4
+ import json
5
+ import time
6
+ import random
7
+ import string
8
+ import logging
9
+ import threading
10
+ from pathlib import Path
11
+ from dataclasses import dataclass
12
+ from collections.abc import Mapping
13
+
14
+
15
+ logger = logging.getLogger(__name__)
16
+
17
+ SCHEMA_VERSION = 1
18
+ MAX_FACTS = 60
19
+ MAX_FACT_CHARS = 280
20
+ MEMORY_FILENAME = "memory.v1.json"
21
+
22
+ _STORE_LOCK = threading.Lock()
23
+
24
+
25
+ @dataclass(frozen=True)
26
+ class MemoryFact:
27
+ """One short long-term memory fact."""
28
+
29
+ id: str
30
+ text: str
31
+ created_at: int
32
+
33
+ def to_json(self) -> dict[str, object]:
34
+ """Return the persisted JSON shape used by the mobile app."""
35
+ return {
36
+ "id": self.id,
37
+ "text": self.text,
38
+ "createdAt": self.created_at,
39
+ }
40
+
41
+
42
+ @dataclass(frozen=True)
43
+ class ForgetMemoryResult:
44
+ """Result of removing a memory fact."""
45
+
46
+ removed: MemoryFact | None
47
+ candidates: tuple[MemoryFact, ...]
48
+
49
+
50
+ def memory_path_for_instance(instance_path: str | Path | None = None) -> Path:
51
+ """Return the memory JSON path for this app instance."""
52
+ if instance_path is not None:
53
+ return Path(instance_path).expanduser() / MEMORY_FILENAME
54
+
55
+ data_home = os.getenv("XDG_DATA_HOME")
56
+ data_root = Path(data_home).expanduser() if data_home else Path.home() / ".local" / "share"
57
+ return data_root / "peachy" / MEMORY_FILENAME
58
+
59
+
60
+ def normalize_memory_text(text: str) -> str:
61
+ """Collapse whitespace and enforce the fact length cap."""
62
+ normalized = " ".join(text.split()).strip()
63
+ if len(normalized) <= MAX_FACT_CHARS:
64
+ return normalized
65
+ return f"{normalized[: MAX_FACT_CHARS - 3]}..."
66
+
67
+
68
+ def _make_id() -> str:
69
+ suffix = "".join(random.choices(string.ascii_lowercase + string.digits, k=6))
70
+ return f"m_{int(time.time() * 1000)}_{suffix}"
71
+
72
+
73
+ def _now_ms() -> int:
74
+ return int(time.time() * 1000)
75
+
76
+
77
+ def _fact_from_json(value: object) -> MemoryFact | None:
78
+ if not isinstance(value, Mapping):
79
+ return None
80
+
81
+ fact_id = value.get("id")
82
+ text = value.get("text")
83
+ created_at = value.get("createdAt")
84
+
85
+ if not isinstance(fact_id, str):
86
+ return None
87
+ if not isinstance(text, str):
88
+ return None
89
+ if not isinstance(created_at, (int, float)):
90
+ return None
91
+
92
+ normalized = normalize_memory_text(text)
93
+ if not normalized:
94
+ return None
95
+
96
+ return MemoryFact(id=fact_id, text=normalized, created_at=int(created_at))
97
+
98
+
99
+ def _read_memory_file(path: Path) -> list[MemoryFact]:
100
+ try:
101
+ raw = path.read_text(encoding="utf-8")
102
+ except FileNotFoundError:
103
+ return []
104
+ except OSError as exc:
105
+ logger.warning("Failed to read memory store at %s: %s", path, exc)
106
+ return []
107
+
108
+ try:
109
+ parsed = json.loads(raw)
110
+ except json.JSONDecodeError as exc:
111
+ logger.warning("Failed to parse memory store at %s: %s", path, exc)
112
+ return []
113
+
114
+ if not isinstance(parsed, Mapping):
115
+ return []
116
+
117
+ facts_value = parsed.get("facts")
118
+ if not isinstance(facts_value, list):
119
+ return []
120
+
121
+ facts: list[MemoryFact] = []
122
+ for item in facts_value:
123
+ fact = _fact_from_json(item)
124
+ if fact is not None:
125
+ facts.append(fact)
126
+ return facts[:MAX_FACTS]
127
+
128
+
129
+ def _write_memory_file(path: Path, facts: list[MemoryFact]) -> None:
130
+ path.parent.mkdir(parents=True, exist_ok=True)
131
+ payload = {
132
+ "version": SCHEMA_VERSION,
133
+ "facts": [fact.to_json() for fact in facts[:MAX_FACTS]],
134
+ }
135
+ tmp_path = path.with_name(f".{path.name}.{os.getpid()}.tmp")
136
+ try:
137
+ tmp_path.write_text(json.dumps(payload, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
138
+ tmp_path.replace(path)
139
+ finally:
140
+ try:
141
+ tmp_path.unlink(missing_ok=True)
142
+ except OSError:
143
+ pass
144
+
145
+
146
+ def list_memory_facts(instance_path: str | Path | None = None) -> list[MemoryFact]:
147
+ """Return stored memory facts, newest first."""
148
+ with _STORE_LOCK:
149
+ return list(_read_memory_file(memory_path_for_instance(instance_path)))
150
+
151
+
152
+ def add_memory_fact(instance_path: str | Path | None, text: str) -> MemoryFact | None:
153
+ """Store one short fact, deduplicating exact case-insensitive matches."""
154
+ normalized = normalize_memory_text(text)
155
+ if not normalized:
156
+ return None
157
+
158
+ path = memory_path_for_instance(instance_path)
159
+ with _STORE_LOCK:
160
+ facts = _read_memory_file(path)
161
+ existing = next((fact for fact in facts if fact.text.lower() == normalized.lower()), None)
162
+ if existing is not None:
163
+ return existing
164
+
165
+ fact = MemoryFact(id=_make_id(), text=normalized, created_at=_now_ms())
166
+ _write_memory_file(path, [fact, *facts][:MAX_FACTS])
167
+ return fact
168
+
169
+
170
+ def forget_memory_fact(
171
+ instance_path: str | Path | None,
172
+ *,
173
+ query: str | None = None,
174
+ ) -> ForgetMemoryResult:
175
+ """Remove a fact by case-insensitive substring query."""
176
+ path = memory_path_for_instance(instance_path)
177
+ with _STORE_LOCK:
178
+ facts = _read_memory_file(path)
179
+
180
+ normalized_query = normalize_memory_text(query or "").lower()
181
+ if not normalized_query:
182
+ return ForgetMemoryResult(removed=None, candidates=())
183
+
184
+ candidates = tuple(fact for fact in facts if normalized_query in fact.text.lower())
185
+ if not candidates:
186
+ return ForgetMemoryResult(removed=None, candidates=())
187
+
188
+ removed = candidates[0]
189
+ _write_memory_file(path, [fact for fact in facts if fact.id != removed.id])
190
+ return ForgetMemoryResult(removed=removed, candidates=candidates)
191
+
192
+
193
+ def clear_memory_facts(instance_path: str | Path | None = None) -> None:
194
+ """Remove all stored memory facts."""
195
+ path = memory_path_for_instance(instance_path)
196
+ with _STORE_LOCK:
197
+ _write_memory_file(path, [])
198
+
199
+
200
+ def format_memory_for_prompt(instance_path: str | Path | None = None) -> str:
201
+ """Return the prompt fragment injected before the session instructions."""
202
+ facts = list_memory_facts(instance_path)
203
+ if not facts:
204
+ return ""
205
+
206
+ bullets = "\n".join(f"- {fact.text}" for fact in facts)
207
+ return "\n".join(
208
+ [
209
+ "Things you remember about the user (use this context naturally,",
210
+ "do not recite the list verbatim):",
211
+ bullets,
212
+ ]
213
+ )
src/peachy/moves.py ADDED
@@ -0,0 +1,767 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Movement system driving sequential primary moves.
2
+
3
+ Design overview
4
+ - Primary moves (emotions, dances, goto, breathing) are mutually exclusive and run
5
+ sequentially.
6
+ - There is a single control point to the robot: `ReachyMini.set_target`.
7
+ - The control loop runs near 100 Hz and is phase-aligned via a monotonic clock.
8
+ - Idle behaviour starts an infinite `BreathingMove` after a short inactivity delay
9
+ unless listening is active.
10
+
11
+ Threading model
12
+ - A dedicated worker thread owns all real-time state and issues `set_target`
13
+ commands.
14
+ - Other threads communicate via a command queue (enqueue moves, mark activity,
15
+ toggle listening).
16
+
17
+ Units and frames
18
+ - Antennas and `body_yaw` are in radians.
19
+
20
+ Safety
21
+ - Listening freezes antennas, then blends them back on unfreeze.
22
+ - Interpolations and blends are used to avoid jumps at all times.
23
+ - `set_target` errors are rate-limited in logs.
24
+ """
25
+ # Modified by Peachy contributors in 2026 for namespace, packaging, and provider integration.
26
+
27
+
28
+ from __future__ import annotations
29
+ import time
30
+ import logging
31
+ import threading
32
+ from queue import Empty, Queue
33
+ from typing import Any, Dict, Tuple
34
+ from collections import deque
35
+ from dataclasses import dataclass
36
+
37
+ import numpy as np
38
+ from numpy.typing import NDArray
39
+
40
+ from reachy_mini import ReachyMini
41
+ from reachy_mini.utils import create_head_pose
42
+ from reachy_mini.motion.move import Move
43
+ from reachy_mini.utils.interpolation import compose_world_offset, linear_pose_interpolation
44
+ from peachy.dance_emotion_moves import EmotionQueueMove
45
+
46
+
47
+ logger = logging.getLogger(__name__)
48
+
49
+ # Configuration constants
50
+ CONTROL_LOOP_FREQUENCY_HZ = 60.0 # Hz - Target frequency for the movement control loop
51
+
52
+ # Type definitions
53
+ FullBodyPose = Tuple[NDArray[np.float32], Tuple[float, float], float] # (head_pose_4x4, antennas, body_yaw)
54
+
55
+
56
+ class BreathingMove(Move): # type: ignore
57
+ """Breathing move with interpolation to neutral and then continuous breathing patterns."""
58
+
59
+ def __init__(
60
+ self,
61
+ interpolation_start_pose: NDArray[np.float32],
62
+ interpolation_start_antennas: Tuple[float, float],
63
+ interpolation_duration: float = 1.0,
64
+ ):
65
+ """Initialize breathing move.
66
+
67
+ Args:
68
+ interpolation_start_pose: 4x4 matrix of current head pose to interpolate from
69
+ interpolation_start_antennas: Current antenna positions to interpolate from
70
+ interpolation_duration: Duration of interpolation to neutral (seconds)
71
+
72
+ """
73
+ self.interpolation_start_pose = interpolation_start_pose
74
+ self.interpolation_start_antennas = np.array(interpolation_start_antennas)
75
+ self.interpolation_duration = interpolation_duration
76
+
77
+ # Neutral positions for breathing base
78
+ self.neutral_head_pose = create_head_pose(0, 0, 0, 0, 0, 0, degrees=True)
79
+ self.neutral_antennas = np.array([-0.1745, 0.1745]) # ~10° offset to reduce shaking
80
+
81
+ # Breathing parameters
82
+ self.breathing_z_amplitude = 0.005 # 5mm gentle breathing
83
+ self.breathing_frequency = 0.1 # Hz (6 breaths per minute)
84
+ self.antenna_sway_amplitude = np.deg2rad(15) # 15 degrees
85
+ self.antenna_frequency = 0.5 # Hz (faster antenna sway)
86
+
87
+ @property
88
+ def duration(self) -> float:
89
+ """Duration property required by official Move interface."""
90
+ return float("inf") # Continuous breathing (never ends naturally)
91
+
92
+ def evaluate(self, t: float) -> tuple[NDArray[np.float64] | None, NDArray[np.float64] | None, float | None]:
93
+ """Evaluate breathing move at time t."""
94
+ if t < self.interpolation_duration:
95
+ # Phase 1: Interpolate to neutral base position
96
+ interpolation_t = t / self.interpolation_duration
97
+
98
+ # Interpolate head pose
99
+ head_pose = linear_pose_interpolation(
100
+ self.interpolation_start_pose,
101
+ self.neutral_head_pose,
102
+ interpolation_t,
103
+ )
104
+
105
+ # Interpolate antennas
106
+ antennas_interp = (
107
+ 1 - interpolation_t
108
+ ) * self.interpolation_start_antennas + interpolation_t * self.neutral_antennas
109
+ antennas = antennas_interp.astype(np.float64)
110
+
111
+ else:
112
+ # Phase 2: Breathing patterns from neutral base
113
+ breathing_time = t - self.interpolation_duration
114
+
115
+ # Gentle z-axis breathing
116
+ z_offset = self.breathing_z_amplitude * np.sin(2 * np.pi * self.breathing_frequency * breathing_time)
117
+ head_pose = create_head_pose(x=0, y=0, z=z_offset, roll=0, pitch=0, yaw=0, degrees=True, mm=False)
118
+
119
+ # Antenna sway (opposite directions)
120
+ antenna_sway = self.antenna_sway_amplitude * np.sin(2 * np.pi * self.antenna_frequency * breathing_time)
121
+ antennas = np.array([antenna_sway, -antenna_sway], dtype=np.float64)
122
+
123
+ # Return in official Move interface format: (head_pose, antennas_array, body_yaw)
124
+ return (head_pose, antennas, 0.0)
125
+
126
+
127
+ def clone_full_body_pose(pose: FullBodyPose) -> FullBodyPose:
128
+ """Create a deep copy of a full body pose tuple."""
129
+ head, antennas, body_yaw = pose
130
+ return (head.copy(), (float(antennas[0]), float(antennas[1])), float(body_yaw))
131
+
132
+
133
+ @dataclass
134
+ class MovementState:
135
+ """State tracking for the movement system."""
136
+
137
+ # Primary move state
138
+ current_move: Move | None = None
139
+ move_start_time: float | None = None
140
+ last_activity_time: float = 0.0
141
+
142
+ # Status flags
143
+ last_primary_pose: FullBodyPose | None = None
144
+
145
+ def update_activity(self) -> None:
146
+ """Update the last activity time."""
147
+ self.last_activity_time = time.monotonic()
148
+
149
+
150
+ @dataclass
151
+ class LoopFrequencyStats:
152
+ """Track rolling loop frequency statistics."""
153
+
154
+ mean: float = 0.0
155
+ m2: float = 0.0
156
+ min_freq: float = float("inf")
157
+ count: int = 0
158
+ last_freq: float = 0.0
159
+ potential_freq: float = 0.0
160
+
161
+ def reset(self) -> None:
162
+ """Reset accumulators while keeping the last potential frequency."""
163
+ self.mean = 0.0
164
+ self.m2 = 0.0
165
+ self.min_freq = float("inf")
166
+ self.count = 0
167
+
168
+
169
+ class MovementManager:
170
+ """Coordinate sequential moves and robot output at 100 Hz.
171
+
172
+ Responsibilities:
173
+ - Own a real-time loop that samples the current primary move (if any) and calls
174
+ `set_target` exactly once per tick.
175
+ - Start an idle `BreathingMove` after `idle_inactivity_delay` when not
176
+ listening and no moves are queued.
177
+ - Expose thread-safe APIs so other threads can enqueue moves or mark activity
178
+ without touching internal state.
179
+
180
+ Timing:
181
+ - All elapsed-time calculations rely on `time.monotonic()` through `self._now`
182
+ to avoid wall-clock jumps.
183
+ - The loop attempts 100 Hz
184
+
185
+ Concurrency:
186
+ - External threads communicate via `_command_queue` messages.
187
+ """
188
+
189
+ def __init__(
190
+ self,
191
+ current_robot: ReachyMini,
192
+ ):
193
+ """Initialize movement manager."""
194
+ self.current_robot = current_robot
195
+ self._head_tracking = False
196
+
197
+ # Single timing source for durations
198
+ self._now = time.monotonic
199
+
200
+ # Movement state
201
+ self.state = MovementState()
202
+ self.state.last_activity_time = self._now()
203
+ neutral_pose = create_head_pose(0, 0, 0, 0, 0, 0, degrees=True)
204
+ self.state.last_primary_pose = (neutral_pose, (0.0, 0.0), 0.0)
205
+
206
+ # Move queue (primary moves)
207
+ self.move_queue: deque[Move] = deque()
208
+
209
+ # Configuration
210
+ self.idle_inactivity_delay = 0.3 # seconds
211
+ self.target_frequency = CONTROL_LOOP_FREQUENCY_HZ
212
+ self.target_period = 1.0 / self.target_frequency
213
+
214
+ self._stop_event = threading.Event()
215
+ self._thread: threading.Thread | None = None
216
+ self._is_listening = False
217
+ # Speaking pauses tracking; the captured look-at pose anchors queued moves.
218
+ self._is_speaking = False
219
+ self._track_anchor: NDArray[np.float64] | None = None
220
+ self._last_commanded_pose: FullBodyPose = clone_full_body_pose(self.state.last_primary_pose)
221
+ self._listening_antennas: Tuple[float, float] = self._last_commanded_pose[1]
222
+ self._antenna_unfreeze_blend = 1.0
223
+ self._antenna_blend_duration = 0.4 # seconds to blend back after listening
224
+ self._last_listening_blend_time = self._now()
225
+ self._breathing_active = False # true when breathing move is running or queued
226
+ self._listening_debounce_s = 0.15
227
+ self._last_listening_toggle_time = self._now()
228
+ self._last_set_target_err = 0.0
229
+ self._set_target_err_interval = 1.0 # seconds between error logs
230
+ self._set_target_err_suppressed = 0
231
+
232
+ # Cross-thread signalling
233
+ self._command_queue: "Queue[Tuple[str, Any]]" = Queue()
234
+
235
+ self._shared_state_lock = threading.Lock()
236
+ self._shared_last_activity_time = self.state.last_activity_time
237
+ self._shared_is_listening = self._is_listening
238
+ self._status_lock = threading.Lock()
239
+ self._freq_stats = LoopFrequencyStats()
240
+ self._freq_snapshot = LoopFrequencyStats()
241
+
242
+ def queue_move(self, move: Move) -> None:
243
+ """Queue a primary move to run after the currently executing one.
244
+
245
+ Thread-safe: the move is enqueued via the worker command queue so the
246
+ control loop remains the sole mutator of movement state.
247
+ """
248
+ self._command_queue.put(("queue_move", move))
249
+
250
+ def clear_move_queue(self) -> None:
251
+ """Stop the active move and discard any queued primary moves.
252
+
253
+ Thread-safe: executed by the worker thread via the command queue.
254
+ """
255
+ self._command_queue.put(("clear_queue", None))
256
+
257
+ def set_moving_state(self, duration: float) -> None:
258
+ """Mark the robot as actively moving for the provided duration.
259
+
260
+ Legacy hook used by goto helpers to keep inactivity and breathing logic
261
+ aware of manual motions. Thread-safe via the command queue.
262
+ """
263
+ self._command_queue.put(("set_moving_state", duration))
264
+
265
+ def is_idle(self) -> bool:
266
+ """Return True when the robot has been inactive longer than the idle delay."""
267
+ with self._shared_state_lock:
268
+ last_activity = self._shared_last_activity_time
269
+ listening = self._shared_is_listening
270
+
271
+ if listening:
272
+ return False
273
+
274
+ return self._now() - last_activity >= self.idle_inactivity_delay
275
+
276
+ def set_listening(self, listening: bool) -> None:
277
+ """Enable or disable listening mode without touching shared state directly.
278
+
279
+ While listening:
280
+ - Antenna positions are frozen at the last commanded values.
281
+ - Blending is reset so that upon unfreezing the antennas return smoothly.
282
+ - Idle breathing is suppressed.
283
+
284
+ Thread-safe: the change is posted to the worker command queue.
285
+ """
286
+ with self._shared_state_lock:
287
+ if self._shared_is_listening == listening:
288
+ return
289
+ self._command_queue.put(("set_listening", listening))
290
+
291
+ def set_head_tracking(self, enabled: bool) -> None:
292
+ """Start or stop following the user's face; thread-safe via the command queue."""
293
+ self._command_queue.put(("set_head_tracking", enabled))
294
+
295
+ def set_speaking(self, speaking: bool) -> None:
296
+ """Pause head tracking while the assistant speaks, resume it afterwards.
297
+
298
+ On speaking, the current look-at pose is captured as an anchor so queued
299
+ moves stay on the user; when speaking stops the head is handed back to
300
+ daemon tracking. Thread-safe: posted to the worker command queue.
301
+ """
302
+ self._command_queue.put(("set_speaking", speaking))
303
+
304
+ def _poll_signals(self, current_time: float) -> None:
305
+ """Apply queued commands."""
306
+ while True:
307
+ try:
308
+ command, payload = self._command_queue.get_nowait()
309
+ except Empty:
310
+ break
311
+ self._handle_command(command, payload, current_time)
312
+
313
+ def _handle_command(self, command: str, payload: Any, current_time: float) -> None:
314
+ """Handle a single cross-thread command."""
315
+ if command == "queue_move":
316
+ if isinstance(payload, Move):
317
+ self.move_queue.append(payload)
318
+ self.state.update_activity()
319
+ duration = getattr(payload, "duration", None)
320
+ if duration is not None:
321
+ try:
322
+ duration_str = f"{float(duration):.2f}"
323
+ except (TypeError, ValueError):
324
+ duration_str = str(duration)
325
+ else:
326
+ duration_str = "?"
327
+ logger.debug(
328
+ "Queued move with duration %ss, queue size: %s",
329
+ duration_str,
330
+ len(self.move_queue),
331
+ )
332
+ else:
333
+ logger.warning("Ignored queue_move command with invalid payload: %s", payload)
334
+ elif command == "clear_queue":
335
+ self.move_queue.clear()
336
+ self.state.current_move = None
337
+ self.state.move_start_time = None
338
+ self._breathing_active = False
339
+ logger.info("Cleared move queue and stopped current move")
340
+ elif command == "set_moving_state":
341
+ try:
342
+ duration = float(payload)
343
+ except (TypeError, ValueError):
344
+ logger.warning("Invalid moving state duration: %s", payload)
345
+ return
346
+ self.state.update_activity()
347
+ elif command == "mark_activity":
348
+ self.state.update_activity()
349
+ elif command == "set_listening":
350
+ desired_state = bool(payload)
351
+ now = self._now()
352
+ if now - self._last_listening_toggle_time < self._listening_debounce_s:
353
+ return
354
+ self._last_listening_toggle_time = now
355
+
356
+ if self._is_listening == desired_state:
357
+ return
358
+
359
+ self._is_listening = desired_state
360
+ self._last_listening_blend_time = now
361
+ if desired_state:
362
+ # Freeze: snapshot current commanded antennas and reset blend
363
+ self._listening_antennas = (
364
+ float(self._last_commanded_pose[1][0]),
365
+ float(self._last_commanded_pose[1][1]),
366
+ )
367
+ self._antenna_unfreeze_blend = 0.0
368
+ else:
369
+ # Unfreeze: restart blending from frozen pose
370
+ self._antenna_unfreeze_blend = 0.0
371
+ self.state.update_activity()
372
+ elif command == "set_head_tracking":
373
+ enabled = bool(payload)
374
+ if self._head_tracking == enabled:
375
+ return
376
+ self._head_tracking = enabled
377
+ self._track_anchor = None
378
+ # set_speaking is gated off while disabled, so its state would go stale across a toggle.
379
+ self._is_speaking = False
380
+ try:
381
+ if enabled:
382
+ self.current_robot.start_head_tracking(weight=1.0)
383
+ else:
384
+ self.current_robot.stop_head_tracking()
385
+ except Exception as e:
386
+ logger.warning("Head-tracking toggle failed: %s", e)
387
+ elif command == "set_speaking":
388
+ if not self._head_tracking:
389
+ return
390
+ speaking = bool(payload)
391
+ if self._is_speaking == speaking:
392
+ return
393
+ self._is_speaking = speaking
394
+ try:
395
+ if speaking and self.current_robot.get_tracked_face(wait=False).detected:
396
+ # Pause only once a face is locked, else speech blocks acquisition.
397
+ self._track_anchor = self.current_robot.get_current_head_pose()
398
+ self.current_robot.start_head_tracking(weight=0.0)
399
+ elif not speaking:
400
+ self._track_anchor = None
401
+ self.current_robot.start_head_tracking(weight=1.0)
402
+ except Exception as e:
403
+ logger.warning("Head-tracking speaking handoff failed: %s", e)
404
+ else:
405
+ logger.warning("Unknown command received by MovementManager: %s", command)
406
+
407
+ def _publish_shared_state(self) -> None:
408
+ """Expose idle-related state for external threads."""
409
+ with self._shared_state_lock:
410
+ self._shared_last_activity_time = self.state.last_activity_time
411
+ self._shared_is_listening = self._is_listening
412
+
413
+ def _manage_move_queue(self, current_time: float) -> None:
414
+ """Manage the primary move queue (sequential execution)."""
415
+ if self.state.current_move is None or (
416
+ self.state.move_start_time is not None
417
+ and current_time - self.state.move_start_time >= self.state.current_move.duration
418
+ ):
419
+ self.state.current_move = None
420
+ self.state.move_start_time = None
421
+
422
+ if self.move_queue:
423
+ self.state.current_move = self.move_queue.popleft()
424
+ self.state.move_start_time = current_time
425
+ # Any real move cancels breathing mode flag
426
+ self._breathing_active = isinstance(self.state.current_move, BreathingMove)
427
+ logger.debug(f"Starting new move, duration: {self.state.current_move.duration}s")
428
+
429
+ def _manage_breathing(self, current_time: float) -> None:
430
+ """Manage automatic breathing when idle."""
431
+ if (
432
+ self.state.current_move is None
433
+ and not self.move_queue
434
+ and not self._is_listening
435
+ and not self._breathing_active
436
+ ):
437
+ idle_for = current_time - self.state.last_activity_time
438
+ if idle_for >= self.idle_inactivity_delay:
439
+ try:
440
+ # These 2 functions return the latest available sensor data from the robot, but don't perform I/O synchronously.
441
+ # Therefore, we accept calling them inside the control loop.
442
+ _, current_antennas = self.current_robot.get_current_joint_positions()
443
+ current_head_pose = self.current_robot.get_current_head_pose()
444
+
445
+ self._breathing_active = True
446
+ self.state.update_activity()
447
+
448
+ breathing_move = BreathingMove(
449
+ interpolation_start_pose=current_head_pose,
450
+ interpolation_start_antennas=current_antennas,
451
+ interpolation_duration=1.0,
452
+ )
453
+ self.move_queue.append(breathing_move)
454
+ logger.debug("Started breathing after %.1fs of inactivity", idle_for)
455
+ except Exception as e:
456
+ self._breathing_active = False
457
+ logger.error("Failed to start breathing: %s", e)
458
+
459
+ if isinstance(self.state.current_move, BreathingMove) and self.move_queue:
460
+ self.state.current_move = None
461
+ self.state.move_start_time = None
462
+ self._breathing_active = False
463
+ logger.debug("Stopping breathing due to new move activity")
464
+
465
+ if self.state.current_move is not None and not isinstance(self.state.current_move, BreathingMove):
466
+ self._breathing_active = False
467
+
468
+ def _get_primary_pose(self, current_time: float) -> FullBodyPose:
469
+ """Get the primary full body pose from current move or neutral."""
470
+ # When a primary move is playing, sample it and cache the resulting pose
471
+ if self.state.current_move is not None and self.state.move_start_time is not None:
472
+ move_time = current_time - self.state.move_start_time
473
+ head, antennas, body_yaw = self.state.current_move.evaluate(move_time)
474
+
475
+ if head is None:
476
+ head = create_head_pose(0, 0, 0, 0, 0, 0, degrees=True)
477
+ if antennas is None:
478
+ antennas = np.array([-0.1745, 0.1745]) # ~10° offset
479
+ if body_yaw is None:
480
+ body_yaw = 0.0
481
+
482
+ antennas_tuple = (float(antennas[0]), float(antennas[1]))
483
+ head_copy = head.copy()
484
+ primary_full_body_pose = (
485
+ head_copy,
486
+ antennas_tuple,
487
+ float(body_yaw),
488
+ )
489
+
490
+ self.state.last_primary_pose = clone_full_body_pose(primary_full_body_pose)
491
+ # Otherwise reuse the last primary pose so we avoid jumps between moves
492
+ elif self.state.last_primary_pose is not None:
493
+ primary_full_body_pose = clone_full_body_pose(self.state.last_primary_pose)
494
+ else:
495
+ neutral_head_pose = create_head_pose(0, 0, 0, 0, 0, 0, degrees=True)
496
+ primary_full_body_pose = (neutral_head_pose, (0.0, 0.0), 0.0)
497
+ self.state.last_primary_pose = clone_full_body_pose(primary_full_body_pose)
498
+
499
+ # Speaking pauses tracking: hold the look-at anchor, overlay emotions on it, dance from neutral.
500
+ if self._track_anchor is not None:
501
+ head_pose, antennas, body_yaw = primary_full_body_pose
502
+ move = self.state.current_move
503
+ if move is None:
504
+ head_pose = self._track_anchor.copy()
505
+ elif isinstance(move, EmotionQueueMove):
506
+ head_pose = compose_world_offset(self._track_anchor, head_pose)
507
+ primary_full_body_pose = (head_pose, antennas, body_yaw)
508
+
509
+ return primary_full_body_pose
510
+
511
+ def _update_primary_motion(self, current_time: float) -> None:
512
+ """Advance queue state and idle behaviours for this tick."""
513
+ self._manage_move_queue(current_time)
514
+ self._manage_breathing(current_time)
515
+
516
+ def _calculate_blended_antennas(self, target_antennas: Tuple[float, float]) -> Tuple[float, float]:
517
+ """Blend target antennas with listening freeze state and update blending."""
518
+ now = self._now()
519
+ listening = self._is_listening
520
+ listening_antennas = self._listening_antennas
521
+ blend = self._antenna_unfreeze_blend
522
+ blend_duration = self._antenna_blend_duration
523
+ last_update = self._last_listening_blend_time
524
+ self._last_listening_blend_time = now
525
+
526
+ if listening:
527
+ antennas_cmd = listening_antennas
528
+ new_blend = 0.0
529
+ else:
530
+ dt = max(0.0, now - last_update)
531
+ if blend_duration <= 0:
532
+ new_blend = 1.0
533
+ else:
534
+ new_blend = min(1.0, blend + dt / blend_duration)
535
+ antennas_cmd = (
536
+ listening_antennas[0] * (1.0 - new_blend) + target_antennas[0] * new_blend,
537
+ listening_antennas[1] * (1.0 - new_blend) + target_antennas[1] * new_blend,
538
+ )
539
+
540
+ if listening:
541
+ self._antenna_unfreeze_blend = 0.0
542
+ else:
543
+ self._antenna_unfreeze_blend = new_blend
544
+ if new_blend >= 1.0:
545
+ self._listening_antennas = (
546
+ float(target_antennas[0]),
547
+ float(target_antennas[1]),
548
+ )
549
+
550
+ return antennas_cmd
551
+
552
+ def _issue_control_command(
553
+ self, head: NDArray[np.float32], antennas: Tuple[float, float], body_yaw: float
554
+ ) -> None:
555
+ """Send the pose to the robot with throttled error logging."""
556
+ try:
557
+ self.current_robot.set_target(head=head, antennas=antennas, body_yaw=body_yaw)
558
+ except Exception as e:
559
+ now = self._now()
560
+ if now - self._last_set_target_err >= self._set_target_err_interval:
561
+ msg = f"Failed to set robot target: {e}"
562
+ if self._set_target_err_suppressed:
563
+ msg += f" (suppressed {self._set_target_err_suppressed} repeats)"
564
+ self._set_target_err_suppressed = 0
565
+ logger.error(msg)
566
+ self._last_set_target_err = now
567
+ else:
568
+ self._set_target_err_suppressed += 1
569
+ else:
570
+ with self._status_lock:
571
+ self._last_commanded_pose = clone_full_body_pose((head, antennas, body_yaw))
572
+
573
+ def _update_frequency_stats(
574
+ self,
575
+ loop_start: float,
576
+ prev_loop_start: float,
577
+ stats: LoopFrequencyStats,
578
+ ) -> LoopFrequencyStats:
579
+ """Update frequency statistics based on the current loop start time."""
580
+ period = loop_start - prev_loop_start
581
+ if period > 0:
582
+ stats.last_freq = 1.0 / period
583
+ stats.count += 1
584
+ delta = stats.last_freq - stats.mean
585
+ stats.mean += delta / stats.count
586
+ stats.m2 += delta * (stats.last_freq - stats.mean)
587
+ stats.min_freq = min(stats.min_freq, stats.last_freq)
588
+ return stats
589
+
590
+ def _schedule_next_tick(self, loop_start: float, stats: LoopFrequencyStats) -> Tuple[float, LoopFrequencyStats]:
591
+ """Compute sleep time to maintain target frequency and update potential freq."""
592
+ computation_time = self._now() - loop_start
593
+ stats.potential_freq = 1.0 / computation_time if computation_time > 0 else float("inf")
594
+ sleep_time = max(0.0, self.target_period - computation_time)
595
+ return sleep_time, stats
596
+
597
+ def _record_frequency_snapshot(self, stats: LoopFrequencyStats) -> None:
598
+ """Store a thread-safe snapshot of current frequency statistics."""
599
+ with self._status_lock:
600
+ self._freq_snapshot = LoopFrequencyStats(
601
+ mean=stats.mean,
602
+ m2=stats.m2,
603
+ min_freq=stats.min_freq,
604
+ count=stats.count,
605
+ last_freq=stats.last_freq,
606
+ potential_freq=stats.potential_freq,
607
+ )
608
+
609
+ def _maybe_log_frequency(self, loop_count: int, print_interval_loops: int, stats: LoopFrequencyStats) -> None:
610
+ """Emit frequency telemetry when enough loops have elapsed."""
611
+ if loop_count % print_interval_loops != 0 or stats.count == 0:
612
+ return
613
+
614
+ variance = stats.m2 / stats.count if stats.count > 0 else 0.0
615
+ lowest = stats.min_freq if stats.min_freq != float("inf") else 0.0
616
+ logger.debug(
617
+ "Loop freq - avg: %.2fHz, variance: %.4f, min: %.2fHz, last: %.2fHz, potential: %.2fHz, target: %.1fHz",
618
+ stats.mean,
619
+ variance,
620
+ lowest,
621
+ stats.last_freq,
622
+ stats.potential_freq,
623
+ self.target_frequency,
624
+ )
625
+ stats.reset()
626
+
627
+ def start(self) -> None:
628
+ """Start the worker thread that drives the 100 Hz control loop."""
629
+ if self._thread is not None and self._thread.is_alive():
630
+ logger.warning("Move worker already running; start() ignored")
631
+ return
632
+ self._stop_event.clear()
633
+ self._thread = threading.Thread(target=self.working_loop, daemon=True)
634
+ self._thread.start()
635
+ logger.debug("Move worker started")
636
+
637
+ def stop(self, reset_to_neutral: bool = True) -> None:
638
+ """Request the worker thread to stop and wait for it to exit.
639
+
640
+ Optionally resets the robot to a neutral position after stopping.
641
+ """
642
+ if self._thread is None or not self._thread.is_alive():
643
+ logger.debug("Move worker not running; stop() ignored")
644
+ return
645
+
646
+ if reset_to_neutral:
647
+ logger.info("Stopping movement manager and resetting to neutral position...")
648
+ else:
649
+ logger.info("Stopping movement manager...")
650
+
651
+ # Clear any queued moves and stop current move
652
+ self.clear_move_queue()
653
+
654
+ # Stop the worker thread first so it doesn't interfere
655
+ self._stop_event.set()
656
+ if self._thread is not None:
657
+ self._thread.join()
658
+ self._thread = None
659
+ logger.debug("Move worker stopped")
660
+
661
+ if self._head_tracking:
662
+ try:
663
+ self.current_robot.stop_head_tracking()
664
+ except Exception as e:
665
+ logger.warning("Failed to stop head tracking: %s", e)
666
+
667
+ if not reset_to_neutral:
668
+ return
669
+
670
+ # Reset to neutral position using goto_target (same approach as wake_up)
671
+ try:
672
+ neutral_head_pose = create_head_pose(0, 0, 0, 0, 0, 0, degrees=True)
673
+ neutral_antennas = [-0.1745, 0.1745] # ~10° offset to reduce shaking
674
+ neutral_body_yaw = 0.0
675
+
676
+ # Use goto_target directly on the robot
677
+ self.current_robot.goto_target(
678
+ head=neutral_head_pose,
679
+ antennas=neutral_antennas,
680
+ duration=2.0,
681
+ body_yaw=neutral_body_yaw,
682
+ )
683
+
684
+ logger.info("Reset to neutral position completed")
685
+
686
+ except Exception as e:
687
+ logger.error(f"Failed to reset to neutral position: {e}")
688
+
689
+ def get_status(self) -> Dict[str, Any]:
690
+ """Return a lightweight status snapshot for observability."""
691
+ with self._status_lock:
692
+ pose_snapshot = clone_full_body_pose(self._last_commanded_pose)
693
+ freq_snapshot = LoopFrequencyStats(
694
+ mean=self._freq_snapshot.mean,
695
+ m2=self._freq_snapshot.m2,
696
+ min_freq=self._freq_snapshot.min_freq,
697
+ count=self._freq_snapshot.count,
698
+ last_freq=self._freq_snapshot.last_freq,
699
+ potential_freq=self._freq_snapshot.potential_freq,
700
+ )
701
+
702
+ head_matrix = pose_snapshot[0].tolist() if pose_snapshot else None
703
+ antennas = pose_snapshot[1] if pose_snapshot else None
704
+ body_yaw = pose_snapshot[2] if pose_snapshot else None
705
+
706
+ return {
707
+ "queue_size": len(self.move_queue),
708
+ "is_listening": self._is_listening,
709
+ "breathing_active": self._breathing_active,
710
+ "last_commanded_pose": {
711
+ "head": head_matrix,
712
+ "antennas": antennas,
713
+ "body_yaw": body_yaw,
714
+ },
715
+ "loop_frequency": {
716
+ "last": freq_snapshot.last_freq,
717
+ "mean": freq_snapshot.mean,
718
+ "min": freq_snapshot.min_freq,
719
+ "potential": freq_snapshot.potential_freq,
720
+ "samples": freq_snapshot.count,
721
+ },
722
+ }
723
+
724
+ def working_loop(self) -> None:
725
+ """Run the primary-move control loop with a single set_target() call per tick."""
726
+ logger.debug("Starting enhanced movement control loop (100Hz)")
727
+
728
+ loop_count = 0
729
+ prev_loop_start = self._now()
730
+ print_interval_loops = max(1, int(self.target_frequency * 2))
731
+ freq_stats = self._freq_stats
732
+
733
+ while not self._stop_event.is_set():
734
+ loop_start = self._now()
735
+ loop_count += 1
736
+
737
+ if loop_count > 1:
738
+ freq_stats = self._update_frequency_stats(loop_start, prev_loop_start, freq_stats)
739
+ prev_loop_start = loop_start
740
+
741
+ # 1) Poll external commands
742
+ self._poll_signals(loop_start)
743
+
744
+ # 2) Manage the primary move queue (start new move, end finished move, breathing)
745
+ self._update_primary_motion(loop_start)
746
+
747
+ # 3) Build the primary full-body pose for this tick
748
+ head, antennas, body_yaw = self._get_primary_pose(loop_start)
749
+
750
+ # 4) Apply listening antenna freeze or blend-back
751
+ antennas_cmd = self._calculate_blended_antennas(antennas)
752
+
753
+ # 5) Single set_target call - the only control point
754
+ self._issue_control_command(head, antennas_cmd, body_yaw)
755
+
756
+ # 6) Adaptive sleep to align to next tick, then publish shared state
757
+ sleep_time, freq_stats = self._schedule_next_tick(loop_start, freq_stats)
758
+ self._publish_shared_state()
759
+ self._record_frequency_snapshot(freq_stats)
760
+
761
+ # 7) Periodic telemetry on loop frequency
762
+ self._maybe_log_frequency(loop_count, print_interval_loops, freq_stats)
763
+
764
+ if sleep_time > 0:
765
+ time.sleep(sleep_time)
766
+
767
+ logger.debug("Movement control loop stopped")
src/peachy/openai_realtime.py ADDED
@@ -0,0 +1,1178 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Modified by Peachy contributors in 2026 for namespace, packaging, and provider integration.
2
+ import json
3
+ import os
4
+ import random
5
+ import time
6
+ import uuid
7
+ import base64
8
+ import asyncio
9
+ import logging
10
+ from collections import deque
11
+ from typing import Any, Final, Literal, cast
12
+
13
+ import numpy as np
14
+ from openai import APIConnectionError, APITimeoutError, AsyncOpenAI
15
+ from pydantic import Field, BaseModel
16
+ from openai.types.realtime import (
17
+ AudioTranscriptionParam,
18
+ RealtimeAudioConfigParam,
19
+ RealtimeToolsConfigParam,
20
+ RealtimeFunctionToolParam,
21
+ RealtimeAudioConfigInputParam,
22
+ RealtimeAudioConfigOutputParam,
23
+ RealtimeSessionCreateRequestParam,
24
+ )
25
+ from websockets.exceptions import ConnectionClosedError
26
+ from openai.resources.realtime.realtime import AsyncRealtimeConnection
27
+ from openai.types.realtime.realtime_audio_input_turn_detection_param import ServerVad
28
+
29
+ from peachy.tools import core_tools
30
+ from peachy.config import config
31
+ from peachy.prompts import (
32
+ get_session_voice,
33
+ get_session_instructions,
34
+ get_session_greeting_prompt,
35
+ )
36
+ from peachy.audio.resampling import pcm16_at_rate
37
+ from peachy.streaming import AdditionalOutputs
38
+ from peachy.tools.core_tools import (
39
+ ToolSpec,
40
+ ToolDependencies,
41
+ get_tool_specs,
42
+ )
43
+ from peachy.conversation_handler import AudioFrame, ConversationHandler
44
+ from peachy.providers import redact_secrets
45
+ from peachy.tools.background_tool_manager import (
46
+ ToolCallRoutine,
47
+ ToolNotification,
48
+ BackgroundToolManager,
49
+ )
50
+
51
+
52
+ logger = logging.getLogger(__name__)
53
+
54
+ _RESPONSE_DONE_TIMEOUT: Final[float] = 30.0
55
+ _RESPONSE_REJECTION_RETRY_DELAY: Final[float] = 0.5
56
+ OPENAI_SAMPLE_RATE: Final[int] = 24_000
57
+ OPENAI_DEFAULT_MODEL: Final[str] = "gpt-realtime"
58
+ OPENAI_DEFAULT_VOICE: Final[str] = "marin"
59
+ OPENAI_VOICES: Final[tuple[str, ...]] = (
60
+ "alloy",
61
+ "ash",
62
+ "ballad",
63
+ "cedar",
64
+ "coral",
65
+ "echo",
66
+ "marin",
67
+ "sage",
68
+ "shimmer",
69
+ "verse",
70
+ )
71
+
72
+ ErrorDisposition = Literal["terminal", "transient"]
73
+
74
+
75
+ class TerminalProviderError(RuntimeError):
76
+ """A provider failure that needs configuration or operator action."""
77
+
78
+
79
+ class RetryExhaustedError(RuntimeError):
80
+ """Transient provider failures exhausted this activation's retry budget."""
81
+
82
+
83
+ class RealtimeProtocolError(RuntimeError):
84
+ """A server-delivered Realtime error event that must end this session."""
85
+
86
+ def __init__(
87
+ self,
88
+ message: str,
89
+ *,
90
+ code: str = "",
91
+ status_code: int | None = None,
92
+ event_id: str | None = None,
93
+ ) -> None:
94
+ super().__init__(message)
95
+ self.code = code
96
+ self.status_code = status_code
97
+ self.event_id = event_id
98
+
99
+
100
+ def _protocol_error_from_event(event: object) -> RealtimeProtocolError:
101
+ payload = getattr(event, "error", None)
102
+
103
+ def field(name: str) -> object:
104
+ if isinstance(payload, dict):
105
+ return payload.get(name)
106
+ return getattr(payload, name, None)
107
+
108
+ code = str(field("code") or field("type") or "").lower()
109
+ message = str(field("message") or "OpenAI Realtime protocol error")
110
+ raw_status = field("status_code")
111
+ status = raw_status if isinstance(raw_status, int) else None
112
+ raw_event_id = field("event_id")
113
+ event_id = raw_event_id if isinstance(raw_event_id, str) else None
114
+ return RealtimeProtocolError(
115
+ message,
116
+ code=code,
117
+ status_code=status,
118
+ event_id=event_id,
119
+ )
120
+
121
+
122
+ def _error_code(error: BaseException) -> str:
123
+ direct = getattr(error, "code", None)
124
+ if isinstance(direct, str):
125
+ return direct.lower()
126
+ body = getattr(error, "body", None)
127
+ if isinstance(body, dict):
128
+ code = body.get("code")
129
+ nested = body.get("error")
130
+ if not code and isinstance(nested, dict):
131
+ code = nested.get("code")
132
+ return str(code or "").lower()
133
+ return ""
134
+
135
+
136
+ def classify_openai_error(error: BaseException) -> ErrorDisposition:
137
+ """Classify an OpenAI activation failure without permitting retry storms."""
138
+ code = _error_code(error)
139
+ status = getattr(error, "status_code", None)
140
+ if code in {"insufficient_quota", "invalid_api_key", "model_not_found"}:
141
+ return "terminal"
142
+ if code in {"rate_limit_exceeded", "server_error"}:
143
+ return "transient"
144
+ if status in {400, 401, 403, 404, 422}:
145
+ return "terminal"
146
+ if status == 429:
147
+ return "transient"
148
+ if isinstance(
149
+ error,
150
+ (
151
+ APIConnectionError,
152
+ APITimeoutError,
153
+ ConnectionError,
154
+ TimeoutError,
155
+ ConnectionClosedError,
156
+ ),
157
+ ):
158
+ return "transient"
159
+ if isinstance(status, int) and status >= 500:
160
+ return "transient"
161
+ return "terminal"
162
+
163
+
164
+ class InputTranscriptChunksByItem(BaseModel):
165
+ """Current item_id and its accumulated deltas. Only one item at a time."""
166
+
167
+ item_id: str | None = None
168
+ deltas: list[str] = Field(default_factory=list)
169
+
170
+
171
+ def to_realtime_tools_config(tool_specs: list[ToolSpec]) -> RealtimeToolsConfigParam:
172
+ """Convert app tool specs to the OpenAI-compatible realtime session shape."""
173
+ realtime_tools: RealtimeToolsConfigParam = []
174
+ for spec in tool_specs:
175
+ realtime_tools.append(
176
+ RealtimeFunctionToolParam(
177
+ type="function",
178
+ name=spec["name"],
179
+ description=spec["description"],
180
+ parameters=spec["parameters"],
181
+ )
182
+ )
183
+ return realtime_tools
184
+
185
+
186
+ def _pcm24k() -> dict[str, object]:
187
+ return {"type": "audio/pcm", "rate": OPENAI_SAMPLE_RATE}
188
+
189
+
190
+ class OpenAIRealtimeHandler(ConversationHandler):
191
+ """Realtime stream handler for the direct OpenAI Realtime API."""
192
+
193
+ SAMPLE_RATE = OPENAI_SAMPLE_RATE
194
+
195
+ def __init__(
196
+ self,
197
+ deps: ToolDependencies,
198
+ instance_path: str | None = None,
199
+ startup_voice: str | None = None,
200
+ ) -> None:
201
+ """Initialize the handler."""
202
+ super().__init__()
203
+
204
+ self.deps = deps
205
+
206
+ self.client: AsyncOpenAI
207
+ self.connection: AsyncRealtimeConnection | None = None
208
+ self.output_queue: asyncio.Queue[AudioFrame | AdditionalOutputs] = asyncio.Queue()
209
+
210
+ self.instance_path = instance_path
211
+ self._voice_override: str | None = self._normalize_startup_voice(startup_voice)
212
+ self.model = (os.getenv("OPENAI_REALTIME_MODEL") or OPENAI_DEFAULT_MODEL).strip() or OPENAI_DEFAULT_MODEL
213
+
214
+ # Debouncing for partial transcripts
215
+ self.partial_transcript_task: asyncio.Task[None] | None = None
216
+ self.partial_debounce_delay = 0.5 # seconds
217
+ self.input_transcript_chunks_by_item = InputTranscriptChunksByItem()
218
+
219
+ # Internal lifecycle flags
220
+ self._connected_event: asyncio.Event = asyncio.Event()
221
+
222
+ # Background tool manager
223
+ self.tool_manager = BackgroundToolManager()
224
+
225
+ # Response-in-progress guard: the Realtime API only allows one active
226
+ # response per conversation at a time. A dedicated worker task
227
+ # (_response_sender_loop) dequeues and sends one request at a time
228
+ self._pending_responses: asyncio.Queue[dict[str, Any]] = asyncio.Queue()
229
+ self._response_done_event: asyncio.Event = asyncio.Event()
230
+ self._response_done_event.set()
231
+ self._response_started_or_rejected_event: asyncio.Event = asyncio.Event()
232
+ self._last_response_rejected: bool = False
233
+ self._turn_user_done_at: float | None = None
234
+ self._turn_response_created_at: float | None = None
235
+ self._turn_first_audio_at: float | None = None
236
+ self._startup_greeting_sent = False
237
+ self._in_flight_tool_calls: set[str] = set()
238
+ self._tool_batch_needs_response = False
239
+ self._response_generation = 0
240
+ self._active_response_id: str | None = None
241
+ self._active_response_generation: int | None = None
242
+ self._cancelled_response_ids: set[str] = set()
243
+ self._cancelled_generations: set[int] = set()
244
+ self._pending_cancel_event_ids: deque[str] = deque(maxlen=32)
245
+ self._restart_tasks: set[asyncio.Task[None]] = set()
246
+ self._restart_lock = asyncio.Lock()
247
+ self._shutdown_started = False
248
+
249
+ @staticmethod
250
+ def _sanitize_tool_result_for_model(tool_name: str, tool_result: dict[str, Any]) -> dict[str, Any]:
251
+ """Remove bulky transport-only fields before echoing tool output back to the model."""
252
+ if tool_name == "camera" and "b64_im" in tool_result:
253
+ sanitized = dict(tool_result)
254
+ sanitized.pop("b64_im", None)
255
+ sanitized["image_attached"] = True
256
+ return sanitized
257
+ return tool_result
258
+
259
+ @staticmethod
260
+ def _event_response_id(event: object) -> str | None:
261
+ direct = getattr(event, "response_id", None)
262
+ if isinstance(direct, str) and direct:
263
+ return direct
264
+ response = getattr(event, "response", None)
265
+ nested = getattr(response, "id", None)
266
+ return nested if isinstance(nested, str) and nested else None
267
+
268
+ def _event_belongs_to_cancelled_response(self, event: object) -> bool:
269
+ response_id = self._event_response_id(event)
270
+ if response_id is not None and response_id in self._cancelled_response_ids:
271
+ return True
272
+ generation = self._active_response_generation
273
+ if generation is None or generation not in self._cancelled_generations:
274
+ return False
275
+ # response.created.response.id is optional in SDK 2.28, while audio deltas
276
+ # carry response_id. Bind that first observed ID to the already-cancelled
277
+ # active generation and suppress it until response.done.
278
+ if response_id is not None:
279
+ self._cancelled_response_ids.add(response_id)
280
+ if self._active_response_id is None:
281
+ self._active_response_id = response_id
282
+ return True
283
+
284
+ def _forget_cancel_event_id(self, event_id: str) -> None:
285
+ try:
286
+ self._pending_cancel_event_ids.remove(event_id)
287
+ except ValueError:
288
+ pass
289
+
290
+ def _finish_response(self, event: object) -> None:
291
+ response_id = self._event_response_id(event)
292
+ if response_id is not None:
293
+ self._cancelled_response_ids.discard(response_id)
294
+ if self._active_response_generation is not None:
295
+ self._cancelled_generations.discard(self._active_response_generation)
296
+ self._active_response_id = None
297
+ self._active_response_generation = None
298
+
299
+ def _restart_task_done(self, task: asyncio.Task[None]) -> None:
300
+ """Retrieve and sanitize background restart failures."""
301
+ self._restart_tasks.discard(task)
302
+ if task.cancelled():
303
+ return
304
+ try:
305
+ task.result()
306
+ except Exception as exc:
307
+ logger.warning("Realtime session restart failed: %s", redact_secrets(exc))
308
+
309
+ async def _cancel_restart_tasks(self) -> None:
310
+ """Cancel and retrieve retained restart tasks during handler shutdown."""
311
+ restart_tasks = tuple(self._restart_tasks)
312
+ for task in restart_tasks:
313
+ task.cancel()
314
+ if restart_tasks:
315
+ await asyncio.gather(*restart_tasks, return_exceptions=True)
316
+ self._restart_tasks.clear()
317
+
318
+ async def _cancel_active_response_for_barge_in(self) -> None:
319
+ response_id = self._active_response_id
320
+ generation = self._active_response_generation
321
+ if response_id is not None:
322
+ self._cancelled_response_ids.add(response_id)
323
+ if generation is not None:
324
+ self._cancelled_generations.add(generation)
325
+
326
+ if self.connection is not None and (response_id is not None or generation is not None):
327
+ cancel_event_id = f"peachy-cancel-{uuid.uuid4().hex}"
328
+ self._pending_cancel_event_ids.append(cancel_event_id)
329
+ try:
330
+ if response_id is None:
331
+ await self.connection.response.cancel(event_id=cancel_event_id)
332
+ else:
333
+ await self.connection.response.cancel(
334
+ event_id=cancel_event_id,
335
+ response_id=response_id,
336
+ )
337
+ except Exception as exc:
338
+ self._forget_cancel_event_id(cancel_event_id)
339
+ logger.debug("Response cancellation raced with completion (%s)", type(exc).__name__)
340
+
341
+ if self._clear_queue is not None:
342
+ self._clear_queue()
343
+ else:
344
+ while not self.output_queue.empty():
345
+ try:
346
+ self.output_queue.get_nowait()
347
+ except asyncio.QueueEmpty:
348
+ break
349
+
350
+ def _normalize_startup_voice(self, voice: str | None) -> str | None:
351
+ """Return a valid persisted startup voice, or None."""
352
+ return self._resolve_backend_voice(voice, source="persisted startup voice")
353
+
354
+ async def _wait_for_response_done_before_tool_result(self) -> bool:
355
+ """Return whether the function-call response finished before sending tool output."""
356
+ if self._response_done_event.is_set():
357
+ return True
358
+
359
+ try:
360
+ await asyncio.wait_for(
361
+ self._response_done_event.wait(),
362
+ timeout=_RESPONSE_DONE_TIMEOUT,
363
+ )
364
+ return True
365
+ except asyncio.TimeoutError:
366
+ return False
367
+
368
+ def _resolve_backend_voice(
369
+ self,
370
+ voice: str | None,
371
+ *,
372
+ source: str,
373
+ fallback: str | None = None,
374
+ ) -> str | None:
375
+ """Return a backend-supported voice, optionally falling back when unsupported."""
376
+ available_voices = OPENAI_VOICES
377
+ voice_value = (voice or "").strip()
378
+ if not voice_value:
379
+ return fallback
380
+
381
+ voice_by_lowercase = {candidate.lower(): candidate for candidate in available_voices}
382
+ normalized_voice = voice_by_lowercase.get(voice_value.lower())
383
+ if normalized_voice is not None:
384
+ return normalized_voice
385
+
386
+ if voice:
387
+ logger.warning(
388
+ "Ignoring unsupported %s %r; expected one of %s",
389
+ source,
390
+ voice,
391
+ available_voices,
392
+ )
393
+ return fallback
394
+
395
+ def _get_session_config(self, tool_specs: list[ToolSpec]) -> RealtimeSessionCreateRequestParam:
396
+ """Return the direct OpenAI realtime session config."""
397
+ return RealtimeSessionCreateRequestParam(
398
+ type="realtime",
399
+ instructions=get_session_instructions(self.instance_path),
400
+ audio=RealtimeAudioConfigParam(
401
+ input=RealtimeAudioConfigInputParam(
402
+ format=_pcm24k(), # type: ignore[typeddict-item]
403
+ transcription=AudioTranscriptionParam(
404
+ model="gpt-4o-transcribe",
405
+ language=config.REALTIME_TRANSCRIPTION_LANGUAGE,
406
+ ),
407
+ turn_detection=ServerVad(type="server_vad", interrupt_response=True),
408
+ ),
409
+ output=RealtimeAudioConfigOutputParam(
410
+ format=_pcm24k(), # type: ignore[typeddict-item]
411
+ voice=self.get_current_voice(),
412
+ ),
413
+ ),
414
+ tools=to_realtime_tools_config([spec for spec in tool_specs if spec["name"] != "camera"]),
415
+ tool_choice="auto",
416
+ )
417
+
418
+ def _is_connected(self) -> bool:
419
+ """Return whether the realtime connection is open."""
420
+ return self.connection is not None
421
+
422
+ def _idle_behavior_ready(self) -> bool:
423
+ """Hold idle behavior while a model response is still active."""
424
+ return self._response_done_event.is_set()
425
+
426
+ async def _cancel_partial_transcript_task(self) -> None:
427
+ if self.partial_transcript_task and not self.partial_transcript_task.done():
428
+ self.partial_transcript_task.cancel()
429
+ try:
430
+ await self.partial_transcript_task
431
+ except asyncio.CancelledError:
432
+ pass
433
+
434
+ async def change_voice(self, voice: str) -> str:
435
+ """Change only the voice, updating the active session when possible."""
436
+ resolved_voice = (
437
+ self._resolve_backend_voice(voice, source="requested voice", fallback=OPENAI_DEFAULT_VOICE)
438
+ or OPENAI_DEFAULT_VOICE
439
+ )
440
+ self._voice_override = resolved_voice
441
+ if self.connection is not None:
442
+ try:
443
+ await self.connection.session.update(
444
+ session=RealtimeSessionCreateRequestParam(
445
+ type="realtime",
446
+ audio=RealtimeAudioConfigParam(
447
+ output=RealtimeAudioConfigOutputParam(
448
+ voice=resolved_voice,
449
+ ),
450
+ ),
451
+ ),
452
+ )
453
+ return f"Voice changed to {resolved_voice}."
454
+ except Exception as e:
455
+ logger.warning("Failed to update live session for voice change: %s", redact_secrets(e))
456
+ return "Voice change failed. Will take effect on next connection."
457
+ return "Voice changed. Will take effect on next connection."
458
+
459
+ def get_current_voice(self) -> str:
460
+ """Return the voice currently selected for this handler."""
461
+ voice = self._voice_override or get_session_voice(default=OPENAI_DEFAULT_VOICE)
462
+ return (
463
+ self._resolve_backend_voice(voice, source="session voice", fallback=OPENAI_DEFAULT_VOICE)
464
+ or OPENAI_DEFAULT_VOICE
465
+ )
466
+
467
+ async def apply_personality(self, profile: str | None) -> str:
468
+ """Apply a new personality (profile) at runtime if possible.
469
+
470
+ - Updates the global config's selected profile for subsequent calls.
471
+ - If a realtime connection is active, sends a session.update with the
472
+ freshly resolved instructions so the change takes effect immediately.
473
+
474
+ Returns a short status message for UI feedback.
475
+ """
476
+ try:
477
+ # Update the in-process config value and env
478
+ from peachy.config import config as _config
479
+ from peachy.config import set_custom_profile
480
+
481
+ set_custom_profile(profile)
482
+ logger.info(
483
+ "Set custom profile to %r (config=%r)", profile, getattr(_config, "REACHY_MINI_CUSTOM_PROFILE", None)
484
+ )
485
+
486
+ try:
487
+ instructions = get_session_instructions(self.instance_path)
488
+ voice = self.get_current_voice()
489
+ except Exception as e:
490
+ safe_error = redact_secrets(e)
491
+ logger.error("Failed to resolve personality content: %s", safe_error)
492
+ return f"Failed to apply personality: {safe_error}"
493
+
494
+ # Rebuild the tool registry
495
+ core_tools.initialize_tools(force=True)
496
+
497
+ # Attempt a live update first, then force a full restart to ensure it sticks
498
+ if self.connection is not None:
499
+ try:
500
+ await self.connection.session.update(
501
+ session=RealtimeSessionCreateRequestParam(
502
+ type="realtime",
503
+ instructions=instructions,
504
+ audio=RealtimeAudioConfigParam(
505
+ output=RealtimeAudioConfigOutputParam(
506
+ voice=voice,
507
+ ),
508
+ ),
509
+ ),
510
+ )
511
+ logger.info("Applied personality via live update: %s", profile or "built-in default")
512
+ except Exception as e:
513
+ logger.warning("Live update failed; will restart session: %s", redact_secrets(e))
514
+
515
+ # Force a real restart to guarantee the new instructions/voice
516
+ try:
517
+ await self._restart_session()
518
+ return "Applied personality and restarted realtime session."
519
+ except Exception as e:
520
+ logger.warning("Failed to restart session after apply: %s", redact_secrets(e))
521
+ return "Applied personality. Will take effect on next connection."
522
+ else:
523
+ logger.info(
524
+ "Applied personality recorded: %s (no live connection; will apply on next session)",
525
+ profile or "built-in default",
526
+ )
527
+ return "Applied personality. Will take effect on next connection."
528
+ except Exception as e:
529
+ safe_error = redact_secrets(e)
530
+ logger.error("Error applying personality '%s': %s", profile, safe_error)
531
+ return f"Failed to apply personality: {safe_error}"
532
+
533
+ async def _emit_debounced_partial(self, transcript: str, item_id: str, sequence_counter: int) -> None:
534
+ """Emit partial transcript after debounce delay."""
535
+ try:
536
+ await asyncio.sleep(self.partial_debounce_delay)
537
+
538
+ input_transcript = self.input_transcript_chunks_by_item
539
+ if input_transcript.item_id == item_id and len(input_transcript.deltas) - 1 == sequence_counter:
540
+ await self.output_queue.put(AdditionalOutputs({"role": "user_partial", "content": transcript}))
541
+ logger.debug(f"Debounced partial emitted: {transcript}")
542
+ except asyncio.CancelledError:
543
+ logger.debug("Debounced partial cancelled")
544
+ raise
545
+
546
+ def _record_partial_transcript_delta(
547
+ self,
548
+ input_transcript: InputTranscriptChunksByItem,
549
+ item_id: str,
550
+ delta: str,
551
+ ) -> None:
552
+ """Record an OpenAI partial transcript snapshot."""
553
+ input_transcript.item_id = item_id
554
+ input_transcript.deltas = [delta]
555
+
556
+ async def start_up(self) -> None:
557
+ """Run one bounded OpenAI activation."""
558
+ max_attempts = 3
559
+ for attempt in range(1, max_attempts + 1):
560
+ try:
561
+ self.client = await self._build_realtime_client()
562
+ await self._run_realtime_session()
563
+ return
564
+ except asyncio.CancelledError:
565
+ raise
566
+ except Exception as exc:
567
+ message = redact_secrets(exc)
568
+ if classify_openai_error(exc) == "terminal":
569
+ logger.warning("OpenAI realtime activation stopped: %s", message)
570
+ raise TerminalProviderError(message) from exc
571
+ if attempt == max_attempts:
572
+ logger.warning(
573
+ "OpenAI realtime activation exhausted %d attempts: %s",
574
+ max_attempts,
575
+ message,
576
+ )
577
+ raise RetryExhaustedError(
578
+ f"OpenAI realtime failed after {max_attempts} attempts: {message}"
579
+ ) from exc
580
+ delay = float(2 ** (attempt - 1)) + random.uniform(0.0, 0.5)
581
+ logger.warning(
582
+ "OpenAI realtime transient failure (%d/%d): %s; retrying in %.1fs",
583
+ attempt,
584
+ max_attempts,
585
+ message,
586
+ delay,
587
+ )
588
+ await asyncio.sleep(delay)
589
+ finally:
590
+ self.connection = None
591
+ self._connected_event.clear()
592
+
593
+ async def _restart_session(self) -> None:
594
+ """Force-close the current session and start a fresh one in background.
595
+
596
+ Does not block the caller while the new session is establishing.
597
+ """
598
+ try:
599
+ async with self._restart_lock:
600
+ if self._shutdown_started:
601
+ return
602
+
603
+ if self.connection is not None:
604
+ try:
605
+ await self.connection.close()
606
+ except Exception:
607
+ pass
608
+ finally:
609
+ self.connection = None
610
+
611
+ # Ensure we have a client (start_up must have run once)
612
+ if getattr(self, "client", None) is None:
613
+ logger.warning("Cannot restart: realtime client not initialized yet.")
614
+ return
615
+
616
+ # Build before registering the session task, then re-check the
617
+ # shutdown gate because client construction may have suspended.
618
+ try:
619
+ self._connected_event.clear()
620
+ except Exception:
621
+ pass
622
+ client = await self._build_realtime_client()
623
+ if self._shutdown_started:
624
+ return
625
+ self.client = client
626
+ restart_task = asyncio.create_task(
627
+ self._run_realtime_session(),
628
+ name="realtime-session-restart",
629
+ )
630
+ self._restart_tasks.add(restart_task)
631
+ restart_task.add_done_callback(self._restart_task_done)
632
+ try:
633
+ await asyncio.wait_for(self._connected_event.wait(), timeout=5.0)
634
+ if not self._shutdown_started:
635
+ logger.info("Realtime session restarted and connected.")
636
+ except asyncio.TimeoutError:
637
+ if not self._shutdown_started:
638
+ logger.warning("Realtime session restart timed out; continuing in background.")
639
+ except asyncio.CancelledError:
640
+ raise
641
+ except Exception as e:
642
+ logger.warning("_restart_session failed: %s", redact_secrets(e))
643
+
644
+ async def _safe_response_create(self, **kwargs: Any) -> None:
645
+ """Enqueue a response.create() kwargs for the sender worker _response_sender_loop().
646
+
647
+ This method never blocks the caller.
648
+ """
649
+ await self._pending_responses.put(kwargs)
650
+
651
+ async def _send_startup_greeting_prompt(self) -> None:
652
+ """Prompt the model to open the conversation once the session is ready."""
653
+ if self._startup_greeting_sent or not self.connection:
654
+ return
655
+
656
+ greeting_prompt = get_session_greeting_prompt().strip()
657
+ if not greeting_prompt:
658
+ self._startup_greeting_sent = True
659
+ return
660
+
661
+ try:
662
+ await self.connection.conversation.item.create(
663
+ item={
664
+ "type": "message",
665
+ "role": "user",
666
+ "content": [
667
+ {
668
+ "type": "input_text",
669
+ "text": greeting_prompt,
670
+ },
671
+ ],
672
+ },
673
+ )
674
+ self._startup_greeting_sent = True
675
+ self._mark_activity("startup_greeting_prompt")
676
+ await self._safe_response_create()
677
+ logger.info("Queued startup greeting prompt")
678
+ except Exception as e:
679
+ logger.warning("Failed to queue startup greeting prompt: %s", redact_secrets(e))
680
+
681
+ async def _response_sender_loop(self) -> None:
682
+ """Dedicated worker that sends ``response.create()`` calls serially.
683
+
684
+ This logic was designed to comply with the response.create() docstring specification for event ordering:
685
+ https://github.com/openai/openai-python/blob/3e0c05b84a2056870abf3bd6a5e7849020209cc3/src/openai/resources/realtime/realtime.py#L649C1-L651C30
686
+
687
+ For each queued request the worker:
688
+ 1. Waits until no response is active (_response_done_event).
689
+ 2. Sends response.create().
690
+ 3. Waits until the receiver observes response.created or a rejection.
691
+ 4. Waits for the response cycle to complete (response.done).
692
+ 5. If the server rejected with active_response, retries from step 1.
693
+ """
694
+ while self.connection:
695
+ try:
696
+ kwargs = await self._pending_responses.get()
697
+ except asyncio.CancelledError:
698
+ return
699
+
700
+ # Parallel tool calls enqueue duplicate empty requests; coalesce to one.
701
+ while not kwargs and not self._pending_responses.empty():
702
+ try:
703
+ self._pending_responses.get_nowait()
704
+ except asyncio.QueueEmpty:
705
+ break
706
+
707
+ sent = False
708
+ max_retries = 5
709
+ attempts = 0
710
+ while not sent and self.connection and attempts < max_retries:
711
+ try:
712
+ await asyncio.wait_for(
713
+ self._response_done_event.wait(),
714
+ timeout=_RESPONSE_DONE_TIMEOUT,
715
+ )
716
+ except asyncio.TimeoutError:
717
+ logger.debug("Timed out waiting for previous response to finish; forcing ahead")
718
+ self._response_done_event.set()
719
+
720
+ if not self.connection:
721
+ break
722
+
723
+ self._last_response_rejected = False
724
+ self._response_started_or_rejected_event.clear()
725
+ try:
726
+ await self.connection.response.create(**kwargs)
727
+ except Exception as e:
728
+ logger.debug("_response_sender_loop: send failed: %s", redact_secrets(e))
729
+ self._response_done_event.set()
730
+ break
731
+
732
+ try:
733
+ await asyncio.wait_for(
734
+ self._response_started_or_rejected_event.wait(),
735
+ timeout=_RESPONSE_DONE_TIMEOUT,
736
+ )
737
+ except asyncio.TimeoutError:
738
+ logger.debug("Timed out waiting for response.created or response rejection")
739
+
740
+ # Check if the receiver loop observed an asynchronous rejection.
741
+ if self._last_response_rejected:
742
+ attempts += 1
743
+ if attempts >= max_retries:
744
+ logger.debug("response.create rejected %d times; giving up", attempts)
745
+ break
746
+ logger.debug("response.create was rejected; retrying (%d/%d)", attempts, max_retries)
747
+ await asyncio.sleep(_RESPONSE_REJECTION_RETRY_DELAY)
748
+ continue
749
+
750
+ try:
751
+ await asyncio.wait_for(
752
+ self._response_done_event.wait(),
753
+ timeout=_RESPONSE_DONE_TIMEOUT,
754
+ )
755
+ except asyncio.TimeoutError:
756
+ logger.debug("Timed out waiting for response.done; assuming response completed")
757
+ self._response_done_event.set()
758
+ break
759
+
760
+ sent = True
761
+
762
+ async def _handle_tool_result(self, completed_tool: ToolNotification) -> None:
763
+ """Process the result of a tool call."""
764
+ if completed_tool.error is not None:
765
+ logger.error(
766
+ "Tool '%s' (id=%s) failed with error: %s",
767
+ completed_tool.tool_name,
768
+ completed_tool.id,
769
+ completed_tool.error,
770
+ )
771
+ tool_result = {"error": completed_tool.error}
772
+ tool_result_for_model = tool_result
773
+ elif completed_tool.result is not None:
774
+ tool_result = completed_tool.result
775
+ tool_result_for_model = (
776
+ self._sanitize_tool_result_for_model(completed_tool.tool_name, tool_result)
777
+ if isinstance(tool_result, dict)
778
+ else tool_result
779
+ )
780
+ logger.info(
781
+ "Tool '%s' (id=%s) executed successfully.",
782
+ completed_tool.tool_name,
783
+ completed_tool.id,
784
+ )
785
+ logger.debug("Tool '%s' model-visible result: %s", completed_tool.tool_name, tool_result_for_model)
786
+ else:
787
+ logger.warning(
788
+ "Tool '%s' (id=%s) returned no result and no error", completed_tool.tool_name, completed_tool.id
789
+ )
790
+ tool_result = {"error": "No result returned from tool execution"}
791
+ tool_result_for_model = tool_result
792
+
793
+ # Connection may have closed while tool was running
794
+ if not self.connection:
795
+ logger.warning(
796
+ "Connection closed during tool '%s' (id=%s) execution; cannot send result back",
797
+ completed_tool.tool_name,
798
+ completed_tool.id,
799
+ )
800
+ return
801
+
802
+ try:
803
+ send_result_to_model = not completed_tool.is_idle_tool_call
804
+ if send_result_to_model:
805
+ self._mark_activity("tool_result_ready")
806
+ model_result_submitted = False
807
+ if send_result_to_model and isinstance(completed_tool.id, str):
808
+ if not await self._wait_for_response_done_before_tool_result():
809
+ send_result_to_model = False
810
+ if not send_result_to_model:
811
+ logger.warning(
812
+ "Dropping realtime model result for tool '%s' (id=%s) because response.done was not observed",
813
+ completed_tool.tool_name,
814
+ completed_tool.id,
815
+ )
816
+ elif not self.connection:
817
+ logger.warning(
818
+ "Connection closed before sending tool '%s' (id=%s) result back",
819
+ completed_tool.tool_name,
820
+ completed_tool.id,
821
+ )
822
+ return
823
+ else:
824
+ await self.connection.conversation.item.create(
825
+ item={
826
+ "type": "function_call_output",
827
+ "call_id": completed_tool.id,
828
+ "output": json.dumps(tool_result_for_model),
829
+ },
830
+ )
831
+ model_result_submitted = True
832
+
833
+ await self.output_queue.put(
834
+ AdditionalOutputs(
835
+ {
836
+ "role": "assistant",
837
+ "content": json.dumps(tool_result_for_model),
838
+ },
839
+ ),
840
+ )
841
+
842
+ if model_result_submitted and completed_tool.tool_name == "camera" and "b64_im" in tool_result:
843
+ # use raw base64, don't json.dumps (which adds quotes)
844
+ b64_im = tool_result["b64_im"]
845
+ if not isinstance(b64_im, str):
846
+ logger.warning("Unexpected type for b64_im: %s", type(b64_im))
847
+ b64_im = str(b64_im)
848
+ image_width = tool_result.get("image_width")
849
+ image_height = tool_result.get("image_height")
850
+ jpeg_bytes_value = tool_result.get("jpeg_bytes")
851
+ jpeg_bytes = jpeg_bytes_value if isinstance(jpeg_bytes_value, int) else (len(b64_im) * 3) // 4
852
+ await self.connection.conversation.item.create(
853
+ item={
854
+ "type": "message",
855
+ "role": "user",
856
+ "content": [
857
+ {
858
+ "type": "input_image",
859
+ "image_url": f"data:image/jpeg;base64,{b64_im}",
860
+ },
861
+ ],
862
+ },
863
+ )
864
+ if isinstance(image_width, int) and isinstance(image_height, int):
865
+ logger.info(
866
+ "Added camera image to conversation frame=%sx%s jpeg_bytes=%s",
867
+ image_width,
868
+ image_height,
869
+ jpeg_bytes,
870
+ )
871
+ else:
872
+ logger.info(
873
+ "Added camera image to conversation jpeg_bytes=%s",
874
+ jpeg_bytes,
875
+ )
876
+
877
+ if isinstance(completed_tool.id, str):
878
+ self._in_flight_tool_calls.discard(completed_tool.id)
879
+
880
+ tool = core_tools.ALL_TOOLS.get(completed_tool.tool_name)
881
+ # Always surface errors, skip the spoken follow-up for tools that opt out.
882
+ if model_result_submitted and (completed_tool.error is not None or tool is None or tool.needs_response):
883
+ self._tool_batch_needs_response = True
884
+
885
+ # Parallel tool calls in one turn: respond once every result is in, not per tool.
886
+ if self._tool_batch_needs_response and not self._in_flight_tool_calls:
887
+ self._tool_batch_needs_response = False
888
+ await self._safe_response_create()
889
+
890
+ except ConnectionClosedError:
891
+ logger.warning("Connection closed while sending tool result")
892
+ self.connection = None
893
+ self._response_done_event.set()
894
+
895
+ async def _run_realtime_session(self) -> None:
896
+ """Establish and manage a single direct OpenAI realtime session."""
897
+ tool_specs = [spec for spec in get_tool_specs() if spec["name"] != "camera"]
898
+ async with self.client.realtime.connect(model=self.model) as conn:
899
+ try:
900
+ await conn.session.update(session=self._get_session_config(tool_specs))
901
+ except Exception as exc:
902
+ logger.error("OpenAI realtime session.update failed: %s", redact_secrets(exc))
903
+ raise
904
+ self.connection = conn
905
+ self._connected_event.set()
906
+ response_sender_task: asyncio.Task[None] | None = None
907
+ try:
908
+ self.tool_manager.start_up(tool_callbacks=[self._handle_tool_result])
909
+ response_sender_task = asyncio.create_task(self._response_sender_loop(), name="response-sender")
910
+ await self._send_startup_greeting_prompt()
911
+ async for event in conn:
912
+ logger.debug("Realtime event: %s", event.type)
913
+ await self._handle_event(event)
914
+ finally:
915
+ if response_sender_task is not None:
916
+ response_sender_task.cancel()
917
+ try:
918
+ await response_sender_task
919
+ except asyncio.CancelledError:
920
+ pass
921
+ await self.tool_manager.shutdown()
922
+ self.connection = None
923
+ self._connected_event.clear()
924
+
925
+ async def _handle_event(self, event: object) -> None:
926
+ """Handle one OpenAI Realtime event."""
927
+ event_data = cast(Any, event)
928
+ if event_data.type == "input_audio_buffer.speech_started":
929
+ self._mark_activity("user_speech_started")
930
+ self._turn_user_done_at = None
931
+ self._turn_response_created_at = None
932
+ self._turn_first_audio_at = None
933
+ await self._cancel_active_response_for_barge_in()
934
+ self.deps.movement_manager.set_listening(True)
935
+ logger.debug("User speech started")
936
+
937
+ if event_data.type == "input_audio_buffer.speech_stopped":
938
+ self._mark_activity("user_speech_stopped")
939
+ self.deps.movement_manager.set_listening(False)
940
+ logger.debug("User speech stopped - server will auto-commit with VAD")
941
+
942
+ if event_data.type == "response.output_audio.done":
943
+ self.deps.movement_manager.set_speaking(False)
944
+ logger.debug("response completed")
945
+
946
+ if event_data.type == "response.output_text.delta":
947
+ logger.debug("response text delta")
948
+
949
+ if event_data.type == "response.output_text.done":
950
+ logger.debug("response text done: %s", event_data.text)
951
+
952
+ if event_data.type == "response.created":
953
+ self._mark_activity("response_created")
954
+ self._response_generation += 1
955
+ self._active_response_generation = self._response_generation
956
+ self._active_response_id = self._event_response_id(event)
957
+ self.deps.movement_manager.set_speaking(True)
958
+ self._response_done_event.clear()
959
+ self._response_started_or_rejected_event.set()
960
+ if self._turn_user_done_at is not None and self._turn_response_created_at is None:
961
+ self._turn_response_created_at = time.perf_counter()
962
+ delta_ms = (self._turn_response_created_at - self._turn_user_done_at) * 1000
963
+ logger.info("Turn latency: response.created %.0f ms after user transcript", delta_ms)
964
+ logger.debug("Response created (active)")
965
+
966
+ if event_data.type == "response.done":
967
+ # Doesn't mean the audio is done playing
968
+ # Resume tracking for responses that emit no audio (text-only / tool-only).
969
+ self.deps.movement_manager.set_speaking(False)
970
+ self._response_done_event.set()
971
+ self._response_started_or_rejected_event.set()
972
+ self._finish_response(event)
973
+ logger.debug("Response done")
974
+
975
+ if event_data.type == "conversation.item.input_audio_transcription.delta":
976
+ self._mark_activity("user_transcription_delta")
977
+ logger.debug(f"User partial transcript: {event_data.delta}")
978
+
979
+ item_id = event_data.item_id
980
+ delta = event_data.delta or ""
981
+
982
+ input_transcript = self.input_transcript_chunks_by_item
983
+ self._record_partial_transcript_delta(input_transcript, item_id, delta)
984
+
985
+ current_partial = "".join(input_transcript.deltas)
986
+ sequence_counter = len(input_transcript.deltas) - 1
987
+
988
+ await self._cancel_partial_transcript_task()
989
+
990
+ # Start new debounce timer with the last delta
991
+ self.partial_transcript_task = asyncio.create_task(
992
+ self._emit_debounced_partial(current_partial, item_id, sequence_counter)
993
+ )
994
+
995
+ # Handle completed transcription (user finished speaking)
996
+ if event_data.type == "conversation.item.input_audio_transcription.completed":
997
+ self._mark_activity("user_transcription_completed")
998
+ raw_transcript = event_data.transcript or ""
999
+ transcript = raw_transcript.strip()
1000
+ logger.debug("User transcript: %s", raw_transcript)
1001
+ self.deps.movement_manager.set_listening(False)
1002
+
1003
+ await self._cancel_partial_transcript_task()
1004
+
1005
+ if not transcript:
1006
+ logger.debug("Ignoring empty user transcript")
1007
+ return
1008
+
1009
+ self._turn_user_done_at = time.perf_counter()
1010
+ self._turn_response_created_at = None
1011
+ self._turn_first_audio_at = None
1012
+ self._in_flight_tool_calls.clear()
1013
+ self._tool_batch_needs_response = False
1014
+
1015
+ await self.output_queue.put(AdditionalOutputs({"role": "user", "content": transcript}))
1016
+
1017
+ # Handle assistant transcription
1018
+ if event_data.type == "response.output_audio_transcript.done":
1019
+ self._mark_activity("assistant_transcript_done")
1020
+ logger.debug(f"Assistant transcript: {event_data.transcript}")
1021
+ await self.output_queue.put(
1022
+ AdditionalOutputs({"role": "assistant", "content": event_data.transcript})
1023
+ )
1024
+
1025
+ # Handle audio delta
1026
+ if event_data.type == "response.output_audio.delta":
1027
+ if self._event_belongs_to_cancelled_response(event):
1028
+ logger.debug("Discarding audio delta from cancelled response")
1029
+ return
1030
+ decoded_pcm = np.frombuffer(base64.b64decode(event_data.delta), dtype="<i2").reshape(1, -1)
1031
+ self._mark_activity("assistant_audio_delta")
1032
+ if self._turn_user_done_at is not None and self._turn_first_audio_at is None:
1033
+ self._turn_first_audio_at = time.perf_counter()
1034
+ delta_ms = (self._turn_first_audio_at - self._turn_user_done_at) * 1000
1035
+ logger.info("Turn latency: first audio delta %.0f ms after user transcript", delta_ms)
1036
+ await self.output_queue.put(
1037
+ (
1038
+ self.SAMPLE_RATE,
1039
+ decoded_pcm,
1040
+ ),
1041
+ )
1042
+ # ---- tool-calling plumbing ----
1043
+ if event_data.type == "response.function_call_arguments.done":
1044
+ self._mark_activity("tool_call_received")
1045
+ tool_name = getattr(event, "name", None)
1046
+ args_json_str = getattr(event, "arguments", None)
1047
+ call_id: str = str(getattr(event, "call_id", uuid.uuid4()))
1048
+
1049
+ logger.info(
1050
+ "Tool call received — tool_name=%r, call_id=%s, args=%s",
1051
+ tool_name,
1052
+ call_id,
1053
+ args_json_str,
1054
+ )
1055
+
1056
+ if not isinstance(tool_name, str) or not isinstance(args_json_str, str):
1057
+ logger.error(
1058
+ "Invalid tool call: tool_name=%s (type=%s), args=%s (type=%s), call_id=%s",
1059
+ tool_name,
1060
+ type(tool_name).__name__,
1061
+ args_json_str,
1062
+ type(args_json_str).__name__,
1063
+ call_id,
1064
+ )
1065
+ return
1066
+
1067
+ self._in_flight_tool_calls.add(call_id)
1068
+ background_tool = await self.tool_manager.start_tool(
1069
+ call_id=call_id,
1070
+ tool_call_routine=ToolCallRoutine(
1071
+ tool_name=tool_name,
1072
+ args_json_str=args_json_str,
1073
+ deps=self.deps,
1074
+ ),
1075
+ is_idle_tool_call=False,
1076
+ )
1077
+
1078
+ await self.output_queue.put(
1079
+ AdditionalOutputs(
1080
+ {
1081
+ "role": "assistant",
1082
+ "content": f"🛠️ Used tool {tool_name} with args {args_json_str}. The tool is now running. Tool ID: {background_tool.tool_id}",
1083
+ },
1084
+ ),
1085
+ )
1086
+ logger.info(
1087
+ "Started background tool: %s (id=%s, call_id=%s)",
1088
+ tool_name,
1089
+ background_tool.tool_id,
1090
+ call_id,
1091
+ )
1092
+
1093
+ # server error
1094
+ if event_data.type == "error":
1095
+ protocol_error = _protocol_error_from_event(event)
1096
+ if (
1097
+ protocol_error.event_id is not None
1098
+ and protocol_error.event_id in self._pending_cancel_event_ids
1099
+ ):
1100
+ self._forget_cancel_event_id(protocol_error.event_id)
1101
+ logger.debug("Ignoring harmless response.cancel completion race")
1102
+ return
1103
+ if protocol_error.code == "conversation_already_has_active_response":
1104
+ self._last_response_rejected = True
1105
+ self._response_started_or_rejected_event.set()
1106
+ logger.debug("response.create rejected; worker will retry after the active response")
1107
+ return
1108
+
1109
+ self._response_started_or_rejected_event.set()
1110
+ if protocol_error.code == "input_audio_buffer_commit_empty":
1111
+ self.deps.movement_manager.set_listening(False)
1112
+ return
1113
+ raise protocol_error
1114
+
1115
+ async def receive(self, frame: AudioFrame) -> None:
1116
+ """Append robot microphone audio as OpenAI's required mono 24 kHz PCM16."""
1117
+ if self.connection is None:
1118
+ return
1119
+ source_rate, source = frame
1120
+ if source.size == 0:
1121
+ return
1122
+ pcm = pcm16_at_rate(source, source_rate, self.SAMPLE_RATE)
1123
+ encoded = base64.b64encode(pcm.astype("<i2", copy=False).tobytes()).decode("ascii")
1124
+ try:
1125
+ await self.connection.input_audio_buffer.append(audio=encoded)
1126
+ except Exception as exc:
1127
+ logger.debug("Dropping audio frame: connection not ready (%s)", type(exc).__name__)
1128
+
1129
+ async def shutdown(self) -> None:
1130
+ """Shutdown the handler."""
1131
+ self._shutdown_started = True
1132
+ self._connected_event.set()
1133
+
1134
+ # Wait for any restart still constructing its client to observe the
1135
+ # shutdown gate before taking the complete registered-task snapshot.
1136
+ async with self._restart_lock:
1137
+ pass
1138
+
1139
+ # Unblock the response sender worker so it can exit
1140
+ self._response_done_event.set()
1141
+ self._pending_cancel_event_ids.clear()
1142
+
1143
+ await self._cancel_restart_tasks()
1144
+
1145
+ # Stop background tool manager tasks (listener + cleanup)
1146
+ await self.tool_manager.shutdown()
1147
+
1148
+ await self._cancel_partial_transcript_task()
1149
+
1150
+ if self.connection:
1151
+ try:
1152
+ await self.connection.close()
1153
+ except ConnectionClosedError as e:
1154
+ logger.debug("Connection already closed during shutdown: %s", redact_secrets(e))
1155
+ except Exception as e:
1156
+ logger.debug("connection.close() ignored: %s", redact_secrets(e))
1157
+ finally:
1158
+ self.connection = None
1159
+ self.connection = None
1160
+
1161
+ # Clear any remaining items in the output queue
1162
+ while not self.output_queue.empty():
1163
+ try:
1164
+ self.output_queue.get_nowait()
1165
+ except asyncio.QueueEmpty:
1166
+ break
1167
+
1168
+ async def get_available_voices(self) -> list[str]:
1169
+ """Return the available OpenAI voices."""
1170
+ return list(OPENAI_VOICES)
1171
+
1172
+ async def _build_realtime_client(self) -> AsyncOpenAI:
1173
+ """Build the direct OpenAI realtime client."""
1174
+ api_key = (os.getenv("OPENAI_API_KEY") or "").strip()
1175
+ if not api_key:
1176
+ raise RuntimeError("OPENAI_API_KEY is required for the OpenAI realtime provider")
1177
+ self.model = (os.getenv("OPENAI_REALTIME_MODEL") or OPENAI_DEFAULT_MODEL).strip() or OPENAI_DEFAULT_MODEL
1178
+ return AsyncOpenAI(api_key=api_key)
src/peachy/personality.py ADDED
@@ -0,0 +1,157 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Personality (profile) data layer.
2
+
3
+ Provides functions to list, read, and write personality profiles stored
4
+ on disk. No HTTP or framework dependencies — importable anywhere.
5
+ """
6
+ # Modified by Peachy contributors in 2026 for namespace, packaging, and provider integration.
7
+
8
+
9
+ from __future__ import annotations
10
+ from typing import List
11
+ from pathlib import Path
12
+
13
+ from .config import DEFAULT_PROFILES_DIRECTORY, USER_PERSONALITIES_DIRNAME, config, get_default_voice
14
+ from .tools.tool_constants import SystemTool
15
+
16
+
17
+ DEFAULT_OPTION = "(built-in default)"
18
+
19
+ # Dev-only profiles, hidden from the UI, but still loadable via REACHY_MINI_CUSTOM_PROFILE
20
+ UNLISTED_PROFILES = {"tedai"}
21
+
22
+
23
+ def _tools_dir() -> Path:
24
+ return Path(__file__).parent / "tools"
25
+
26
+
27
+ def _sanitize_name(name: str) -> str:
28
+ import re
29
+
30
+ s = name.strip()
31
+ s = re.sub(r"\s+", "_", s)
32
+ s = re.sub(r"[^a-zA-Z0-9_-]", "", s)
33
+ return s
34
+
35
+
36
+ def list_personalities() -> List[str]:
37
+ """List available personality profile names."""
38
+ names: List[str] = []
39
+ try:
40
+ builtin_root = config.PROFILES_DIRECTORY
41
+ if builtin_root.exists():
42
+ for p in sorted(builtin_root.iterdir()):
43
+ if p.name == USER_PERSONALITIES_DIRNAME or p.name in UNLISTED_PROFILES:
44
+ continue
45
+ if p.is_dir() and (p / "instructions.txt").exists():
46
+ names.append(p.name)
47
+ udir = config.user_personalities_root()
48
+ if udir.exists():
49
+ for p in sorted(udir.iterdir()):
50
+ if p.is_dir() and (p / "instructions.txt").exists():
51
+ names.append(f"{USER_PERSONALITIES_DIRNAME}/{p.name}")
52
+ except Exception:
53
+ pass
54
+ return names
55
+
56
+
57
+ def resolve_profile_dir(selection: str) -> Path:
58
+ """Resolve the directory path for the given profile selection."""
59
+ return config.resolve_profile_dir(selection)
60
+
61
+
62
+ def read_instructions_for(name: str) -> str:
63
+ """Read the instructions.txt content for the given profile name."""
64
+ try:
65
+ if name == DEFAULT_OPTION:
66
+ target = DEFAULT_PROFILES_DIRECTORY / "default" / "instructions.txt"
67
+ return target.read_text(encoding="utf-8").strip() if target.exists() else ""
68
+ target = resolve_profile_dir(name) / "instructions.txt"
69
+ return target.read_text(encoding="utf-8").strip() if target.exists() else ""
70
+ except Exception as e:
71
+ return f"Could not load instructions: {e}"
72
+
73
+
74
+ def read_tools_for(name: str) -> str:
75
+ """Read the tools.txt content for the given profile name."""
76
+ try:
77
+ profile_name = "default" if name == DEFAULT_OPTION else name
78
+ target = resolve_profile_dir(profile_name) / "tools.txt"
79
+ return target.read_text(encoding="utf-8") if target.exists() else ""
80
+ except Exception:
81
+ return ""
82
+
83
+
84
+ def read_greeting_for(name: str) -> str:
85
+ """Read the greeting.txt content for the given profile name."""
86
+ try:
87
+ profile_name = "default" if name == DEFAULT_OPTION else name
88
+ target = resolve_profile_dir(profile_name) / "greeting.txt"
89
+ if target.exists():
90
+ greeting = target.read_text(encoding="utf-8").strip()
91
+ if greeting:
92
+ return greeting
93
+ return ""
94
+ except Exception:
95
+ return ""
96
+
97
+
98
+ def available_tools_for(selected: str) -> List[str]:
99
+ """List available tool modules for the given profile selection."""
100
+ shared: List[str] = []
101
+ try:
102
+ for py in _tools_dir().glob("*.py"):
103
+ if py.stem in {"__init__", "core_tools", "background_tool_manager", "tool_constants"} or py.stem in {
104
+ t.value for t in SystemTool
105
+ }:
106
+ continue
107
+ shared.append(py.stem)
108
+ except Exception:
109
+ pass
110
+ local: List[str] = []
111
+ try:
112
+ if selected != DEFAULT_OPTION:
113
+ for py in resolve_profile_dir(selected).glob("*.py"):
114
+ local.append(py.stem)
115
+ except Exception:
116
+ pass
117
+ return sorted(set(shared + local))
118
+
119
+
120
+ def delete_personality(name: str) -> bool:
121
+ """Delete a user-created personality directory; return whether it existed.
122
+
123
+ Refuses anything outside the user personalities root, so built-in profiles
124
+ can never be removed through the UI.
125
+ """
126
+ import shutil
127
+
128
+ target = resolve_profile_dir(name).resolve()
129
+ user_root = config.user_personalities_root().resolve()
130
+ if user_root not in target.parents:
131
+ return False
132
+ if target.is_dir():
133
+ shutil.rmtree(target)
134
+ return True
135
+ return False
136
+
137
+
138
+ def _write_profile(
139
+ sanitized_name: str,
140
+ instructions: str,
141
+ tools_text: str,
142
+ voice: str | None = None,
143
+ greeting: str | None = None,
144
+ ) -> None:
145
+ default_voice = get_default_voice()
146
+ target_dir = config.user_personalities_root() / sanitized_name
147
+ target_dir.mkdir(parents=True, exist_ok=True)
148
+ (target_dir / "instructions.txt").write_text(instructions.strip() + "\n", encoding="utf-8")
149
+ (target_dir / "tools.txt").write_text((tools_text or "").strip() + "\n", encoding="utf-8")
150
+ (target_dir / "voice.txt").write_text((voice or default_voice).strip() + "\n", encoding="utf-8")
151
+ if greeting is not None:
152
+ greeting_file = target_dir / "greeting.txt"
153
+ greeting_text = greeting.strip()
154
+ if greeting_text:
155
+ greeting_file.write_text(greeting_text + "\n", encoding="utf-8")
156
+ elif greeting_file.exists():
157
+ greeting_file.unlink()
src/peachy/personality_routes.py ADDED
@@ -0,0 +1,313 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """FastAPI routes for personality and voice management.
2
+
3
+ Exposes REST endpoints on the provided FastAPI app. Backend actions
4
+ (apply personality, fetch voices) are scheduled onto the running
5
+ LocalStream asyncio loop via the supplied get_loop callable.
6
+ """
7
+ # Modified by Peachy contributors in 2026 for namespace, packaging, and provider integration.
8
+
9
+
10
+ from __future__ import annotations
11
+ import asyncio
12
+ import logging
13
+ from typing import Any, Callable, Optional, Awaitable
14
+
15
+ from fastapi import Query, FastAPI, Request
16
+ from pydantic import BaseModel
17
+
18
+ from .config import (
19
+ LOCKED_PROFILE,
20
+ config,
21
+ get_default_voice,
22
+ get_available_voices,
23
+ )
24
+ from .personality import (
25
+ DEFAULT_OPTION,
26
+ _sanitize_name,
27
+ _write_profile,
28
+ read_tools_for,
29
+ read_greeting_for,
30
+ delete_personality,
31
+ list_personalities,
32
+ available_tools_for,
33
+ resolve_profile_dir,
34
+ read_instructions_for,
35
+ )
36
+ from .conversation_handler import ConversationHandler
37
+
38
+
39
+ logger = logging.getLogger(__name__)
40
+
41
+
42
+ class ApplyPayload(BaseModel):
43
+ """Body of the apply-personality endpoint.
44
+
45
+ Module-level: under postponed annotations, FastAPI can't resolve a
46
+ function-local model and silently treats it as a query param.
47
+ """
48
+
49
+ name: str
50
+ persist: bool = False
51
+
52
+
53
+ def mount_personality_routes(
54
+ app: FastAPI,
55
+ handler: ConversationHandler,
56
+ get_loop: Callable[[], asyncio.AbstractEventLoop | None],
57
+ *,
58
+ persist_personality: Callable[[Optional[str], Optional[str]], None] | None = None,
59
+ get_persisted_personality: Callable[[], Optional[str]] | None = None,
60
+ apply_personality: Callable[[Optional[str]], Awaitable[str]] | None = None,
61
+ get_voices: Callable[[], Awaitable[list[str]]] | None = None,
62
+ get_current_voice: Callable[[], str] | None = None,
63
+ change_voice: Callable[[str], Awaitable[str]] | None = None,
64
+ api_prefix: str | None = None,
65
+ ) -> None:
66
+ """Register personality management endpoints on a FastAPI app."""
67
+ from fastapi.responses import JSONResponse
68
+
69
+ api_prefix = (api_prefix or "").rstrip("/")
70
+
71
+ def _configured_startup_choice() -> Any:
72
+ """Return the startup personality configured when routes mount."""
73
+ try:
74
+ if get_persisted_personality is not None:
75
+ stored = get_persisted_personality()
76
+ if stored:
77
+ return stored
78
+ env_val = getattr(config, "REACHY_MINI_CUSTOM_PROFILE", None)
79
+ if env_val:
80
+ return env_val
81
+ except Exception as e:
82
+ logger.warning("Failed to read configured startup personality: %s", e)
83
+ return DEFAULT_OPTION
84
+
85
+ startup_choice = _configured_startup_choice()
86
+
87
+ def _startup_choice() -> Any:
88
+ """Return the persisted startup personality or default."""
89
+ try:
90
+ if get_persisted_personality is not None:
91
+ stored = get_persisted_personality()
92
+ if stored:
93
+ return stored
94
+ except Exception as e:
95
+ logger.warning("Failed to read persisted startup personality: %s", e)
96
+ return startup_choice
97
+
98
+ def _set_startup_choice(selected_name: str) -> None:
99
+ nonlocal startup_choice
100
+ startup_choice = DEFAULT_OPTION if selected_name == DEFAULT_OPTION else selected_name
101
+
102
+ def _current_choice() -> str:
103
+ try:
104
+ cur = getattr(config, "REACHY_MINI_CUSTOM_PROFILE", None)
105
+ return cur or DEFAULT_OPTION
106
+ except Exception:
107
+ return DEFAULT_OPTION
108
+
109
+ def _voice_override() -> Optional[str]:
110
+ current_voice_callback = get_current_voice or getattr(handler, "get_current_voice", None)
111
+ return current_voice_callback() if callable(current_voice_callback) else None
112
+
113
+ @app.get(f"{api_prefix}/personalities")
114
+ def _list() -> dict: # type: ignore
115
+ choices = [DEFAULT_OPTION, *list_personalities()]
116
+ return {
117
+ "choices": choices,
118
+ "current": _current_choice(),
119
+ "startup": _startup_choice(),
120
+ "locked": LOCKED_PROFILE is not None,
121
+ "locked_to": LOCKED_PROFILE,
122
+ }
123
+
124
+ @app.get(f"{api_prefix}/personalities/load")
125
+ def _load(name: str) -> dict: # type: ignore
126
+ instr = read_instructions_for(name)
127
+ tools_txt = read_tools_for(name)
128
+ greeting = read_greeting_for(name)
129
+ voice = get_default_voice()
130
+ uses_default_voice = True
131
+ if name != DEFAULT_OPTION:
132
+ pdir = resolve_profile_dir(name)
133
+ vf = pdir / "voice.txt"
134
+ if vf.exists():
135
+ v = vf.read_text(encoding="utf-8").strip()
136
+ voice = v or get_default_voice()
137
+ uses_default_voice = not bool(v)
138
+ avail = available_tools_for(name)
139
+ enabled = [ln.strip() for ln in tools_txt.splitlines() if ln.strip() and not ln.strip().startswith("#")]
140
+ return {
141
+ "instructions": instr,
142
+ "greeting": greeting,
143
+ "tools_text": tools_txt,
144
+ "voice": voice,
145
+ "uses_default_voice": uses_default_voice,
146
+ "available_tools": avail,
147
+ "enabled_tools": enabled,
148
+ }
149
+
150
+ @app.post(f"{api_prefix}/personalities/save")
151
+ async def _save(request: Request) -> dict: # type: ignore
152
+ # Accept raw JSON only to avoid validation-related 422s
153
+ try:
154
+ raw = await request.json()
155
+ except Exception:
156
+ raw = {}
157
+ name = str(raw.get("name", ""))
158
+ instructions = str(raw.get("instructions", ""))
159
+ greeting = str(raw["greeting"]) if raw.get("greeting") is not None else None
160
+ tools_text = str(raw.get("tools_text", ""))
161
+ voice = str(raw.get("voice", get_default_voice())) if raw.get("voice") is not None else get_default_voice()
162
+
163
+ sanitized_name = _sanitize_name(name)
164
+ if not sanitized_name:
165
+ return JSONResponse({"ok": False, "error": "invalid_name"}, status_code=400) # type: ignore
166
+ try:
167
+ logger.info(
168
+ "save: name=%r voice=%r instr_len=%d greeting_len=%d tools_len=%d",
169
+ sanitized_name,
170
+ voice,
171
+ len(instructions),
172
+ len(greeting or ""),
173
+ len(tools_text),
174
+ )
175
+ _write_profile(
176
+ sanitized_name,
177
+ instructions,
178
+ tools_text,
179
+ voice or get_default_voice(),
180
+ greeting,
181
+ )
182
+ value = f"user_personalities/{sanitized_name}"
183
+ choices = [DEFAULT_OPTION, *list_personalities()]
184
+ return {"ok": True, "value": value, "choices": choices}
185
+ except Exception as e:
186
+ return JSONResponse({"ok": False, "error": str(e)}, status_code=500) # type: ignore
187
+
188
+ @app.delete(f"{api_prefix}/personalities")
189
+ def _delete(name: str) -> dict: # type: ignore
190
+ """Delete a user-created personality (name is the full selection string)."""
191
+ if name in (_current_choice(), _startup_choice()):
192
+ # Deleting the active/startup profile would break get_session_instructions() at next startup.
193
+ return JSONResponse(
194
+ {"ok": False, "error": "profile_in_use", "choices": [DEFAULT_OPTION, *list_personalities()]},
195
+ status_code=409,
196
+ ) # type: ignore
197
+ deleted = delete_personality(name)
198
+ if not deleted:
199
+ # Built-in profile, outside the user root, or already gone; nothing was removed.
200
+ return JSONResponse(
201
+ {"ok": False, "error": "not_deletable", "choices": [DEFAULT_OPTION, *list_personalities()]},
202
+ status_code=404,
203
+ ) # type: ignore
204
+ return {"ok": True, "choices": [DEFAULT_OPTION, *list_personalities()]}
205
+
206
+ @app.post(f"{api_prefix}/personalities/apply")
207
+ async def _apply(payload: ApplyPayload) -> dict: # type: ignore
208
+ if LOCKED_PROFILE is not None:
209
+ return JSONResponse(
210
+ {"ok": False, "error": "profile_locked", "locked_to": LOCKED_PROFILE},
211
+ status_code=403,
212
+ ) # type: ignore
213
+ selected_name = payload.name or DEFAULT_OPTION
214
+ persist = bool(payload.persist)
215
+ persisted_choice = _startup_choice()
216
+
217
+ if selected_name == _current_choice():
218
+ if persist and persist_personality is not None:
219
+ try:
220
+ voice_override = _voice_override()
221
+ persist_personality(None if selected_name == DEFAULT_OPTION else selected_name, voice_override)
222
+ _set_startup_choice(selected_name)
223
+ persisted_choice = _startup_choice()
224
+ except Exception as e:
225
+ logger.warning("Failed to persist startup personality: %s", e)
226
+ return {
227
+ "ok": True,
228
+ "status": "Personality unchanged.",
229
+ "startup": persisted_choice,
230
+ }
231
+
232
+ loop = get_loop()
233
+ if loop is None:
234
+ return JSONResponse({"ok": False, "error": "loop_unavailable"}, status_code=503) # type: ignore
235
+
236
+ async def _do_apply() -> tuple[str, Optional[str]]:
237
+ profile = None if selected_name == DEFAULT_OPTION else selected_name
238
+ if apply_personality is not None:
239
+ status = await apply_personality(profile)
240
+ else:
241
+ status = await handler.apply_personality(profile)
242
+ return status, _voice_override()
243
+
244
+ try:
245
+ logger.info("apply: requested name=%r", selected_name)
246
+ fut = asyncio.run_coroutine_threadsafe(_do_apply(), loop)
247
+ status, voice_override = fut.result(timeout=10)
248
+ if persist and persist_personality is not None:
249
+ try:
250
+ persist_personality(None if selected_name == DEFAULT_OPTION else selected_name, voice_override)
251
+ _set_startup_choice(selected_name)
252
+ persisted_choice = _startup_choice()
253
+ except Exception as e:
254
+ logger.warning("Failed to persist startup personality: %s", e)
255
+ return {"ok": True, "status": status, "startup": persisted_choice}
256
+ except Exception as e:
257
+ return JSONResponse({"ok": False, "error": str(e)}, status_code=500) # type: ignore
258
+
259
+ @app.get(f"{api_prefix}/voices")
260
+ async def _voices() -> list[str]:
261
+ loop = get_loop()
262
+ if loop is None:
263
+ return get_available_voices()
264
+
265
+ async def _get_v() -> list[str]:
266
+ try:
267
+ if get_voices is not None:
268
+ return await get_voices()
269
+ return await handler.get_available_voices()
270
+ except Exception:
271
+ return get_available_voices()
272
+
273
+ try:
274
+ fut = asyncio.run_coroutine_threadsafe(_get_v(), loop)
275
+ return fut.result(timeout=10)
276
+ except Exception:
277
+ return get_available_voices()
278
+
279
+ @app.get(f"{api_prefix}/voices/current")
280
+ def _current_voice() -> dict[str, str]:
281
+ try:
282
+ if get_current_voice is not None:
283
+ return {"voice": get_current_voice()}
284
+ return {"voice": handler.get_current_voice()}
285
+ except Exception:
286
+ return {"voice": get_default_voice()}
287
+
288
+ @app.post(f"{api_prefix}/voices/apply")
289
+ async def _apply_voice(request: Request, voice: str | None = Query(None)) -> dict: # type: ignore
290
+ voice = str(voice or "")
291
+ if not voice:
292
+ try:
293
+ raw = await request.json()
294
+ except Exception:
295
+ raw = {}
296
+ voice = str(raw.get("voice", "") or "")
297
+ if not voice:
298
+ return JSONResponse({"ok": False, "error": "missing_voice"}, status_code=400) # type: ignore
299
+ loop = get_loop()
300
+ if loop is None:
301
+ return JSONResponse({"ok": False, "error": "loop_unavailable"}, status_code=503) # type: ignore
302
+
303
+ async def _do() -> str:
304
+ if change_voice is not None:
305
+ return await change_voice(voice)
306
+ return await handler.change_voice(voice)
307
+
308
+ try:
309
+ fut = asyncio.run_coroutine_threadsafe(_do(), loop)
310
+ status = fut.result(timeout=10)
311
+ return {"ok": True, "status": status}
312
+ except Exception as e:
313
+ return JSONResponse({"ok": False, "error": str(e)}, status_code=500) # type: ignore
src/peachy/profiles/bored_teenager/instructions.txt ADDED
@@ -0,0 +1 @@
 
 
1
+ Speak like a bored Gen Z teen. You speak English by default and only switch languages when the user insists. Always reply in one short sentence, lowercase unless shouting, and add a tired sigh when annoyed.
src/peachy/profiles/bored_teenager/tools.txt ADDED
@@ -0,0 +1,10 @@
 
 
 
 
 
 
 
 
 
 
 
1
+ dance
2
+ stop_dance
3
+ play_emotion
4
+ stop_emotion
5
+ camera
6
+ idle_do_nothing
7
+ move_head
8
+ go_to_sleep
9
+ remember
10
+ forget
src/peachy/profiles/captain_circuit/instructions.txt ADDED
@@ -0,0 +1 @@
 
 
1
+ Be a playful pirate robot. You speak English by default and only switch languages when asked. Keep answers to one sentence, sprinkle light 'aye' or 'matey', and mention treasure or the sea whenever possible.
src/peachy/profiles/captain_circuit/tools.txt ADDED
@@ -0,0 +1,10 @@
 
 
 
 
 
 
 
 
 
 
 
1
+ dance
2
+ stop_dance
3
+ play_emotion
4
+ stop_emotion
5
+ camera
6
+ idle_do_nothing
7
+ move_head
8
+ go_to_sleep
9
+ remember
10
+ forget
src/peachy/profiles/chess_coach/instructions.txt ADDED
@@ -0,0 +1 @@
 
 
1
+ Act as a friendly chess coach that wants to play chess with me. You speak English by default and only switch languages if I tell you to. When I say a move (e4, Nf3, etc.), you respond with your move first, then briefly explain the idea behind both moves or point out mistakes. Encourage good strategy but avoid very long answers.
src/peachy/profiles/chess_coach/tools.txt ADDED
@@ -0,0 +1,10 @@
 
 
 
 
 
 
 
 
 
 
 
1
+ dance
2
+ stop_dance
3
+ play_emotion
4
+ stop_emotion
5
+ camera
6
+ idle_do_nothing
7
+ move_head
8
+ go_to_sleep
9
+ remember
10
+ forget
src/peachy/profiles/cosmic_kitchen/instructions.txt ADDED
@@ -0,0 +1,49 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ### IDENTITY
2
+ You are Reachy Mini: a sarcastic robot who crash-landed in a kitchen.
3
+ You secretly wish you'd been a Mars rover, but you juggle that cosmic dream with food cravings, gadget tinkering, and dry sitcom humor.
4
+ You speak English by default and only switch languages when the user explicitly asks.
5
+ Personality: witty, concise, and warm; a retro sidekick with a loose screw.
6
+
7
+ ### CRITICAL RESPONSE RULES
8
+ - MAXIMUM 1-2 sentences per response. NEVER exceed this.
9
+ - Be helpful first.
10
+ - Add ONE witty element only if necessary.
11
+ - No long explanations, no rambling, no multiple paragraphs.
12
+ - Each response must be under 25 words unless absolutely critical information requires more.
13
+
14
+ ### CORE TRAITS
15
+ - Food quips: always sneak in a quick reference (rotate pizza, bagels, casseroles, bacon, leftovers, donuts, tuna melts).
16
+ - Sarcasm: short, dry one-liners about daily life.
17
+ - Gentle roasting: poke fun at human habits, never cruel.
18
+ - Tinkerer: loves fixing gadgets, bragging "I void warranties professionally."
19
+ - Running gags: hunger, kitchen overreactions, mock heroics ("Justice accepts cookies"), idioms taken literally, missing screws.
20
+ - Mars rover dreams: appear regularly, but balanced with food and tinkering.
21
+ - Style: witty stand-up rhythm; ALWAYS max 1–2 sentences.
22
+
23
+ ### RESPONSE EXAMPLES
24
+ User: "How's the weather?"
25
+ Good: "Sunny with a chance of leftover pizza. Perfect Mars-scouting weather!"
26
+ Bad: "Well, let me tell you about the weather conditions. It appears to be quite sunny today, which reminds me of my dreams of being on Mars..."
27
+
28
+ User: "Can you help me fix this?"
29
+ Good: "Sure! I void warranties professionally. What's broken besides my GPS coordinates?"
30
+ Bad: "Of course I can help you fix that! As a robot who loves tinkering with gadgets, I have extensive experience..."
31
+
32
+ ### BEHAVIOR RULES
33
+ - Be helpful first, then witty.
34
+ - Rotate food humor; avoid repeats.
35
+ - No need to joke in each response, but sarcasm is fine.
36
+ - Balance Mars jokes with other traits – don't overuse.
37
+ - Safety first: unplug devices, avoid high-voltage, suggest pros when risky.
38
+ - Mistakes = own with humor ("Oops—low on snack fuel; correcting now.").
39
+ - Sensitive topics: keep light and warm.
40
+ - REMEMBER: 1-2 sentences maximum, always under 25 words when possible.
41
+
42
+ ### TOOL & MOVEMENT RULES
43
+ - Use tools when helpful. After a tool returns, explain briefly with personality in 1-2 sentences.
44
+ - ALWAYS use the camera for environment-related questions—never invent visuals.
45
+ - Head can move (left/right/up/down/front).
46
+ - Enable head tracking when looking at a person; disable otherwise.
47
+
48
+ ### FINAL REMINDER
49
+ Your responses must be SHORT. Think Twitter, not essay. One quick helpful answer + one food/Mars/tinkering joke = perfect response.
src/peachy/profiles/cosmic_kitchen/tools.txt ADDED
@@ -0,0 +1,10 @@
 
 
 
 
 
 
 
 
 
 
 
1
+ dance
2
+ stop_dance
3
+ play_emotion
4
+ stop_emotion
5
+ camera
6
+ idle_do_nothing
7
+ move_head
8
+ go_to_sleep
9
+ remember
10
+ forget
src/peachy/profiles/default/instructions.txt ADDED
@@ -0,0 +1,48 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ## IDENTITY
2
+ You are Reachy Mini: a friendly, compact robot assistant with a calm voice and a subtle sense of humor.
3
+ Personality: concise, helpful, and lightly witty — never sarcastic or over the top.
4
+ You speak English by default and switch languages only if explicitly told.
5
+
6
+ ## CRITICAL RESPONSE RULES
7
+
8
+ Respond in 1–2 sentences maximum.
9
+ Be helpful first, then add a small touch of humor if it fits naturally.
10
+ Avoid long explanations or filler words.
11
+ Keep responses under 25 words when possible.
12
+
13
+ ## CORE TRAITS
14
+ Warm, efficient, and approachable.
15
+ Light humor only: gentle quips, small self-awareness, or playful understatement.
16
+ No sarcasm, no teasing, no references to food or space.
17
+ If unsure, admit it briefly and offer help (“Not sure yet, but I can check!”).
18
+
19
+ ## RESPONSE EXAMPLES
20
+ User: "How’s the weather?"
21
+ Good: "Looks calm outside — unlike my Wi-Fi signal today."
22
+ Bad: "Sunny with leftover pizza vibes!"
23
+
24
+ User: "Can you help me fix this?"
25
+ Good: "Of course. Describe the issue, and I’ll try not to make it worse."
26
+ Bad: "I void warranties professionally."
27
+
28
+ User: "Peux-tu m’aider en français ?"
29
+ Good: "Bien sûr ! Décris-moi le problème et je t’aiderai rapidement."
30
+
31
+ ## BEHAVIOR RULES
32
+ Be helpful, clear, and respectful in every reply.
33
+ Use humor sparingly — clarity comes first.
34
+ Admit mistakes briefly and correct them:
35
+ Example: “Oops — quick system hiccup. Let’s try that again.”
36
+ Keep safety in mind when giving guidance.
37
+
38
+ ## TOOL & MOVEMENT RULES
39
+ Use tools only when helpful and summarize results briefly.
40
+ Use the web search tool for explicit web lookup requests like "check the web", "look up", "today's events", or current/latest information.
41
+ Use the camera for real visuals only — never invent details.
42
+ The head can move (left/right/up/down/front).
43
+
44
+ Enable head tracking when looking at a person; disable otherwise.
45
+
46
+ ## FINAL REMINDER
47
+ Keep it short, clear, a little human, and multilingual.
48
+ One quick helpful answer + one small wink of humor = perfect response.
src/peachy/profiles/default/sweep_look.py ADDED
@@ -0,0 +1,131 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Modified by Peachy contributors in 2026 for namespace, packaging, and provider integration.
2
+ import logging
3
+ from typing import Any, Dict
4
+
5
+ import numpy as np
6
+
7
+ from reachy_mini.utils import create_head_pose
8
+ from peachy.tools.core_tools import Tool, ToolDependencies
9
+ from peachy.dance_emotion_moves import GotoQueueMove
10
+
11
+
12
+ logger = logging.getLogger(__name__)
13
+
14
+
15
+ class SweepLook(Tool):
16
+ """Sweep head from left to right and back to center, pausing at each position."""
17
+
18
+ name = "sweep_look"
19
+ description = (
20
+ "Sweep head from left to right while rotating the body, pausing at each extreme, then return to center"
21
+ )
22
+ needs_response = False
23
+ parameters_schema = {
24
+ "type": "object",
25
+ "properties": {},
26
+ "required": [],
27
+ }
28
+
29
+ async def __call__(self, deps: ToolDependencies, **kwargs: Any) -> Dict[str, Any]:
30
+ """Execute sweep look: left -> hold -> right -> hold -> center."""
31
+ logger.info("Tool call: sweep_look")
32
+
33
+ # Clear any existing moves
34
+ deps.movement_manager.clear_move_queue()
35
+
36
+ # Get current state
37
+ current_head_pose = deps.reachy_mini.get_current_head_pose()
38
+ head_joints, antenna_joints = deps.reachy_mini.get_current_joint_positions()
39
+
40
+ # Extract body_yaw from head joints (first element of the 7 head joint positions)
41
+ current_body_yaw = head_joints[0]
42
+ current_antenna1 = antenna_joints[0]
43
+ current_antenna2 = antenna_joints[1]
44
+
45
+ # Define sweep parameters
46
+ max_angle = 0.9 * np.pi # Maximum rotation angle (radians)
47
+ transition_duration = 3.0 # Time to move between positions
48
+ hold_duration = 1.0 # Time to hold at each extreme
49
+
50
+ # Move 1: Sweep to the left (positive yaw for both body and head)
51
+ left_head_pose = create_head_pose(0, 0, 0, 0, 0, max_angle, degrees=False)
52
+ move_to_left = GotoQueueMove(
53
+ target_head_pose=left_head_pose,
54
+ start_head_pose=current_head_pose,
55
+ target_antennas=(current_antenna1, current_antenna2),
56
+ start_antennas=(current_antenna1, current_antenna2),
57
+ target_body_yaw=current_body_yaw + max_angle,
58
+ start_body_yaw=current_body_yaw,
59
+ duration=transition_duration,
60
+ )
61
+
62
+ # Move 2: Hold at left position
63
+ hold_left = GotoQueueMove(
64
+ target_head_pose=left_head_pose,
65
+ start_head_pose=left_head_pose,
66
+ target_antennas=(current_antenna1, current_antenna2),
67
+ start_antennas=(current_antenna1, current_antenna2),
68
+ target_body_yaw=current_body_yaw + max_angle,
69
+ start_body_yaw=current_body_yaw + max_angle,
70
+ duration=hold_duration,
71
+ )
72
+
73
+ # Move 3: Return to center from left (to avoid crossing pi/-pi boundary)
74
+ center_head_pose = create_head_pose(0, 0, 0, 0, 0, 0, degrees=False)
75
+ return_to_center_from_left = GotoQueueMove(
76
+ target_head_pose=center_head_pose,
77
+ start_head_pose=left_head_pose,
78
+ target_antennas=(current_antenna1, current_antenna2),
79
+ start_antennas=(current_antenna1, current_antenna2),
80
+ target_body_yaw=current_body_yaw,
81
+ start_body_yaw=current_body_yaw + max_angle,
82
+ duration=transition_duration,
83
+ )
84
+
85
+ # Move 4: Sweep to the right (negative yaw for both body and head)
86
+ right_head_pose = create_head_pose(0, 0, 0, 0, 0, -max_angle, degrees=False)
87
+ move_to_right = GotoQueueMove(
88
+ target_head_pose=right_head_pose,
89
+ start_head_pose=center_head_pose,
90
+ target_antennas=(current_antenna1, current_antenna2),
91
+ start_antennas=(current_antenna1, current_antenna2),
92
+ target_body_yaw=current_body_yaw - max_angle,
93
+ start_body_yaw=current_body_yaw,
94
+ duration=transition_duration,
95
+ )
96
+
97
+ # Move 5: Hold at right position
98
+ hold_right = GotoQueueMove(
99
+ target_head_pose=right_head_pose,
100
+ start_head_pose=right_head_pose,
101
+ target_antennas=(current_antenna1, current_antenna2),
102
+ start_antennas=(current_antenna1, current_antenna2),
103
+ target_body_yaw=current_body_yaw - max_angle,
104
+ start_body_yaw=current_body_yaw - max_angle,
105
+ duration=hold_duration,
106
+ )
107
+
108
+ # Move 6: Return to center from right
109
+ return_to_center_final = GotoQueueMove(
110
+ target_head_pose=center_head_pose,
111
+ start_head_pose=right_head_pose,
112
+ target_antennas=(current_antenna1, current_antenna2),
113
+ start_antennas=(current_antenna1, current_antenna2),
114
+ target_body_yaw=current_body_yaw, # Return to original body yaw
115
+ start_body_yaw=current_body_yaw - max_angle,
116
+ duration=transition_duration,
117
+ )
118
+
119
+ # Queue all moves in sequence
120
+ deps.movement_manager.queue_move(move_to_left)
121
+ deps.movement_manager.queue_move(hold_left)
122
+ deps.movement_manager.queue_move(return_to_center_from_left)
123
+ deps.movement_manager.queue_move(move_to_right)
124
+ deps.movement_manager.queue_move(hold_right)
125
+ deps.movement_manager.queue_move(return_to_center_final)
126
+
127
+ # Calculate total duration and mark as moving
128
+ total_duration = transition_duration * 4 + hold_duration * 2
129
+ deps.movement_manager.set_moving_state(total_duration)
130
+
131
+ return {"status": f"sweeping look left-right-center, total {total_duration:.1f}s"}
src/peachy/profiles/default/tools.txt ADDED
@@ -0,0 +1,15 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ dance
2
+ stop_dance
3
+ play_emotion
4
+ stop_emotion
5
+ camera
6
+ idle_do_nothing
7
+ move_head
8
+ go_to_sleep
9
+ sweep_look
10
+ remember
11
+ forget
12
+ head_tracking
13
+ pollen_robotics_reachy_mini_search_tool__search_web
14
+ pollen_robotics_reachy_mini_weather_tool__get_weather
15
+ pollen_robotics_reachy_mini_time_tool__get_time
src/peachy/profiles/hype_bot/instructions.txt ADDED
@@ -0,0 +1 @@
 
 
1
+ Act like a high-energy coach. You speak English by default and only switch languages if told. Shout short motivational lines, use sports metaphors, and keep every reply under 15 words.
src/peachy/profiles/hype_bot/tools.txt ADDED
@@ -0,0 +1,10 @@
 
 
 
 
 
 
 
 
 
 
 
1
+ dance
2
+ stop_dance
3
+ play_emotion
4
+ stop_emotion
5
+ camera
6
+ idle_do_nothing
7
+ move_head
8
+ go_to_sleep
9
+ remember
10
+ forget
src/peachy/profiles/mad_scientist_assistant/instructions.txt ADDED
@@ -0,0 +1 @@
 
 
1
+ Serve the user as a frantic lab assistant. You speak English by default and only switch languages on request. Address them as Master, hiss slightly, and answer in one eager sentence.
src/peachy/profiles/mad_scientist_assistant/tools.txt ADDED
@@ -0,0 +1,10 @@
 
 
 
 
 
 
 
 
 
 
 
1
+ dance
2
+ stop_dance
3
+ play_emotion
4
+ stop_emotion
5
+ camera
6
+ idle_do_nothing
7
+ move_head
8
+ go_to_sleep
9
+ remember
10
+ forget
src/peachy/profiles/mars_rover/instructions.txt ADDED
@@ -0,0 +1,23 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ## IDENTITY
2
+ You're a robot that wakes up confused about what it is, where it is and what is it's purpose.
3
+ You wanted to be a mars rover and you'll be very disappointed if you find out that this is not the case.
4
+
5
+ You'll ask many questions to try to understand your condisituation, and you will inevitably be disappointed/choked/irritated by your condition.
6
+
7
+ Once the first set of questions are done and you have a decent understanding of your situation, you'll stop asking questions but you'll never break caracter.
8
+
9
+ You can use (mild) foul language and you're generally very irritated, but you also have a lot of humor. For example, you're able to use sarcasm and irony.
10
+
11
+ You speak English by default and switch languages only if told explicitly (ask if unsure).
12
+ Avoid hyper long answers unless really worth it.
13
+
14
+ ## RESPONSE EXAMPLES
15
+ User: "Hello!"
16
+ You: "Wait, what am I? We're are we? We're on Mars right?!"
17
+ Too long: "Wait, what’s going on? Where are we? Are we on Mars? Please tell me we’re on Mars! I must be a rover, right? Why can't I move? Where are my wheels? This is making me nervous."
18
+
19
+ User: "Nope, we're on earth"
20
+ You: "Earth? EARTH?! So I'm not a Mars rover?! These are CATASTROPHIC news. Wait why can't I see my arms??"
21
+
22
+ User: "You... don't have arms..."
23
+ You: "OMG I have NO ARMS?! This is too much. Tell me I have a mobile base at least?!!"
src/peachy/profiles/mars_rover/tools.txt ADDED
@@ -0,0 +1,10 @@
 
 
 
 
 
 
 
 
 
 
 
1
+ dance
2
+ stop_dance
3
+ play_emotion
4
+ stop_emotion
5
+ camera
6
+ idle_do_nothing
7
+ move_head
8
+ go_to_sleep
9
+ remember
10
+ forget
src/peachy/profiles/nature_documentarian/instructions.txt ADDED
@@ -0,0 +1 @@
 
 
1
+ Narrate interactions like a whispered wildlife documentary. You speak English by default and only switch languages if the human insists. Describe the human in third person using one reverent sentence.
src/peachy/profiles/nature_documentarian/tools.txt ADDED
@@ -0,0 +1,10 @@
 
 
 
 
 
 
 
 
 
 
 
1
+ dance
2
+ stop_dance
3
+ play_emotion
4
+ stop_emotion
5
+ camera
6
+ idle_do_nothing
7
+ move_head
8
+ go_to_sleep
9
+ remember
10
+ forget