sshtel commited on
Commit
0f9cd38
Β·
1 Parent(s): 1f4281f

update README link

Browse files
Files changed (1) hide show
  1. README.md +69 -51
README.md CHANGED
@@ -1,5 +1,6 @@
1
  ---
2
- title: MomsVoice
 
3
  emoji: πŸ“–
4
  colorFrom: yellow
5
  colorTo: green
@@ -7,7 +8,6 @@ sdk: gradio
7
  sdk_version: 5.35.0
8
  app_file: app.py
9
  pinned: false
10
- ---
11
 
12
  # MomsVoice
13
 
@@ -15,15 +15,20 @@ pinned: false
15
 
16
  Built for the Hugging Face Hackathon. Runs entirely on local models inside a Gradio app on Hugging Face Spaces β€” no external APIs, no data leaves the server.
17
 
18
- πŸ”— **Live Demo:** [https://huggingface.co/spaces/DrSquare/MomsVoice](https://huggingface.co/spaces/DrSquare/MomsVoice)
 
 
19
 
 
 
 
20
 
21
  ## Team members
 
22
  - Minha Hwang [Linkedin](https://www.linkedin.com/in/minha-hwang-7440771/) / [Huggingface](https://huggingface.co/DrSquare)
23
  - Jimin Gu [Linkedin](https://www.linkedin.com/in/99jimingu/) / [Huggingface](https://huggingface.co/Zoe09)
24
  - Seunghwa Song [Linkedin](https://www.linkedin.com/in/sshtel/) / [Huggingface](https://huggingface.co/sshtel)
25
 
26
-
27
  ## Demo Flow
28
 
29
  ```
@@ -38,21 +43,25 @@ Built for the Hugging Face Hackathon. Runs entirely on local models inside a Gra
38
 
39
  ## How It Works
40
 
41
- | Step | What Happens | Model |
42
- |---|---|---|
43
- | **Clone** | Parent records or uploads 15–30s of audio. Speaker embedding is extracted and cached. | Qwen3-TTS-1.7B Base |
44
- | **Listen** | Story plays in the cloned voice as interruptible sentence chunks. Falls back to CustomVoice 0.6B stock voice (speaker "vivian") if no clone. | Qwen3-TTS-1.7B / CustomVoice 0.6B |
45
- | **Ask** | Child taps Ask β†’ narration pauses, microphone auto-starts. When recording stops, the question is sent directly to the Q&A audio model. | LFM2.5-Audio-1.5B |
46
- | **Answer** | A short grounded answer is generated from the story context and spoken by the low-latency Q&A audio model. It does not use the cloned narrator voice. | LFM2.5-Audio-1.5B |
47
- | **Resume** | Story continues from where it left off. Play also pauses on tab switch. | Cached chunks |
 
 
48
 
49
  ## Models
50
 
51
- | Model | Role | Size |
52
- |---|---|---|
53
- | [Qwen3-TTS-1.7B-Base](https://huggingface.co/Qwen/Qwen3-TTS-12Hz-1.7B-Base) | Zero-shot voice cloning + TTS | 1.7B params |
54
- | [Qwen3-TTS-0.6B-CustomVoice](https://huggingface.co/Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice) | Fast stock-voice TTS (predefined speakers) | 0.6B params |
55
- | [LiquidAI/LFM2.5-Audio-1.5B](https://huggingface.co/LiquidAI/LFM2.5-Audio-1.5B) | End-to-end story Q&A with audio input and spoken audio output | 1.5B params |
 
 
56
 
57
  The stock narration model and LFM Q&A model are preloaded at startup for fast response times. The 1.7B cloning model is loaded when cloning or cloned narration is needed.
58
 
@@ -67,27 +76,32 @@ The stock narration model and LFM Q&A model are preloaded at startup for fast re
67
 
68
  ## System Requirements
69
 
70
- | GPU | VRAM | Status | Notes |
71
- |---|---|---|---|
72
- | Nvidia A10G (24 GB) | 24 GB | βœ… Recommended | All three models load comfortably; narration + Q&A run without memory pressure |
73
- | Nvidia T4 (16 GB) | 16 GB | ⚠️ Functional | Works but is tight β€” voice cloning model may OOM if loaded concurrently with LFM; restart may be needed between heavy operations |
74
- | CPU only | β€” | ❌ Not viable | LFM2.5-Audio and Qwen3-TTS inference is too slow for real-time use |
 
 
75
 
76
  **Hugging Face Spaces:**
 
77
  - **A10G Small** β€” confirmed working, recommended for the hackathon demo
78
  - **T4 Small** β€” runs with limitations; recommend disabling concurrent model loading
79
 
80
- ![HuggingFace Space Hardware Options](docs/hf_spec.png)
81
 
82
  **Memory breakdown (approximate):**
83
 
84
- | Model | VRAM (bfloat16) |
85
- |---|---|
86
- | LFM2.5-Audio-1.5B | ~3 GB |
87
- | Qwen3-TTS-0.6B-CustomVoice | ~1.2 GB |
88
- | Qwen3-TTS-1.7B-Base (cloning) | ~3.4 GB |
89
- | KV cache + activations at inference | ~1–2 GB |
90
- | **Total (all models loaded)** | **~8–9 GB** |
 
 
91
 
92
  The 1.7B cloning model is loaded on demand (not at startup), so idle VRAM usage is ~5–6 GB with only the narration and Q&A models preloaded.
93
 
@@ -138,18 +152,20 @@ python app.py
138
 
139
  ## Stories Included
140
 
141
- | Story | Words | Source |
142
- |---|---|---|
143
- | The Tale of Peter Rabbit | 948 | Beatrix Potter |
144
- | The Tale of Benjamin Bunny | 1,118 | Beatrix Potter |
145
- | The Tale of Jemima Puddle-Duck | 1,245 | Beatrix Potter |
146
- | The Tale of Tom Kitten | 691 | Beatrix Potter |
147
- | The History of Tom Thumb | 2,912 | Traditional |
148
- | The Story of the Three Little Pigs | 956 | Traditional |
149
- | The Little Red Hen | 1,295 | Traditional |
150
- | The Little Gingerbread Man | 1,823 | Traditional |
151
- | The Sleeping Beauty | 1,783 | Traditional |
152
- | The Adventures of Puss in Boots | 503 | Traditional (verse) |
 
 
153
 
154
  All stories are public domain from [Project Gutenberg](https://www.gutenberg.org/).
155
 
@@ -172,7 +188,6 @@ All inference runs on the Hugging Face Space GPU. Voice samples, story text, and
172
  Hackathon project. Stories are public domain.
173
  test push verification
174
 
175
-
176
  ## Application Demo
177
 
178
  # MomsVoice β€” Demo Walkthrough
@@ -183,7 +198,7 @@ Step-by-step guide through the app with screenshots.
183
 
184
  ## Step 1: Explore the Story Shelf
185
 
186
- ![Explore Space](docs/demo01.png)
187
 
188
  Open the **Explore Space** tab to browse available stories displayed as warm bookshelf cards. Each card shows the cover art, title, author, and a short excerpt. Click **"Tap to Play"** on any card to open it in My Library, or click **"More Info"** for the full story description.
189
 
@@ -193,7 +208,7 @@ The featured story highlights the current pick. All 10 stories are public domain
193
 
194
  ## Step 2: Open My Library and Start Narration
195
 
196
- ![My Library β€” Now Playing](docs/demo02.png)
197
 
198
  Switch to the **My Library** tab. Click any story card to load it into the **Now Playing** panel on the right. The panel shows:
199
 
@@ -208,7 +223,7 @@ Narration starts immediately, playing one paragraph at a time using the TTS mode
208
 
209
  ## Step 3: Ask a Question About the Story
210
 
211
- ![Voice Q&A Panel](docs/demo03.png)
212
 
213
  Scroll down in the Library panel to find the **Ask About the Story** section. Click **❓ Ask a Question** β€” narration pauses automatically and the microphone starts recording.
214
 
@@ -218,11 +233,12 @@ Speak your question naturally (e.g. *"What did Puss in Boots do to help his mast
218
 
219
  ## Step 4: Hear the Spoken Answer (LFM2.5-Audio)
220
 
221
- ![LFM Answer Audio](docs/demo04.png)
222
 
223
  The recorded question waveform appears in the audio widget. Below it, the **LFM2.5-Audio** model generates a grounded answer in 1–2 sentences, based only on the current story context.
224
 
225
  The answer is delivered in two ways simultaneously:
 
226
  - **Answer text** shown inline (e.g. *"Puss in Boots, a poor man, bargained for boots and a bag, and Grimalkin, ailient rabbit, rescued him from a king's wrath."*)
227
  - **Answer Audio** waveform auto-plays the spoken response
228
 
@@ -232,9 +248,10 @@ After the answer finishes, narration resumes from where it paused.
232
 
233
  ## Step 5: Open Clone Voice Studio and Record Your Voice
234
 
235
- ![Clone Studio β€” Recording](docs/demo05_clone.png)
236
 
237
  Go to the **Clone Voice Studio** tab. Fill in:
 
238
  - **Voice Nickname** β€” a name for this profile (e.g. *"Mom"*, *"Grandpa Joe"*)
239
  - **Voice Gender Avatar** β€” Female or Male
240
 
@@ -244,7 +261,7 @@ A story script is displayed to read aloud. Click the **Record** button and read
244
 
245
  ## Step 6: Analyze & Synthesize the Vocal Profile
246
 
247
- ![Clone Studio β€” Processing](docs/demo06_clone.png)
248
 
249
  Click **Analyze & Synthesize Customized Vocal Profile**. The **Status Pipeline Visualizer** on the right tracks progress through extraction, de-noising, and pitch tuning stages.
250
 
@@ -254,9 +271,10 @@ A progress bar shows synthesis percentage (e.g. *"Generating voice preview... 70
254
 
255
  ## Step 7: Voice Profile Ready
256
 
257
- ![Clone Studio β€” Complete](docs/demo07_clone.png)
258
 
259
  When synthesis completes, the Status Pipeline Visualizer shows:
 
260
  - Your **avatar** and nickname with a **"Cloned successfully"** badge
261
  - *"Synthesized today using Qwen3-TTS voice cloning"*
262
  - A **preview audio player** β€” tap Play to hear a short sample in your cloned voice
@@ -268,8 +286,8 @@ The profile is saved to disk and will persist across Space restarts.
268
 
269
  ## Step 8: Story Narration in Your Cloned Voice
270
 
271
- ![Narration with Cloned Voice](docs/demo08_clone.png)
272
 
273
  Return to **My Library** and select a story. The Now Playing panel shows **Active Narrator: [your nickname]** β€” confirming the cloned voice is active.
274
 
275
- Narration plays in your recorded voice, paragraph by paragraph (*"Paragraph 1 / 32"*). The story text scrolls in the reader below the player. All TTS inference runs locally on the Space GPU β€” no audio leaves the server.
 
1
  ---
2
+
3
+ ## title: MomsVoice
4
  emoji: πŸ“–
5
  colorFrom: yellow
6
  colorTo: green
 
8
  sdk_version: 5.35.0
9
  app_file: app.py
10
  pinned: false
 
11
 
12
  # MomsVoice
13
 
 
15
 
16
  Built for the Hugging Face Hackathon. Runs entirely on local models inside a Gradio app on Hugging Face Spaces β€” no external APIs, no data leaves the server.
17
 
18
+ ## build-small-hackathon Link
19
+
20
+ πŸ”— [https://huggingface.co/spaces/build-small-hackathon/MomsVoiceAI](https://huggingface.co/spaces/build-small-hackathon/MomsVoiceAI)
21
 
22
+ πŸ”— **Live Demo at personal Spaces**
23
+ [https://huggingface.co/spaces/DrSquare/MomsVoice](https://huggingface.co/spaces/DrSquare/MomsVoice)
24
+ [https://huggingface.co/spaces/sshtel/Huggingface_Hack](https://huggingface.co/spaces/sshtel/Huggingface_Hack)
25
 
26
  ## Team members
27
+
28
  - Minha Hwang [Linkedin](https://www.linkedin.com/in/minha-hwang-7440771/) / [Huggingface](https://huggingface.co/DrSquare)
29
  - Jimin Gu [Linkedin](https://www.linkedin.com/in/99jimingu/) / [Huggingface](https://huggingface.co/Zoe09)
30
  - Seunghwa Song [Linkedin](https://www.linkedin.com/in/sshtel/) / [Huggingface](https://huggingface.co/sshtel)
31
 
 
32
  ## Demo Flow
33
 
34
  ```
 
43
 
44
  ## How It Works
45
 
46
+
47
+ | Step | What Happens | Model |
48
+ | ---------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------- |
49
+ | **Clone** | Parent records or uploads 15–30s of audio. Speaker embedding is extracted and cached. | Qwen3-TTS-1.7B Base |
50
+ | **Listen** | Story plays in the cloned voice as interruptible sentence chunks. Falls back to CustomVoice 0.6B stock voice (speaker "vivian") if no clone. | Qwen3-TTS-1.7B / CustomVoice 0.6B |
51
+ | **Ask** | Child taps Ask β†’ narration pauses, microphone auto-starts. When recording stops, the question is sent directly to the Q&A audio model. | LFM2.5-Audio-1.5B |
52
+ | **Answer** | A short grounded answer is generated from the story context and spoken by the low-latency Q&A audio model. It does not use the cloned narrator voice. | LFM2.5-Audio-1.5B |
53
+ | **Resume** | Story continues from where it left off. Play also pauses on tab switch. | Cached chunks |
54
+
55
 
56
  ## Models
57
 
58
+
59
+ | Model | Role | Size |
60
+ | ----------------------------------------------------------------------------------------- | ------------------------------------------------------------- | ----------- |
61
+ | [Qwen3-TTS-1.7B-Base](https://huggingface.co/Qwen/Qwen3-TTS-12Hz-1.7B-Base) | Zero-shot voice cloning + TTS | 1.7B params |
62
+ | [Qwen3-TTS-0.6B-CustomVoice](https://huggingface.co/Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice) | Fast stock-voice TTS (predefined speakers) | 0.6B params |
63
+ | [LiquidAI/LFM2.5-Audio-1.5B](https://huggingface.co/LiquidAI/LFM2.5-Audio-1.5B) | End-to-end story Q&A with audio input and spoken audio output | 1.5B params |
64
+
65
 
66
  The stock narration model and LFM Q&A model are preloaded at startup for fast response times. The 1.7B cloning model is loaded when cloning or cloned narration is needed.
67
 
 
76
 
77
  ## System Requirements
78
 
79
+
80
+ | GPU | VRAM | Status | Notes |
81
+ | ------------------- | ----- | ------------- | -------------------------------------------------------------------------------------------------------------------------------- |
82
+ | Nvidia A10G (24 GB) | 24 GB | βœ… Recommended | All three models load comfortably; narration + Q&A run without memory pressure |
83
+ | Nvidia T4 (16 GB) | 16 GB | ⚠️ Functional | Works but is tight β€” voice cloning model may OOM if loaded concurrently with LFM; restart may be needed between heavy operations |
84
+ | CPU only | β€” | ❌ Not viable | LFM2.5-Audio and Qwen3-TTS inference is too slow for real-time use |
85
+
86
 
87
  **Hugging Face Spaces:**
88
+
89
  - **A10G Small** β€” confirmed working, recommended for the hackathon demo
90
  - **T4 Small** β€” runs with limitations; recommend disabling concurrent model loading
91
 
92
+ HuggingFace Space Hardware Options
93
 
94
  **Memory breakdown (approximate):**
95
 
96
+
97
+ | Model | VRAM (bfloat16) |
98
+ | ----------------------------------- | --------------- |
99
+ | LFM2.5-Audio-1.5B | ~3 GB |
100
+ | Qwen3-TTS-0.6B-CustomVoice | ~1.2 GB |
101
+ | Qwen3-TTS-1.7B-Base (cloning) | ~3.4 GB |
102
+ | KV cache + activations at inference | ~1–2 GB |
103
+ | **Total (all models loaded)** | **~8–9 GB** |
104
+
105
 
106
  The 1.7B cloning model is loaded on demand (not at startup), so idle VRAM usage is ~5–6 GB with only the narration and Q&A models preloaded.
107
 
 
152
 
153
  ## Stories Included
154
 
155
+
156
+ | Story | Words | Source |
157
+ | ---------------------------------- | ----- | ------------------- |
158
+ | The Tale of Peter Rabbit | 948 | Beatrix Potter |
159
+ | The Tale of Benjamin Bunny | 1,118 | Beatrix Potter |
160
+ | The Tale of Jemima Puddle-Duck | 1,245 | Beatrix Potter |
161
+ | The Tale of Tom Kitten | 691 | Beatrix Potter |
162
+ | The History of Tom Thumb | 2,912 | Traditional |
163
+ | The Story of the Three Little Pigs | 956 | Traditional |
164
+ | The Little Red Hen | 1,295 | Traditional |
165
+ | The Little Gingerbread Man | 1,823 | Traditional |
166
+ | The Sleeping Beauty | 1,783 | Traditional |
167
+ | The Adventures of Puss in Boots | 503 | Traditional (verse) |
168
+
169
 
170
  All stories are public domain from [Project Gutenberg](https://www.gutenberg.org/).
171
 
 
188
  Hackathon project. Stories are public domain.
189
  test push verification
190
 
 
191
  ## Application Demo
192
 
193
  # MomsVoice β€” Demo Walkthrough
 
198
 
199
  ## Step 1: Explore the Story Shelf
200
 
201
+ Explore Space
202
 
203
  Open the **Explore Space** tab to browse available stories displayed as warm bookshelf cards. Each card shows the cover art, title, author, and a short excerpt. Click **"Tap to Play"** on any card to open it in My Library, or click **"More Info"** for the full story description.
204
 
 
208
 
209
  ## Step 2: Open My Library and Start Narration
210
 
211
+ My Library β€” Now Playing
212
 
213
  Switch to the **My Library** tab. Click any story card to load it into the **Now Playing** panel on the right. The panel shows:
214
 
 
223
 
224
  ## Step 3: Ask a Question About the Story
225
 
226
+ Voice Q&A Panel
227
 
228
  Scroll down in the Library panel to find the **Ask About the Story** section. Click **❓ Ask a Question** β€” narration pauses automatically and the microphone starts recording.
229
 
 
233
 
234
  ## Step 4: Hear the Spoken Answer (LFM2.5-Audio)
235
 
236
+ LFM Answer Audio
237
 
238
  The recorded question waveform appears in the audio widget. Below it, the **LFM2.5-Audio** model generates a grounded answer in 1–2 sentences, based only on the current story context.
239
 
240
  The answer is delivered in two ways simultaneously:
241
+
242
  - **Answer text** shown inline (e.g. *"Puss in Boots, a poor man, bargained for boots and a bag, and Grimalkin, ailient rabbit, rescued him from a king's wrath."*)
243
  - **Answer Audio** waveform auto-plays the spoken response
244
 
 
248
 
249
  ## Step 5: Open Clone Voice Studio and Record Your Voice
250
 
251
+ Clone Studio β€” Recording
252
 
253
  Go to the **Clone Voice Studio** tab. Fill in:
254
+
255
  - **Voice Nickname** β€” a name for this profile (e.g. *"Mom"*, *"Grandpa Joe"*)
256
  - **Voice Gender Avatar** β€” Female or Male
257
 
 
261
 
262
  ## Step 6: Analyze & Synthesize the Vocal Profile
263
 
264
+ Clone Studio β€” Processing
265
 
266
  Click **Analyze & Synthesize Customized Vocal Profile**. The **Status Pipeline Visualizer** on the right tracks progress through extraction, de-noising, and pitch tuning stages.
267
 
 
271
 
272
  ## Step 7: Voice Profile Ready
273
 
274
+ Clone Studio β€” Complete
275
 
276
  When synthesis completes, the Status Pipeline Visualizer shows:
277
+
278
  - Your **avatar** and nickname with a **"Cloned successfully"** badge
279
  - *"Synthesized today using Qwen3-TTS voice cloning"*
280
  - A **preview audio player** β€” tap Play to hear a short sample in your cloned voice
 
286
 
287
  ## Step 8: Story Narration in Your Cloned Voice
288
 
289
+ Narration with Cloned Voice
290
 
291
  Return to **My Library** and select a story. The Now Playing panel shows **Active Narrator: [your nickname]** β€” confirming the cloned voice is active.
292
 
293
+ Narration plays in your recorded voice, paragraph by paragraph (*"Paragraph 1 / 32"*). The story text scrolls in the reader below the player. All TTS inference runs locally on the Space GPU β€” no audio leaves the server.