minhahwang Copilot commited on
Commit
3cfcf98
Β·
1 Parent(s): 57efb6d

docs: update README - reflect Qwen3-TTS, voice-first Q&A, all current features

Browse files
Files changed (1) hide show
  1. README.md +46 -29
README.md CHANGED
@@ -9,49 +9,64 @@ app_file: app.py
9
  pinned: false
10
  ---
11
 
12
- # ReadBookMom
13
 
14
  **A parent records their voice. A bedtime story plays in that voice. The child can ask questions and hear answers β€” all in mom or dad's voice.**
15
 
16
  Built for the Hugging Face Hackathon. Runs entirely on local models inside a Gradio app on Hugging Face Spaces β€” no external APIs, no data leaves the server.
17
 
18
- ## Demo
 
 
19
 
20
  ```
21
  🎀 Record 15s of your voice β†’ πŸ“– Pick a story β†’ ▢️ Story plays in your voice
22
  ↓
23
- ❓ Child taps Ask β†’ Story pauses
24
  ↓
25
- Child asks a question β†’ Hears answer in your voice
26
  ↓
27
- ▢️ Story resumes
28
  ```
29
 
30
  ## How It Works
31
 
32
  | Step | What Happens | Model |
33
  |---|---|---|
34
- | **Clone** | Parent records or uploads 15–30s of audio. Speaker embedding is extracted and cached server-side. | Qwen3-TTS-1.7B |
35
- | **Listen** | Story plays in the cloned voice as interruptible sentence chunks. Falls back to Supertonic stock voice if no clone. | Qwen3-TTS-1.7B / Supertonic |
36
- | **Ask** | Child taps Ask, narration pauses, child types or speaks a question. | Whisper-small (ASR, on-demand) |
37
- | **Answer** | A short grounded answer is generated from the story context and spoken in the cloned voice. | Qwen2.5-3B-Instruct + Qwen3-TTS-1.7B |
38
- | **Resume** | Story continues from where it left off. | Cached chunks |
39
 
40
  ## Models
41
 
42
  | Model | Role | Size |
43
  |---|---|---|
44
  | [Qwen3-TTS-1.7B-Base](https://huggingface.co/Qwen/Qwen3-TTS-12Hz-1.7B-Base) | Zero-shot voice cloning + TTS | 1.7B params |
45
- | [Supertonic TTS](https://huggingface.co/niclas-pham/supertonic-3) | Fast stock-voice TTS fallback | ONNX model |
46
- | [Qwen2.5-3B-Instruct](https://huggingface.co/Qwen/Qwen2.5-3B-Instruct) | Story Q&A | 3B params (4-bit on T4) |
47
- | [Whisper-small](https://huggingface.co/openai/whisper-small) | Child speech-to-text | 244M params (loaded on demand) |
 
 
48
 
49
  ## Tech Stack
50
 
51
- - **UI:** Gradio 5.x + gr.Server (custom CSS/JS for Google Stitch-inspired design)
52
- - **Runtime:** Single `app.py` process on HF Spaces (T4 or A10G GPU)
53
  - **Stories:** 10 public domain `.txt` files (downloaded and cleaned from Project Gutenberg)
54
- - **Storage:** In-memory session cache (no database)
 
 
 
 
 
 
 
 
 
 
 
55
 
56
  ## Quick Start (Local Dev)
57
 
@@ -66,22 +81,23 @@ python app.py
66
  ## Project Structure
67
 
68
  ```
69
- β”œβ”€β”€ app.py # Main Gradio app (UI + wiring)
70
- β”œβ”€β”€ voice_clone.py # Qwen3-TTS voice cloning + profile cache
71
- β”œβ”€β”€ tts.py # Unified TTS interface (Qwen3 or Supertonic)
72
- β”œβ”€β”€ inference.py # ASR (Whisper) + Q&A (Qwen2.5-3B-Instruct)
73
- β”œβ”€β”€ requirements.txt # Python deps
74
  β”œβ”€β”€ stories/ # 10 cleaned public domain story texts (TTS-ready)
75
  β”œβ”€β”€ story_downloader/ # Story acquisition & cleaning pipeline
76
  β”‚ β”œβ”€β”€ gutenberg_downloader.py # Reusable Project Gutenberg downloader/parser
77
  β”‚ β”œβ”€β”€ download_stories.py # Downloads 10 children's stories
78
  β”‚ └── clean_stories.py # Strips Gutenberg boilerplate for TTS
79
- β”œβ”€β”€ static/ # Custom CSS/JS for Stitch-style UI
80
  β”œβ”€β”€ assets/covers/ # Story cover images
 
81
  β”œβ”€β”€ test_modules/ # Component and integration tests
82
  β”œβ”€β”€ mission.md # Product vision
83
  β”œβ”€β”€ sprint.md # 2-day hackathon sprint plan
84
- β”œβ”€β”€ tech_stack.md # Technical architecture
85
  └── future_mobile_app_considerations.md # Mobile deployment guidance
86
  ```
87
 
@@ -100,19 +116,20 @@ python app.py
100
  | The Sleeping Beauty | 1,783 | Traditional |
101
  | The Adventures of Puss in Boots | 503 | Traditional (verse) |
102
 
103
- All stories are public domain from [Project Gutenberg](https://www.gutenberg.org/). Each file uses a simple format: title on line 1, blank line, then story prose β€” ready for direct TTS consumption.
104
 
105
  ## Key Design Decisions
106
 
107
  - **All local inference** β€” voice, Q&A, and ASR run on the Space GPU. No external APIs.
108
- - **Interruptible chunked streaming** β€” paragraphs synthesized and played one at a time for fast start and clean pause/resume.
109
- - **Pre-generated Q&A** β€” anticipated questions are generated in the background during narration for sub-1s response on cache hits.
110
- - **Button-based interruption** β€” tap Ask to pause. No always-listening mic (privacy + complexity).
111
- - **Privacy-first** β€” no audio leaves the server, no user accounts, no database.
 
112
 
113
  ## Privacy
114
 
115
- All inference runs on the Hugging Face Space GPU. Voice samples, story text, and generated audio stay on the server runtime and are not persisted after the session ends. No external APIs are called.
116
 
117
  ## License
118
 
 
9
  pinned: false
10
  ---
11
 
12
+ # MomsVoice
13
 
14
  **A parent records their voice. A bedtime story plays in that voice. The child can ask questions and hear answers β€” all in mom or dad's voice.**
15
 
16
  Built for the Hugging Face Hackathon. Runs entirely on local models inside a Gradio app on Hugging Face Spaces β€” no external APIs, no data leaves the server.
17
 
18
+ πŸ”— **Live Demo:** [https://huggingface.co/spaces/DrSquare/MomsVoice](https://huggingface.co/spaces/DrSquare/MomsVoice)
19
+
20
+ ## Demo Flow
21
 
22
  ```
23
  🎀 Record 15s of your voice β†’ πŸ“– Pick a story β†’ ▢️ Story plays in your voice
24
  ↓
25
+ ❓ Child taps Ask β†’ Story pauses, mic auto-starts
26
  ↓
27
+ Child asks a question β†’ Answer auto-plays in narrator's voice
28
  ↓
29
+ ↩️ Resume β†’ Story continues from where it paused
30
  ```
31
 
32
  ## How It Works
33
 
34
  | Step | What Happens | Model |
35
  |---|---|---|
36
+ | **Clone** | Parent records or uploads 15–30s of audio. Speaker embedding is extracted and cached. | Qwen3-TTS-1.7B Base |
37
+ | **Listen** | Story plays in the cloned voice as interruptible sentence chunks. Falls back to CustomVoice 0.6B stock voice (speaker "vivian") if no clone. | Qwen3-TTS-1.7B / CustomVoice 0.6B |
38
+ | **Ask** | Child taps Ask β†’ narration pauses, microphone auto-starts. When recording stops, question is transcribed and answer auto-generates. | Whisper-small (ASR) |
39
+ | **Answer** | A short grounded answer is generated from the full story context and spoken in the narrator's voice. Auto-plays immediately. | Qwen2.5-3B-Instruct + Qwen3-TTS |
40
+ | **Resume** | Story continues from where it left off. Play also pauses on tab switch. | Cached chunks |
41
 
42
  ## Models
43
 
44
  | Model | Role | Size |
45
  |---|---|---|
46
  | [Qwen3-TTS-1.7B-Base](https://huggingface.co/Qwen/Qwen3-TTS-12Hz-1.7B-Base) | Zero-shot voice cloning + TTS | 1.7B params |
47
+ | [Qwen3-TTS-0.6B-CustomVoice](https://huggingface.co/Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice) | Fast stock-voice TTS (predefined speakers) | 0.6B params |
48
+ | [Qwen2.5-3B-Instruct](https://huggingface.co/Qwen/Qwen2.5-3B-Instruct) | Story Q&A (context-grounded answers) | 3B params |
49
+ | [Whisper-small](https://huggingface.co/openai/whisper-small) | Child speech-to-text | 244M params |
50
+
51
+ All models are preloaded at startup for fast response times.
52
 
53
  ## Tech Stack
54
 
55
+ - **UI:** Gradio 5.35.0 with custom CSS/JS (Google Stitch-inspired warm design)
56
+ - **Runtime:** Single `app.py` process on HF Spaces (GPU: A10G or larger recommended)
57
  - **Stories:** 10 public domain `.txt` files (downloaded and cleaned from Project Gutenberg)
58
+ - **Storage:** In-memory session cache + disk-persisted voice profiles (`Voice_Profile/`)
59
+ - **TTS Backends:** Qwen3-TTS Base (cloned voice) + CustomVoice 0.6B (stock fallback)
60
+
61
+ ## Features
62
+
63
+ - 🎀 **Voice Cloning** β€” Record 15s, get a narrator that sounds like you
64
+ - πŸ“š **Explore & Library** β€” Browse and select stories with clickable book cards
65
+ - ▢️ **Chunked Streaming** β€” Stories play paragraph-by-paragraph with real-time highlighting
66
+ - ❓ **Voice Q&A** β€” Ask questions about the story; auto-submit on recording stop, autoplay answer
67
+ - ⏸ **Smart Pause** β€” Playback pauses on Ask, tab switch (Explore/Clone Studio), or Pause button
68
+ - πŸŽ™οΈ **Voice Selection** β€” Click voice cards to switch between cloned narrators
69
+ - πŸ’Ύ **Persistent Profiles** β€” Cloned voices survive app restarts
70
 
71
  ## Quick Start (Local Dev)
72
 
 
81
  ## Project Structure
82
 
83
  ```
84
+ β”œβ”€β”€ app.py # Main Gradio app (UI + all event wiring)
85
+ β”œβ”€β”€ voice_clone.py # Qwen3-TTS voice cloning + profile persistence
86
+ β”œβ”€β”€ tts.py # Unified TTS interface (Base cloned + CustomVoice stock)
87
+ β”œβ”€β”€ inference.py # ASR (Whisper-small) + Q&A (Qwen2.5-3B-Instruct)
88
+ β”œβ”€β”€ requirements.txt # Python deps (gradio, qwen-tts, transformers, torch)
89
  β”œβ”€β”€ stories/ # 10 cleaned public domain story texts (TTS-ready)
90
  β”œβ”€β”€ story_downloader/ # Story acquisition & cleaning pipeline
91
  β”‚ β”œβ”€β”€ gutenberg_downloader.py # Reusable Project Gutenberg downloader/parser
92
  β”‚ β”œβ”€β”€ download_stories.py # Downloads 10 children's stories
93
  β”‚ └── clean_stories.py # Strips Gutenberg boilerplate for TTS
94
+ β”œβ”€β”€ static/style.css # Custom CSS for warm storybook UI
95
  β”œβ”€β”€ assets/covers/ # Story cover images
96
+ β”œβ”€β”€ Voice_Profile/ # Persisted cloned voice profiles
97
  β”œβ”€β”€ test_modules/ # Component and integration tests
98
  β”œβ”€β”€ mission.md # Product vision
99
  β”œβ”€β”€ sprint.md # 2-day hackathon sprint plan
100
+ β”œβ”€β”€ tech_stack.md # Technical architecture
101
  └── future_mobile_app_considerations.md # Mobile deployment guidance
102
  ```
103
 
 
116
  | The Sleeping Beauty | 1,783 | Traditional |
117
  | The Adventures of Puss in Boots | 503 | Traditional (verse) |
118
 
119
+ All stories are public domain from [Project Gutenberg](https://www.gutenberg.org/).
120
 
121
  ## Key Design Decisions
122
 
123
  - **All local inference** β€” voice, Q&A, and ASR run on the Space GPU. No external APIs.
124
+ - **Interruptible chunked streaming** β€” sentences synthesized and played one at a time for fast start and clean pause/resume.
125
+ - **Voice-first Q&A** β€” microphone auto-starts on Ask; answer auto-plays when ready. No manual button clicks needed.
126
+ - **Full story context** β€” Q&A uses the entire story (or top-5 relevant paragraphs for long stories) plus current position for grounded answers.
127
+ - **Smart pause** β€” playback pauses on Ask, tab switch, or Pause button. Resume continues from exact position.
128
+ - **Privacy-first** β€” no audio leaves the server, no user accounts, no external database.
129
 
130
  ## Privacy
131
 
132
+ All inference runs on the Hugging Face Space GPU. Voice samples, story text, and generated audio stay on the server runtime. Voice profiles are saved to the Space's persistent storage but no data is sent to external services.
133
 
134
  ## License
135