tech_stack.md

# Tech Stack

## Architecture (Hackathon-Simple)

Single Python file Gradio app. Everything runs in one process on a GPU-enabled HF Space.

```
┌─────────────────────────────────────────────────┐
│  Hugging Face Space (GPU: T4 or A10G)           │
│                                                 │
│  app.py (Gradio + gr.Server)                    │
│    ├── UI: Stitch-styled tabs/cards             │
│    ├── QWEN-TTS-0.6B (voice clone + chunks)     │
│    ├── Whisper-small (ASR for child questions)  │
│    └── Qwen2.5-3B-Instruct (story Q&A)          │
│                                                 │
│  stories/ (3–5 .txt files, public domain)       │
└─────────────────────────────────────────────────┘
```

No database. No external storage. No external LLM API. Stories are flat files. Audio is generated as interruptible chunks, then cached in the session for replay and resume.

**VRAM Budget (T4 — 16 GB):**

| Component | Estimated VRAM | Notes |
|---|---|---|
| QWEN-TTS-0.6B | ~1.2 GB | Always loaded |
| Qwen2.5-3B-Instruct (4-bit) | ~2 GB | Always loaded |
| Whisper-small | ~1 GB | Loaded on demand |
| Gradio + PyTorch overhead | ~1–2 GB | Runtime |
| **Total** | **~5–6 GB** | ~10 GB headroom for KV cache and activations |

**Concurrency:** Single-process Gradio serializes concurrent users. The hackathon demo is single-user. For multi-user, consider Gradio queue or separate worker processes.

---

## 1. Front-End

| Choice | Why |
|---|---|
| Gradio 5.x | Zero frontend code, instant HF Space deploy |
| `gr.Server` | Custom CSS/JS for Stitch-style polish (animations, palette, layout) |
| `gr.Audio` | Record/upload parent voice sample |
| `gr.Dropdown` / `gr.Gallery` | Story selection with cover art |
| `gr.Audio` (output) | Playback of streamed story chunks / Q&A answer |
| `gr.Textbox` + `gr.Audio` (input) | Child question via text or voice |
| Play/Pause/Ask buttons | Manual interruption and resume without open-mic barge-in |

**UI Tabs:**
1. **🎤 Clone Voice** — record/upload 15–30s, preview clone
2. **📖 Listen** — pick story, hear streamed chunks in cloned voice
3. **❓ Ask** — pause narration, ask about the story, hear answer, resume

---

## 2. Voice Model (QWEN-TTS-0.6B)

| Aspect | Detail |
|---|---|
| Model | `Qwen/Qwen-TTS-0.6B` from Hugging Face Hub |
| Size | 0.6B params — fits comfortably on T4 (16GB VRAM) |
| Capability | Zero-shot voice cloning + TTS from text chunks |
| Input | Reference audio (≥5s) + target text |
| Output | WAV audio in cloned voice (24 kHz, 16-bit, mono) |
| Latency | ~3–5s for a paragraph on T4 |
| Optimization | Cache the voice representation after recording; generate story audio in interruptible paragraph chunks |

---

## 3. ASR (Child Voice Input)

| Choice | Detail |
|---|---|
| Model | Whisper-small checkpoint via Transformers (local, 244M params) |
| Why | Fast, accurate for short child utterances; fits in GPU alongside TTS |
| Load strategy | Load only when the Ask tab receives audio; text questions bypass ASR |
| Fallback | Whisper-tiny/base or browser transcription if GPU memory or latency is tight |

---

## 4. Q&A (Story Comprehension)

| Choice | Detail |
|---|---|
| Model | `Qwen/Qwen2.5-3B-Instruct` |
| Why | Strong small-model instruction following with lower latency and VRAM pressure than an 8B-class model |
| Method | Current story position + relevant story passages + strict answer-from-story instruction + child question → short answer |
| Retrieval | TF-IDF cosine similarity between the child's question and each story paragraph; return the top-2 paragraphs as context. Full-story prompt is the fallback when retrieval scores are low. |
| Output cap | 1–2 child-friendly sentences, typically 40–80 new tokens |
| Runtime note | Use 4-bit/8-bit loading on T4; use bf16 or 8-bit on A10G for more headroom |

---

## 5. Stories (Content)

10 public domain children's stories stored as `.txt` in `stories/`, sourced from [Project Gutenberg](https://www.gutenberg.org/) via the `story_downloader/` pipeline:

| Story | Words | Author/Tradition |
|---|---|---|
| The Tale of Peter Rabbit | 948 | Beatrix Potter |
| The Tale of Benjamin Bunny | 1,118 | Beatrix Potter |
| The Tale of Jemima Puddle-Duck | 1,245 | Beatrix Potter |
| The Tale of Tom Kitten | 691 | Beatrix Potter |
| The History of Tom Thumb | 2,912 | Traditional |
| The Story of the Three Little Pigs | 956 | Traditional |
| The Little Red Hen | 1,295 | Traditional |
| The Little Gingerbread Man | 1,823 | Traditional |
| The Sleeping Beauty | 1,783 | Traditional |
| The Adventures of Puss in Boots | 503 | Traditional (verse) |

Each file: title on line 1, blank line, then story prose — ready for direct TTS chunking. No metadata DB needed.

**Story Pipeline (`story_downloader/`):**
- `gutenberg_downloader.py` — reusable downloader/parser for Project Gutenberg texts
- `download_stories.py` — fetches 10 specific children's stories by Gutenberg ID
- `clean_stories.py` — strips Gutenberg headers/footers, illustration tags, and metadata for TTS-clean output

---

## 6. Deployment

| What | How |
|---|---|
| Platform | Hugging Face Spaces |
| SDK | Gradio |
| Hardware | T4 with quantized Qwen for the budget path; A10G for lower risk live demos |
| Deploy | `git push` to HF Space repo |
| Secrets | None for LLM inference; `HF_TOKEN` only if any selected model requires gated access |
| Domain | `huggingface.co/spaces/{user}/readbookmom` |

---

## 7. Latency Plan

| Flow | Target | Implementation |
|---|---|---|
| Voice setup | One-time after recording | Compute and cache the voice representation before story generation. |
| Story narration start | First streamed chunk in ≤ 5s | Split the story into paragraph chunks; synthesize and play the first chunk first. |
| Narration interruption | Pause in ≤ 500ms after Ask tap | Stop playback, preserve current chunk index, and cancel or deprioritize queued narration jobs. |
| Q&A interruption loop | Spoken answer starts in ≤ 8s | Use current story position, retrieve relevant passages, cap answer length, then synthesize the final answer. |
| Story resume | ≤ 1s when next chunk is cached | Resume from the paused chunk or the next queued chunk after the answer finishes. |
| Story replay | Immediate after first generation | Cache generated audio by voice session and story ID. |
| Child audio transcription | 1–2s target | Load ASR only for audio questions; prefer lighter ASR fallback for demo mode. |
| Q&A text answer | 1–3s target | Send only relevant story passages to Qwen and cap output tokens. |
| Spoken Q&A answer | ≤ 8s total target | Synthesize only the final short answer, not intermediate reasoning or context. |

## 8. Interaction State

| State | Meaning | Key Data |
|---|---|---|
| `playing` | Story chunk is currently playing. | `story_id`, `voice_session_id`, `current_chunk_index` |
| `paused` | Playback is paused by user action. | Current chunk, elapsed position if available |
| `asking` | Narration is interrupted while the child asks a question. | Current chunk, relevant passages, pending ASR input |
| `answering` | Qwen answer or answer TTS is being generated. | Question text, short answer, answer audio path |
| `resuming` | Answer finished and story playback is restarting. | Resume chunk index, cached next chunk |
| `finished` | Story narration completed. | Cached full-story audio |

**Legal transitions:**

```
playing → paused → playing
playing → asking → answering → resuming → playing
playing → finished
paused  → asking  → answering → resuming → playing
asking  → asking  (child asks a follow-up before answer starts)
```

All other transitions are illegal. The UI should disable buttons that would trigger an illegal transition.

---

## 9. Local Dev

```bash
pip install gradio transformers torch accelerate bitsandbytes soundfile numpy
python app.py
# → http://localhost:7860
```

No Docker, no DB, no infra setup.

---

## 10. Dependencies

```
gradio>=5.0
transformers
torch
accelerate
bitsandbytes
soundfile
numpy
```

---

## 11. Google Stitch UI Customization (via gr.Server)

`gr.Server` injects custom HTML/CSS/JS to achieve Stitch-quality polish:

- **Custom CSS**: Rounded cards, warm color palette (#FFB347 accent, #FFF8E7 background), playful fonts (Nunito/Fredoka)
- **Micro-animations**: Fade-in on story cards, pulse on recording button, waveform visualization
- **Layout overrides**: Full-bleed hero on clone tab, grid gallery for stories
- **Custom favicon + title**: Branded for demo presentation

All in a `static/` folder loaded by gr.Server mount.

---

## 12. Review Notes

| Area | Critique | Upgrade |
|---|---|---|
| Privacy | Using an external Q&A API would undermine the local-inference claim. | Qwen2.5-3B-Instruct keeps questions, story text, and generated answers inside the Space runtime. |
| GPU fit | QWEN-TTS-0.6B, Whisper-small, and a 3B-class LLM are a more realistic fit for a single Space than an 8B-class LLM, but running all three hot can still pressure T4. | Quantize Qwen on T4, use A10G for demo headroom, cache voice/story artifacts, and load ASR only when the Ask tab receives audio. |
| Latency | Full-story generation and spoken Q&A can feel slow if every step waits for complete outputs. | Use interruptible paragraph chunks, cache full narration, retrieve only relevant passages, and synthesize only short final answers. |
| Interaction | Streaming without cancellation can still feel rigid if the child must wait for a chunk to finish. | Add explicit playback state, Ask interruption, queued job cancellation/deprioritization, and resume from the saved chunk. |
| Dependencies | External LLM SDKs and API secrets are no longer aligned with the model choice. | Use local inference dependencies and optional HF authentication only. |