Spaces:
Sleeping
Sleeping
| # Read-Along AI: API & Backend Contract Specification | |
| ## 1. Architecture Objective | |
| This application utilizes a decoupled architecture. The Gradio frontend must remain completely agnostic to the underlying inference engine. | |
| The application uses a **Dual-Mode Hybrid Architecture**. The Gradio UI events must pass through an abstraction layer (wrapper functions) in `app.py` that routes to either Modal endpoints (Turbo Mode) or local HF Space inference (Off the Grid Mode). | |
| ## 2. The Abstraction Layer & Routing | |
| The Gradio UI events should call wrapper functions in `app.py`, never Modal endpoints directly. A UI toggle determines whether these wrappers route payloads to `modal_inference.py` or `local_inference.py`. | |
| * `def transcribe_audio(audio_filepath: str) -> str:` | |
| * **Input:** The local file path to the `.wav` file generated by the Gradio microphone component. | |
| * **Output:** A clean, lower-cased string of the transcribed text. | |
| * `def synthesize_speech(target_text: str) -> str:` | |
| * **Input:** The string of text to be spoken. | |
| * **Output:** The local file path to the generated `.wav` file to be played by the Gradio UI. | |
| * `def ask_minicpm_judge(target_text: str, transcript: str) -> bool:` | |
| * **Input:** The displayed target sentence and ASR transcript. | |
| * **Output:** `True` when the transcript is an acceptable phonetic match, otherwise `False`. | |
| ## 3. Modal Endpoint Contracts (Phase 1 Backend) | |
| Codex should write the Modal stub functions that the wrappers above will call. We will use Modal's `@app.function()` decorator for direct RPC calls rather than setting up web webhooks to reduce latency. | |
| ### Endpoint A: Speech-to-Text (Cohere Transcribe) | |
| * **Modal Function Name:** `run_cohere_asr` | |
| * **Payload In:** `audio_bytes` (bytes) - The Gradio wrapper must read the `.wav` file into bytes before passing it to Modal to avoid file-path resolution errors across the cloud boundary. | |
| * **Payload Out:** `dict` | |
| * Schema: `{"text": "the dog ran fast", "status": "success"}` | |
| ### Endpoint B: Text-to-Speech (OpenBMB VoxCPM) | |
| * **Modal Function Name:** `run_voxcpm_tts` | |
| * **Payload In:** `text` (str) - The target sentence or word. | |
| * **Payload Out:** `bytes` - The raw audio buffer of the generated speech. The Gradio wrapper is responsible for catching these bytes, writing them to a temporary `.wav` file, and passing the path back to the UI. | |
| ### Endpoint C: Phonetic Evaluator (Fine-Tuned MiniCPM) | |
| * **Modal Function Name:** `run_minicpm_evaluator` | |
| * **Model:** `kingkw1/minicpm-phonetic-evaluator` | |
| * **Payload In:** target sentence or target text (str), `transcript` (str) | |
| * **Payload Out:** `str` | |
| * Schema: `"True"` or `"False"` | |
| * **Behavior:** The endpoint loads the fine-tuned MiniCPM model with `trust_remote_code=True`, formats the prompt using the same instruction/input/output structure used during training, and returns a binary verdict for whether the ASR transcript is a valid phonetic match for the target sentence. | |
| ## 3.5 Local Endpoint Contracts (Off the Grid Backend) | |
| Codex should write equivalent functions in `local_inference.py` to mirror the inputs/outputs above. | |
| * **ASR:** Implement `faster-whisper` using the `tiny.en` model. | |
| * **TTS / Audio Help:** Default Off the Grid audio help should use committed curriculum WAVs in `data/curriculum_audio/` and prewarm word clips by slicing the matching label/timing files. Live local VoxCPM remains an optional fallback only when `LOCAL_LIVE_TTS=1` is set. | |
| * **Evaluator:** Use `llama-cpp-python` to load the `minicpm-phonetic-evaluator-Q4_K_M.gguf` file. | |
| ## 4. Error Handling & Fallbacks | |
| Young users cannot parse stack traces. | |
| * If the Modal ASR endpoint times out or fails, the `transcribe_audio` wrapper must catch the exception and return a specific string: `"[ASR_ERROR]"`. The Gradio UI must handle this silently by asking the user to "Try pressing record again!" | |
| * If the TTS endpoint fails, `synthesize_speech` must return `None`, and the UI should gracefully fail open (no audio plays, but the UI does not freeze). | |
| * If the MiniCPM evaluator fails, the wrapper should fail closed and return `False` so incorrect readings are not accidentally marked as successful. | |
| * Word-click assistance should not block the UI. The current app prewarms word clips from committed curriculum label timings in Off the Grid Mode, pre-generates from sentence TTS in Turbo Mode where possible, and falls back to browser speech synthesis when cached audio is unavailable. | |
| ## 5. Testing Strategy | |
| As explicitly defined in Phase 1 (Verification Checkpoint 2), the backend API contract must be strictly isolated for testing: | |
| * Backend and app contract tests live under `tests/`. | |
| * Local unit tests should verify the Gradio-facing contracts without calling deployed services by default. | |
| * Modal endpoint checks should live as explicit integration tests and run only when the developer opts in, since they call deployed infrastructure. | |