# RISE — integrated Colab backend + Hugging Face frontend This package replaces the earlier fragile adapter that depended on variables already loaded in a notebook. The API is self-contained: it loads the models in the API process, and it does **not** call `output.eval_js`, `IPython.display.Audio`, or `run_reading_session()`. ## What is integrated - **Text generation:** `Qwen/Qwen2.5-1.5B-Instruct`; if it cannot load, the backend tries `google/flan-t5-base`. A small reviewed fallback passage is used if generation fails for a particular passage. - **Speech recognition:** `openai/whisper-small`, receiving browser-recorded audio converted by `ffmpeg` to mono 16 kHz audio. - **Pronunciation feedback:** fuzzy word matching and gTTS slow-English pronunciation audio for hesitated, mismatched, or skipped words. - **Progression:** 5 levels × sentence → paragraph → story; score is 50% word accuracy + 30% pronunciation similarity + 20% completion. Pass threshold is 80%, matching the first code in `Prompt.txt`. A failed attempt retries the same generated passage; passing advances the curriculum. - **Hand gestures:** 1–5 fingers jump to levels 1–5; thumbs up restarts at Level 1 / sentence; thumbs down ends the session. Browser camera frames are sent to the API for MediaPipe processing. - **Generation evaluation:** passage word/sentence/paragraph counts, length/structure checks, and fallback rate are included in the end-of-session summary. These heuristic checks do not prove language quality. ## Part A — Start from a clean Google Colab runtime 1. Open a new Colab notebook: https://colab.research.google.com/ . Choose **Runtime → Change runtime type → T4 GPU** if available. Do not run the old notebook's `all_metrics = run_reading_session()` cell. 2. Upload this ZIP file to the Colab Files panel, or upload `rise_colab_api.py` and `requirements-api.txt` directly. 3. If you uploaded the ZIP, run: ```python import zipfile with zipfile.ZipFile('/content/RISE_Complete_Integration.zip') as z: z.extractall('/content/RISE_Complete_Integration') ``` If you uploaded the two files separately, ensure they are in `/content/` and adjust the paths below. 4. Install OS and Python dependencies. This may take several minutes: ```python !apt-get update -qq !apt-get install -y -qq ffmpeg !pip -q install -r /content/RISE_Complete_Integration/requirements-api.txt ``` 5. Start the API as a **separate background process**. This avoids the `asyncio.run() cannot be called from a running event loop` problem in Colab: ```python %cd /content/RISE_Complete_Integration !nohup uvicorn rise_colab_api:app --host 0.0.0.0 --port 8000 > /content/rise_api.log 2>&1 & ``` 6. Give the first model downloads a few minutes, then check the log: ```python !tail -n 60 /content/rise_api.log ``` Look for `Application startup complete`. The first run downloads Whisper and the text-generation model; it can take time and uses substantial RAM/GPU memory. 7. Test the health endpoint from Python: ```python import requests, time for i in range(30): try: r = requests.get('http://127.0.0.1:8000/api/health', timeout=5) print(r.status_code, r.json()) break except requests.RequestException: print('Waiting for API/model startup…') time.sleep(10) ``` You want HTTP `200` and `"ok": true`. Confirm `models.whisper`, `models.text_generation`, and `models.gesture` are true. If a model is false, inspect `/content/rise_api.log` before proceeding. ## Part B — Expose the Colab API over HTTPS A browser on Hugging Face cannot call Colab's `127.0.0.1`. Use a temporary HTTPS tunnel for development/testing. 1. In a **second Colab cell**, start Cloudflare Tunnel: ```python !wget -q https://github.com/cloudflare/cloudflared/releases/latest/download/cloudflared-linux-amd64 -O /usr/local/bin/cloudflared !chmod +x /usr/local/bin/cloudflared !cloudflared tunnel --url http://127.0.0.1:8000 --no-autoupdate ``` 2. Wait for a URL ending in `.trycloudflare.com`. Keep this cell running. 3. Open `https://YOUR-URL.trycloudflare.com/api/health` in a browser. It should show the same health JSON. Copy the URL exactly. Do not add `/api/health` to the frontend base URL. **Important:** Colab runtimes stop, and quick-tunnel URLs change. This is a development bridge, not a permanent production backend. Never put secrets in frontend JavaScript. ## Part C — Connect the widget to the existing Hugging Face Space Space: https://huggingface.co/spaces/bChaukar/RISE/tree/main **First check the Space's `README.md` front matter.** The uploaded project has custom pages (`index.html`, `dashboard.html`, `assessment.html`, `speaking.html`) as well as an `app.py` described as a basic Gradio welcome UI. The Space SDK decides which entrypoint actually runs: - If the Space is intended to serve the existing custom HTML website, set the Space metadata to the Static SDK, with `sdk: static` and `app_file: index.html` (or omit `app_file` if the Space's current static configuration supports auto-detection). Keep all HTML/CSS/JS files at the repository root. A static Space can host the frontend, but it cannot execute the Colab Python models. - If you keep `sdk: gradio`, uploading `speaking.html` alone does **not** automatically make it a routed page. You must integrate the widget into the actual Gradio `app.py` UI, or switch to the static frontend approach above. Do not change the SDK until you have checked the current `README.md` metadata and know which UI you want to keep. Then connect the widget: 1. Download `rise-frontend.js` from this package. 2. In the Space's **Files** tab, upload `rise-frontend.js` at the repository root, next to `speaking.html` and `style.css`. 3. Edit `speaking.html`. Add this where you want the reading practice widget to appear (usually inside the main content): ```html
``` 4. Immediately before ``, add these scripts. Replace the URL with your actual current Cloudflare HTTPS URL: ```html ``` 5. Commit the changes in the Space. Wait for the static Space to rebuild/reload, then open its `speaking` page. Grant microphone and camera permission when the browser asks. 6. Click **Start practice**, then **Start recording**, read the passage, and click **Stop & evaluate**. The result panel should show Whisper's transcript and the metrics. Click **Hear word** next to an error to play gTTS audio. Click **Detect gesture** and show a gesture to the camera. 7. If the page says the backend cannot be reached, check (a) the Colab API cell/process is still running, (b) the Cloudflare tunnel is still running, (c) `RISE_API_BASE` exactly matches the current tunnel URL, and (d) the `/api/health` URL works in your browser. ### If the Space is static HTML The browser code only calls the temporary API; the Python API does not run inside a static Space. Keep the Space's existing frontend files. The added `rise-frontend.js` is a widget for `speaking.html`, not a replacement for your entire design. ## API endpoints - `GET /api/health` — server and model status. - `POST /api/session/start` — generate the first passage. - `GET /api/session/{session_id}` — current session state. - `POST /api/session/{session_id}/evaluate` — upload an audio recording and receive transcript, word events, scores, and next passage. - `GET /api/pronounce/{word}` — slow-English pronunciation MP3. - `POST /api/session/{session_id}/gesture` — upload one JPEG camera frame and apply the recognised gesture. - `POST /api/session/{session_id}/finish` — end session and receive summary/generation checks. ## Troubleshooting - **Connection refused:** read `/content/rise_api.log`; verify `uvicorn` is running and `/api/health` returns 200. - **Models false in health JSON:** read the error list in health JSON and `/content/rise_api.log`. Model download or RAM/GPU limits are common causes. - **ffmpeg error / audio upload fails:** confirm `!ffmpeg -version`; use Chrome/Edge and grant microphone permission. - **Camera gesture unavailable:** allow camera access; verify MediaPipe is true in health JSON and that `hand_landmarker.task` was downloaded. - **TTS fails:** gTTS requires internet access at the time a pronunciation is requested. - **Space reloads but the widget is blank:** confirm `rise-frontend.js` is at the same root path and that the HTML contains both the widget `
` and script tags. - **Do not run the old notebook auto-start line:** remove/disable `all_metrics = run_reading_session()` when using this API package. This API owns its own session lifecycle. ## Scope and limitations This is a development integration. In-memory sessions reset when the API process restarts; a temporary tunnel is not reliable for public production. Speech scores are heuristic estimates from ASR and fuzzy matching, not a clinical or certified pronunciation assessment. The model and generation evaluation code should be tested with real recordings and browser permissions before a live demonstration.