File size: 9,305 Bytes
d608554
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
# RISE β€” integrated Colab backend + Hugging Face frontend

This package replaces the earlier fragile adapter that depended on variables already loaded in a notebook. The API is self-contained: it loads the models in the API process, and it does **not** call `output.eval_js`, `IPython.display.Audio`, or `run_reading_session()`.

## What is integrated

- **Text generation:** `Qwen/Qwen2.5-1.5B-Instruct`; if it cannot load, the backend tries `google/flan-t5-base`. A small reviewed fallback passage is used if generation fails for a particular passage.
- **Speech recognition:** `openai/whisper-small`, receiving browser-recorded audio converted by `ffmpeg` to mono 16 kHz audio.
- **Pronunciation feedback:** fuzzy word matching and gTTS slow-English pronunciation audio for hesitated, mismatched, or skipped words.
- **Progression:** 5 levels Γ— sentence β†’ paragraph β†’ story; score is 50% word accuracy + 30% pronunciation similarity + 20% completion. Pass threshold is 80%, matching the first code in `Prompt.txt`. A failed attempt retries the same generated passage; passing advances the curriculum.
- **Hand gestures:** 1–5 fingers jump to levels 1–5; thumbs up restarts at Level 1 / sentence; thumbs down ends the session. Browser camera frames are sent to the API for MediaPipe processing.
- **Generation evaluation:** passage word/sentence/paragraph counts, length/structure checks, and fallback rate are included in the end-of-session summary. These heuristic checks do not prove language quality.

## Part A β€” Start from a clean Google Colab runtime

1. Open a new Colab notebook: https://colab.research.google.com/ . Choose **Runtime β†’ Change runtime type β†’ T4 GPU** if available. Do not run the old notebook's `all_metrics = run_reading_session()` cell.
2. Upload this ZIP file to the Colab Files panel, or upload `rise_colab_api.py` and `requirements-api.txt` directly.
3. If you uploaded the ZIP, run:

   ```python
   import zipfile
   with zipfile.ZipFile('/content/RISE_Complete_Integration.zip') as z:
       z.extractall('/content/RISE_Complete_Integration')
   ```

   If you uploaded the two files separately, ensure they are in `/content/` and adjust the paths below.

4. Install OS and Python dependencies. This may take several minutes:

   ```python
   !apt-get update -qq
   !apt-get install -y -qq ffmpeg
   !pip -q install -r /content/RISE_Complete_Integration/requirements-api.txt
   ```

5. Start the API as a **separate background process**. This avoids the `asyncio.run() cannot be called from a running event loop` problem in Colab:

   ```python
   %cd /content/RISE_Complete_Integration
   !nohup uvicorn rise_colab_api:app --host 0.0.0.0 --port 8000 > /content/rise_api.log 2>&1 &
   ```

6. Give the first model downloads a few minutes, then check the log:

   ```python
   !tail -n 60 /content/rise_api.log
   ```

   Look for `Application startup complete`. The first run downloads Whisper and the text-generation model; it can take time and uses substantial RAM/GPU memory.

7. Test the health endpoint from Python:

   ```python
   import requests, time
   for i in range(30):
       try:
           r = requests.get('http://127.0.0.1:8000/api/health', timeout=5)
           print(r.status_code, r.json())
           break
       except requests.RequestException:
           print('Waiting for API/model startup…')
           time.sleep(10)
   ```

   You want HTTP `200` and `"ok": true`. Confirm `models.whisper`, `models.text_generation`, and `models.gesture` are true. If a model is false, inspect `/content/rise_api.log` before proceeding.

## Part B β€” Expose the Colab API over HTTPS

A browser on Hugging Face cannot call Colab's `127.0.0.1`. Use a temporary HTTPS tunnel for development/testing.

1. In a **second Colab cell**, start Cloudflare Tunnel:

   ```python
   !wget -q https://github.com/cloudflare/cloudflared/releases/latest/download/cloudflared-linux-amd64 -O /usr/local/bin/cloudflared
   !chmod +x /usr/local/bin/cloudflared
   !cloudflared tunnel --url http://127.0.0.1:8000 --no-autoupdate
   ```

2. Wait for a URL ending in `.trycloudflare.com`. Keep this cell running.
3. Open `https://YOUR-URL.trycloudflare.com/api/health` in a browser. It should show the same health JSON. Copy the URL exactly. Do not add `/api/health` to the frontend base URL.

**Important:** Colab runtimes stop, and quick-tunnel URLs change. This is a development bridge, not a permanent production backend. Never put secrets in frontend JavaScript.

## Part C β€” Connect the widget to the existing Hugging Face Space

Space: https://huggingface.co/spaces/bChaukar/RISE/tree/main

**First check the Space's `README.md` front matter.** The uploaded project has custom pages (`index.html`, `dashboard.html`, `assessment.html`, `speaking.html`) as well as an `app.py` described as a basic Gradio welcome UI. The Space SDK decides which entrypoint actually runs:

- If the Space is intended to serve the existing custom HTML website, set the Space metadata to the Static SDK, with `sdk: static` and `app_file: index.html` (or omit `app_file` if the Space's current static configuration supports auto-detection). Keep all HTML/CSS/JS files at the repository root. A static Space can host the frontend, but it cannot execute the Colab Python models.
- If you keep `sdk: gradio`, uploading `speaking.html` alone does **not** automatically make it a routed page. You must integrate the widget into the actual Gradio `app.py` UI, or switch to the static frontend approach above. Do not change the SDK until you have checked the current `README.md` metadata and know which UI you want to keep.

Then connect the widget:

1. Download `rise-frontend.js` from this package.
2. In the Space's **Files** tab, upload `rise-frontend.js` at the repository root, next to `speaking.html` and `style.css`.
3. Edit `speaking.html`. Add this where you want the reading practice widget to appear (usually inside the main content):

   ```html
   <div id="rise-reading-app"></div>
   ```

4. Immediately before `</body>`, add these scripts. Replace the URL with your actual current Cloudflare HTTPS URL:

   ```html
   <script>
     window.RISE_API_BASE = "https://YOUR-URL.trycloudflare.com";
   </script>
   <script src="rise-frontend.js"></script>
   ```

5. Commit the changes in the Space. Wait for the static Space to rebuild/reload, then open its `speaking` page. Grant microphone and camera permission when the browser asks.
6. Click **Start practice**, then **Start recording**, read the passage, and click **Stop & evaluate**. The result panel should show Whisper's transcript and the metrics. Click **Hear word** next to an error to play gTTS audio. Click **Detect gesture** and show a gesture to the camera.
7. If the page says the backend cannot be reached, check (a) the Colab API cell/process is still running, (b) the Cloudflare tunnel is still running, (c) `RISE_API_BASE` exactly matches the current tunnel URL, and (d) the `/api/health` URL works in your browser.

### If the Space is static HTML

The browser code only calls the temporary API; the Python API does not run inside a static Space. Keep the Space's existing frontend files. The added `rise-frontend.js` is a widget for `speaking.html`, not a replacement for your entire design.

## API endpoints

- `GET /api/health` β€” server and model status.
- `POST /api/session/start` β€” generate the first passage.
- `GET /api/session/{session_id}` β€” current session state.
- `POST /api/session/{session_id}/evaluate` β€” upload an audio recording and receive transcript, word events, scores, and next passage.
- `GET /api/pronounce/{word}` β€” slow-English pronunciation MP3.
- `POST /api/session/{session_id}/gesture` β€” upload one JPEG camera frame and apply the recognised gesture.
- `POST /api/session/{session_id}/finish` β€” end session and receive summary/generation checks.

## Troubleshooting

- **Connection refused:** read `/content/rise_api.log`; verify `uvicorn` is running and `/api/health` returns 200.
- **Models false in health JSON:** read the error list in health JSON and `/content/rise_api.log`. Model download or RAM/GPU limits are common causes.
- **ffmpeg error / audio upload fails:** confirm `!ffmpeg -version`; use Chrome/Edge and grant microphone permission.
- **Camera gesture unavailable:** allow camera access; verify MediaPipe is true in health JSON and that `hand_landmarker.task` was downloaded.
- **TTS fails:** gTTS requires internet access at the time a pronunciation is requested.
- **Space reloads but the widget is blank:** confirm `rise-frontend.js` is at the same root path and that the HTML contains both the widget `<div>` and script tags.
- **Do not run the old notebook auto-start line:** remove/disable `all_metrics = run_reading_session()` when using this API package. This API owns its own session lifecycle.

## Scope and limitations

This is a development integration. In-memory sessions reset when the API process restarts; a temporary tunnel is not reliable for public production. Speech scores are heuristic estimates from ASR and fuzzy matching, not a clinical or certified pronunciation assessment. The model and generation evaluation code should be tested with real recordings and browser permissions before a live demonstration.