NiniSS commited on
Commit
d608554
·
verified ·
1 Parent(s): d16a97b

Upload 5 files

Browse files
Files changed (5) hide show
  1. README.md +142 -11
  2. README_START_HERE.txt +3 -0
  3. requirements-api.txt +11 -0
  4. rise-frontend.js +120 -0
  5. rise_colab_api.py +611 -0
README.md CHANGED
@@ -1,11 +1,142 @@
1
- ---
2
- title: RISE
3
- emoji: 🌍
4
- colorFrom: yellow
5
- colorTo: green
6
- sdk: static
7
- pinned: false
8
- license: mit
9
- ---
10
-
11
- Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # RISE — integrated Colab backend + Hugging Face frontend
2
+
3
+ This package replaces the earlier fragile adapter that depended on variables already loaded in a notebook. The API is self-contained: it loads the models in the API process, and it does **not** call `output.eval_js`, `IPython.display.Audio`, or `run_reading_session()`.
4
+
5
+ ## What is integrated
6
+
7
+ - **Text generation:** `Qwen/Qwen2.5-1.5B-Instruct`; if it cannot load, the backend tries `google/flan-t5-base`. A small reviewed fallback passage is used if generation fails for a particular passage.
8
+ - **Speech recognition:** `openai/whisper-small`, receiving browser-recorded audio converted by `ffmpeg` to mono 16 kHz audio.
9
+ - **Pronunciation feedback:** fuzzy word matching and gTTS slow-English pronunciation audio for hesitated, mismatched, or skipped words.
10
+ - **Progression:** 5 levels × sentence → paragraph → story; score is 50% word accuracy + 30% pronunciation similarity + 20% completion. Pass threshold is 80%, matching the first code in `Prompt.txt`. A failed attempt retries the same generated passage; passing advances the curriculum.
11
+ - **Hand gestures:** 1–5 fingers jump to levels 1–5; thumbs up restarts at Level 1 / sentence; thumbs down ends the session. Browser camera frames are sent to the API for MediaPipe processing.
12
+ - **Generation evaluation:** passage word/sentence/paragraph counts, length/structure checks, and fallback rate are included in the end-of-session summary. These heuristic checks do not prove language quality.
13
+
14
+ ## Part A — Start from a clean Google Colab runtime
15
+
16
+ 1. Open a new Colab notebook: https://colab.research.google.com/ . Choose **Runtime → Change runtime type → T4 GPU** if available. Do not run the old notebook's `all_metrics = run_reading_session()` cell.
17
+ 2. Upload this ZIP file to the Colab Files panel, or upload `rise_colab_api.py` and `requirements-api.txt` directly.
18
+ 3. If you uploaded the ZIP, run:
19
+
20
+ ```python
21
+ import zipfile
22
+ with zipfile.ZipFile('/content/RISE_Complete_Integration.zip') as z:
23
+ z.extractall('/content/RISE_Complete_Integration')
24
+ ```
25
+
26
+ If you uploaded the two files separately, ensure they are in `/content/` and adjust the paths below.
27
+
28
+ 4. Install OS and Python dependencies. This may take several minutes:
29
+
30
+ ```python
31
+ !apt-get update -qq
32
+ !apt-get install -y -qq ffmpeg
33
+ !pip -q install -r /content/RISE_Complete_Integration/requirements-api.txt
34
+ ```
35
+
36
+ 5. Start the API as a **separate background process**. This avoids the `asyncio.run() cannot be called from a running event loop` problem in Colab:
37
+
38
+ ```python
39
+ %cd /content/RISE_Complete_Integration
40
+ !nohup uvicorn rise_colab_api:app --host 0.0.0.0 --port 8000 > /content/rise_api.log 2>&1 &
41
+ ```
42
+
43
+ 6. Give the first model downloads a few minutes, then check the log:
44
+
45
+ ```python
46
+ !tail -n 60 /content/rise_api.log
47
+ ```
48
+
49
+ Look for `Application startup complete`. The first run downloads Whisper and the text-generation model; it can take time and uses substantial RAM/GPU memory.
50
+
51
+ 7. Test the health endpoint from Python:
52
+
53
+ ```python
54
+ import requests, time
55
+ for i in range(30):
56
+ try:
57
+ r = requests.get('http://127.0.0.1:8000/api/health', timeout=5)
58
+ print(r.status_code, r.json())
59
+ break
60
+ except requests.RequestException:
61
+ print('Waiting for API/model startup…')
62
+ time.sleep(10)
63
+ ```
64
+
65
+ You want HTTP `200` and `"ok": true`. Confirm `models.whisper`, `models.text_generation`, and `models.gesture` are true. If a model is false, inspect `/content/rise_api.log` before proceeding.
66
+
67
+ ## Part B — Expose the Colab API over HTTPS
68
+
69
+ A browser on Hugging Face cannot call Colab's `127.0.0.1`. Use a temporary HTTPS tunnel for development/testing.
70
+
71
+ 1. In a **second Colab cell**, start Cloudflare Tunnel:
72
+
73
+ ```python
74
+ !wget -q https://github.com/cloudflare/cloudflared/releases/latest/download/cloudflared-linux-amd64 -O /usr/local/bin/cloudflared
75
+ !chmod +x /usr/local/bin/cloudflared
76
+ !cloudflared tunnel --url http://127.0.0.1:8000 --no-autoupdate
77
+ ```
78
+
79
+ 2. Wait for a URL ending in `.trycloudflare.com`. Keep this cell running.
80
+ 3. Open `https://YOUR-URL.trycloudflare.com/api/health` in a browser. It should show the same health JSON. Copy the URL exactly. Do not add `/api/health` to the frontend base URL.
81
+
82
+ **Important:** Colab runtimes stop, and quick-tunnel URLs change. This is a development bridge, not a permanent production backend. Never put secrets in frontend JavaScript.
83
+
84
+ ## Part C — Connect the widget to the existing Hugging Face Space
85
+
86
+ Space: https://huggingface.co/spaces/bChaukar/RISE/tree/main
87
+
88
+ **First check the Space's `README.md` front matter.** The uploaded project has custom pages (`index.html`, `dashboard.html`, `assessment.html`, `speaking.html`) as well as an `app.py` described as a basic Gradio welcome UI. The Space SDK decides which entrypoint actually runs:
89
+
90
+ - If the Space is intended to serve the existing custom HTML website, set the Space metadata to the Static SDK, with `sdk: static` and `app_file: index.html` (or omit `app_file` if the Space's current static configuration supports auto-detection). Keep all HTML/CSS/JS files at the repository root. A static Space can host the frontend, but it cannot execute the Colab Python models.
91
+ - If you keep `sdk: gradio`, uploading `speaking.html` alone does **not** automatically make it a routed page. You must integrate the widget into the actual Gradio `app.py` UI, or switch to the static frontend approach above. Do not change the SDK until you have checked the current `README.md` metadata and know which UI you want to keep.
92
+
93
+ Then connect the widget:
94
+
95
+ 1. Download `rise-frontend.js` from this package.
96
+ 2. In the Space's **Files** tab, upload `rise-frontend.js` at the repository root, next to `speaking.html` and `style.css`.
97
+ 3. Edit `speaking.html`. Add this where you want the reading practice widget to appear (usually inside the main content):
98
+
99
+ ```html
100
+ <div id="rise-reading-app"></div>
101
+ ```
102
+
103
+ 4. Immediately before `</body>`, add these scripts. Replace the URL with your actual current Cloudflare HTTPS URL:
104
+
105
+ ```html
106
+ <script>
107
+ window.RISE_API_BASE = "https://YOUR-URL.trycloudflare.com";
108
+ </script>
109
+ <script src="rise-frontend.js"></script>
110
+ ```
111
+
112
+ 5. Commit the changes in the Space. Wait for the static Space to rebuild/reload, then open its `speaking` page. Grant microphone and camera permission when the browser asks.
113
+ 6. Click **Start practice**, then **Start recording**, read the passage, and click **Stop & evaluate**. The result panel should show Whisper's transcript and the metrics. Click **Hear word** next to an error to play gTTS audio. Click **Detect gesture** and show a gesture to the camera.
114
+ 7. If the page says the backend cannot be reached, check (a) the Colab API cell/process is still running, (b) the Cloudflare tunnel is still running, (c) `RISE_API_BASE` exactly matches the current tunnel URL, and (d) the `/api/health` URL works in your browser.
115
+
116
+ ### If the Space is static HTML
117
+
118
+ The browser code only calls the temporary API; the Python API does not run inside a static Space. Keep the Space's existing frontend files. The added `rise-frontend.js` is a widget for `speaking.html`, not a replacement for your entire design.
119
+
120
+ ## API endpoints
121
+
122
+ - `GET /api/health` — server and model status.
123
+ - `POST /api/session/start` — generate the first passage.
124
+ - `GET /api/session/{session_id}` — current session state.
125
+ - `POST /api/session/{session_id}/evaluate` — upload an audio recording and receive transcript, word events, scores, and next passage.
126
+ - `GET /api/pronounce/{word}` — slow-English pronunciation MP3.
127
+ - `POST /api/session/{session_id}/gesture` — upload one JPEG camera frame and apply the recognised gesture.
128
+ - `POST /api/session/{session_id}/finish` — end session and receive summary/generation checks.
129
+
130
+ ## Troubleshooting
131
+
132
+ - **Connection refused:** read `/content/rise_api.log`; verify `uvicorn` is running and `/api/health` returns 200.
133
+ - **Models false in health JSON:** read the error list in health JSON and `/content/rise_api.log`. Model download or RAM/GPU limits are common causes.
134
+ - **ffmpeg error / audio upload fails:** confirm `!ffmpeg -version`; use Chrome/Edge and grant microphone permission.
135
+ - **Camera gesture unavailable:** allow camera access; verify MediaPipe is true in health JSON and that `hand_landmarker.task` was downloaded.
136
+ - **TTS fails:** gTTS requires internet access at the time a pronunciation is requested.
137
+ - **Space reloads but the widget is blank:** confirm `rise-frontend.js` is at the same root path and that the HTML contains both the widget `<div>` and script tags.
138
+ - **Do not run the old notebook auto-start line:** remove/disable `all_metrics = run_reading_session()` when using this API package. This API owns its own session lifecycle.
139
+
140
+ ## Scope and limitations
141
+
142
+ This is a development integration. In-memory sessions reset when the API process restarts; a temporary tunnel is not reliable for public production. Speech scores are heuristic estimates from ASR and fuzzy matching, not a clinical or certified pronunciation assessment. The model and generation evaluation code should be tested with real recordings and browser permissions before a live demonstration.
README_START_HERE.txt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ Start here: read README.md from top to bottom.
2
+ The intended flow is: fresh Colab runtime -> install requirements -> start rise_colab_api.py with nohup/uvicorn -> verify /api/health -> start Cloudflare tunnel -> add rise-frontend.js and its script tags to speaking.html in the Hugging Face Space.
3
+ Do not run the original notebook line all_metrics = run_reading_session(); the API runs sessions through HTTP endpoints.
requirements-api.txt ADDED
@@ -0,0 +1,11 @@
 
 
 
 
 
 
 
 
 
 
 
 
1
+ fastapi>=0.115,<1.0
2
+ uvicorn[standard]>=0.30,<1.0
3
+ python-multipart>=0.0.9
4
+ transformers>=4.45,<5.0
5
+ accelerate>=0.34
6
+ soundfile>=0.12
7
+ numpy>=1.26
8
+ opencv-python-headless>=4.9
9
+ mediapipe>=0.10.14,<0.11
10
+ gTTS>=2.5
11
+ sentencepiece>=0.2
rise-frontend.js ADDED
@@ -0,0 +1,120 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ /* RISE reading widget. Set window.RISE_API_BASE before loading this script. */
2
+ (() => {
3
+ const API = (window.RISE_API_BASE || "").replace(/\/$/, "");
4
+ const root = document.getElementById("rise-reading-app");
5
+ if (!root) return;
6
+ if (!API) {
7
+ root.innerHTML = '<p style="color:#b91c1c">RISE API is not configured. Set window.RISE_API_BASE to your HTTPS backend URL.</p>';
8
+ return;
9
+ }
10
+ root.innerHTML = `
11
+ <style>
12
+ #rise-reading-app{font-family:system-ui,sans-serif;color:#172033;max-width:900px;margin:20px auto}
13
+ #rise-reading-app .rise-card{border:1px solid #dbe3ef;border-radius:16px;padding:20px;margin:12px 0;background:#fff;box-shadow:0 5px 20px #0f172a0d}
14
+ #rise-reading-app .rise-row{display:flex;gap:10px;flex-wrap:wrap;align-items:center}
15
+ #rise-reading-app button{border:0;border-radius:10px;padding:10px 14px;background:#1d4ed8;color:white;font-weight:650;cursor:pointer}
16
+ #rise-reading-app button.secondary{background:#475569} #rise-reading-app button.danger{background:#b91c1c}
17
+ #rise-reading-app button:disabled{opacity:.5;cursor:not-allowed}
18
+ #rise-reading-app #rise-passage{font-size:1.2rem;line-height:1.9;white-space:pre-wrap;background:#f8fafc;padding:16px;border-radius:12px}
19
+ #rise-reading-app .rise-muted{color:#64748b;font-size:.92rem} #rise-reading-app .rise-metrics{display:grid;grid-template-columns:repeat(auto-fit,minmax(130px,1fr));gap:8px}
20
+ #rise-reading-app .rise-metric{background:#f1f5f9;border-radius:10px;padding:10px} #rise-reading-app .rise-metric b{display:block;font-size:1.15rem}
21
+ #rise-reading-app #rise-status{min-height:1.5em} #rise-reading-app .rise-err{color:#b91c1c}
22
+ </style>
23
+ <section class="rise-card">
24
+ <h2>📖 RISE Reading Practice</h2>
25
+ <p class="rise-muted">AI-generated passages · Whisper speech recognition · pronunciation feedback · hand gestures</p>
26
+ <div class="rise-row"><button id="rise-start">Start practice</button><button id="rise-record" disabled>● Start recording</button><button id="rise-stop" disabled>Stop & evaluate</button><button id="rise-finish" class="danger" disabled>Finish session</button></div>
27
+ <p id="rise-status" role="status">Ready to start.</p>
28
+ <p id="rise-level" class="rise-muted">Level and part will appear here.</p>
29
+ <div id="rise-passage">Press “Start practice” to generate your reading passage.</div>
30
+ <div id="rise-actions" class="rise-row" style="margin-top:12px"></div>
31
+ </section>
32
+ <section class="rise-card"><h3>Speech evaluation</h3><p id="rise-transcript">Your transcription will appear here.</p><div id="rise-metrics" class="rise-metrics"></div><p id="rise-feedback"></p><div id="rise-errors"></div></section>
33
+ <section class="rise-card"><h3>Hand gesture control</h3><p class="rise-muted">Allow camera access, then capture a gesture: 1–5 fingers select a level, thumbs up restarts at Level 1, thumbs down ends the session.</p><div class="rise-row"><button id="rise-gesture" class="secondary" disabled>📷 Detect gesture</button><span id="rise-gesture-result"></span></div></section>
34
+ `;
35
+ const $ = id => root.querySelector(`#${id}`);
36
+ let sessionId = null, recorder = null, chunks = [], startedAt = 0, stream = null;
37
+ const setStatus = (text, isError=false) => { $("rise-status").textContent = text; $("rise-status").className = isError ? "rise-err" : ""; };
38
+ const setBusy = busy => { ["rise-start","rise-record","rise-stop","rise-finish","rise-gesture"].forEach(id => $(id).disabled = busy || (id !== "rise-start" && !sessionId)); };
39
+ async function api(path, options={}) {
40
+ const res = await fetch(`${API}${path}`, options);
41
+ const body = await res.json().catch(() => ({}));
42
+ if (!res.ok) throw new Error(body.detail || `Request failed (${res.status})`);
43
+ return body;
44
+ }
45
+ function showPassage(data) {
46
+ $("rise-level").textContent = `Level: ${data.level.replaceAll("_"," ")} · Part: ${data.part} · Pass mark: ${data.threshold || 80}%`;
47
+ $("rise-passage").textContent = data.passage || "Session completed.";
48
+ }
49
+ $("rise-start").addEventListener("click", async () => {
50
+ try {
51
+ setBusy(true); setStatus("Starting session and preparing the first passage…");
52
+ const data = await api("/api/session/start", {method:"POST", headers:{"Content-Type":"application/json"}, body:JSON.stringify({level:"beginner",part:"sentence"})});
53
+ sessionId = data.session_id; showPassage(data); setStatus(`Ready. Text model: ${data.model}`);
54
+ $("rise-start").disabled = true; $("rise-record").disabled = false; $("rise-finish").disabled = false; $("rise-gesture").disabled = false;
55
+ } catch(e) { setStatus(e.message, true); }
56
+ finally { if (!sessionId) $("rise-start").disabled = false; }
57
+ });
58
+ $("rise-record").addEventListener("click", async () => {
59
+ try {
60
+ stream = await navigator.mediaDevices.getUserMedia({audio:{channelCount:1,echoCancellation:true,noiseSuppression:true}});
61
+ const mime = MediaRecorder.isTypeSupported("audio/webm;codecs=opus") ? "audio/webm;codecs=opus" : "";
62
+ recorder = mime ? new MediaRecorder(stream,{mimeType:mime}) : new MediaRecorder(stream);
63
+ chunks = []; recorder.ondataavailable = e => { if(e.data.size) chunks.push(e.data); };
64
+ recorder.start(150); startedAt = Date.now();
65
+ $("rise-record").disabled = true; $("rise-stop").disabled = false; setStatus("Recording… read the passage aloud, then press Stop & evaluate.");
66
+ } catch(e) { setStatus(`Could not start microphone: ${e.message}. Check browser microphone permission.`, true); }
67
+ });
68
+ $("rise-stop").addEventListener("click", async () => {
69
+ if (!recorder || recorder.state === "inactive") return;
70
+ $("rise-stop").disabled = true; setStatus("Uploading audio and evaluating speech with Whisper…");
71
+ recorder.onstop = async () => {
72
+ if (stream) stream.getTracks().forEach(t => t.stop());
73
+ try {
74
+ const blob = new Blob(chunks,{type:recorder.mimeType || "audio/webm"});
75
+ const fd = new FormData(); fd.append("audio",blob,"reading.webm"); fd.append("elapsed_seconds",String((Date.now()-startedAt)/1000));
76
+ const data = await api(`/api/session/${sessionId}/evaluate`,{method:"POST",body:fd});
77
+ $("rise-transcript").textContent = `Heard: ${data.transcript}`;
78
+ $("rise-metrics").innerHTML = [["Overall",data.metrics.overall+"%"],["Word accuracy",data.metrics.word_accuracy+"%"],["Pronunciation",data.metrics.pronunciation+"%"],["Completion",data.metrics.completion+"%"],["WER",data.wer+"%"],["Speaking rate",data.metrics.speaking_rate+" wpm"]].map(([k,v])=>`<div class="rise-metric"><span>${k}</span><b>${v}</b></div>`).join("");
79
+ $("rise-feedback").textContent = `${data.passed ? "✅ Passed. " : "🔁 Try again. "}${data.message}`;
80
+ $("rise-errors").innerHTML = "";
81
+ for (const ev of data.events.filter(e=>e.type!=="correct")) {
82
+ const line = document.createElement("div"); line.className="rise-row"; line.style.margin="6px 0";
83
+ const label = document.createElement("span"); label.textContent = `${ev.word}: ${ev.type}${ev.heard ? ` (heard “${ev.heard}”)` : ""}`;
84
+ line.appendChild(label);
85
+ const btn = document.createElement("button"); btn.className="secondary"; btn.textContent=`🔊 Hear “${ev.word}”`; btn.addEventListener("click",()=>{const a=new Audio(`${API}/api/pronounce/${encodeURIComponent(ev.word)}`); a.play().catch(err=>setStatus(`Could not play pronunciation: ${err.message}`,true));}); line.appendChild(btn); $("rise-errors").appendChild(line);
86
+ }
87
+ if (data.completed) { setStatus(data.message); $("rise-passage").textContent="You completed all five levels. Finish session to see the summary."; $("rise-record").disabled=true; $("rise-gesture").disabled=true; }
88
+ else { showPassage({level:data.next_level,part:data.next_part,passage:data.next_passage,threshold:data.threshold}); setStatus(data.message); $("rise-record").disabled=false; }
89
+ } catch(e) { setStatus(e.message,true); $("rise-record").disabled=false; }
90
+ finally { $("rise-stop").disabled=true; }
91
+ };
92
+ recorder.stop();
93
+ });
94
+ $("rise-finish").addEventListener("click", async () => {
95
+ if (!sessionId) return;
96
+ try {
97
+ const data = await api(`/api/session/${sessionId}/finish`,{method:"POST"});
98
+ setStatus(`Session finished. Attempts: ${data.attempts}; average overall: ${data.average_overall}%; average WER: ${data.average_wer}%.`);
99
+ $("rise-feedback").textContent = JSON.stringify(data.text_generation,null,2);
100
+ $("rise-record").disabled=true; $("rise-stop").disabled=true; $("rise-gesture").disabled=true; $("rise-finish").disabled=true;
101
+ } catch(e) { setStatus(e.message,true); }
102
+ });
103
+ $("rise-gesture").addEventListener("click", async () => {
104
+ try {
105
+ const cam = await navigator.mediaDevices.getUserMedia({video:{facingMode:"user"}});
106
+ const video = document.createElement("video"); video.srcObject=cam; video.muted=true; await video.play();
107
+ await new Promise(r=>setTimeout(r,500));
108
+ const canvas=document.createElement("canvas"); canvas.width=video.videoWidth; canvas.height=video.videoHeight;
109
+ canvas.getContext("2d").drawImage(video,0,0); cam.getTracks().forEach(t=>t.stop());
110
+ const blob=await new Promise(resolve=>canvas.toBlob(resolve,"image/jpeg",0.85));
111
+ const fd=new FormData(); fd.append("image",blob,"gesture.jpg");
112
+ const data=await api(`/api/session/${sessionId}/gesture`,{method:"POST",body:fd});
113
+ $("rise-gesture-result").textContent=`Detected: ${data.gesture} · Action: ${data.action}`;
114
+ if (data.passage) showPassage({level:data.level,part:data.part,passage:data.passage,threshold:80});
115
+ if (data.ended) { setStatus("Session ended by thumbs-down gesture."); $("rise-record").disabled=true; $("rise-gesture").disabled=true; }
116
+ else if (data.action!=="none") setStatus(`Gesture applied: ${data.action}.`);
117
+ } catch(e) { setStatus(`Camera/gesture error: ${e.message}`,true); }
118
+ });
119
+ api("/api/health").then(h=>setStatus(`Backend connected. Models: Whisper ${h.models.whisper?"ready":"loading/unavailable"}, text ${h.models.text_model||"loading/unavailable"}, gestures ${h.models.gesture?"ready":"unavailable"}.`)).catch(()=>setStatus("Backend not reachable yet. Check the API URL and ensure Colab server/tunnel are running.",true));
120
+ })();
rise_colab_api.py ADDED
@@ -0,0 +1,611 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """RISE reading-assistant API for a Google Colab backend.
2
+
3
+ Uses Qwen2.5-1.5B-Instruct (FLAN-T5-base fallback), Whisper Small,
4
+ gTTS pronunciation audio, and MediaPipe Hand Landmarker.
5
+ No notebook-only output.eval_js or IPython.display calls are used in API routes.
6
+ """
7
+ from __future__ import annotations
8
+
9
+ import base64
10
+ import io
11
+ import os
12
+ import re
13
+ import random
14
+ import tempfile
15
+ import subprocess
16
+ import time
17
+ import uuid
18
+ import urllib.request
19
+ from pathlib import Path
20
+ from difflib import SequenceMatcher
21
+ from typing import Any
22
+
23
+ import numpy as np
24
+ import soundfile as sf
25
+ import torch
26
+ import cv2
27
+ from fastapi import FastAPI, File, Form, HTTPException, UploadFile
28
+ from fastapi.middleware.cors import CORSMiddleware
29
+ from fastapi.responses import StreamingResponse
30
+ from pydantic import BaseModel
31
+ from gtts import gTTS
32
+ from transformers import (
33
+ pipeline as hf_pipeline,
34
+ AutoTokenizer,
35
+ AutoModelForSeq2SeqLM,
36
+ AutoModelForCausalLM,
37
+ )
38
+
39
+ # --------------------------- Curriculum/configuration -----------------------
40
+ ADVANCE_THRESHOLD = 80.0
41
+ LEVELS = ["beginner", "elementary", "intermediate", "upper_intermediate", "advanced"]
42
+ PARTS = ["sentence", "paragraph", "story"]
43
+ NUM_SENTENCES = 3
44
+ SAMPLE_RATE = 16000
45
+ AUDIO_DIR = Path(os.environ.get("RISE_AUDIO_DIR", "/content/reading_assistant_audio"))
46
+ AUDIO_DIR.mkdir(parents=True, exist_ok=True)
47
+ LEVEL_SPECS = {
48
+ "beginner": {"sentence_words": (4, 7), "paragraph_words": 40, "story_paragraphs": 3, "story_words_per_paragraph": 38},
49
+ "elementary": {"sentence_words": (7, 10), "paragraph_words": 60, "story_paragraphs": 3, "story_words_per_paragraph": 35},
50
+ "intermediate": {"sentence_words": (10, 14), "paragraph_words": 80, "story_paragraphs": 4, "story_words_per_paragraph": 34},
51
+ "upper_intermediate": {"sentence_words": (14, 18), "paragraph_words": 90, "story_paragraphs": 4, "story_words_per_paragraph": 38},
52
+ "advanced": {"sentence_words": (18, 24), "paragraph_words": 110, "story_paragraphs": 5, "story_words_per_paragraph": 34},
53
+ }
54
+ LEVEL_GUIDANCE = {
55
+ "beginner": "Use very common everyday words, concrete topics, and short, simple sentences.",
56
+ "elementary": "Use familiar vocabulary, clear sentences, and a little descriptive detail.",
57
+ "intermediate": "Use moderately varied vocabulary, connected ideas, and some compound sentences.",
58
+ "upper_intermediate": "Use varied vocabulary, descriptive details, and more complex sentence structures.",
59
+ "advanced": "Use sophisticated vocabulary, nuanced ideas, varied syntax, and complex but readable sentences.",
60
+ }
61
+ FALLBACK = {
62
+ "beginner": {
63
+ "sentence": ["The cat sat on the mat. The dog ran to the door. The sun is very bright today."],
64
+ "paragraph": ["Mia went to the park with her brother. They played with a red ball near a large tree. After a while, they sat on a bench and drank some water. They walked home happily before dinner."],
65
+ "story": ["One sunny morning, Leo found a small turtle near a pond. He carried it carefully to his garden. His mother helped him make a safe place for it. They gave the turtle water and fresh leaves. After a few days, it was strong and active. Leo took it back to the pond and watched it swim away."]},
66
+ "elementary": {
67
+ "sentence": ["The children built a sandcastle near the edge of the water. She forgot her umbrella and got soaked in the rain. He carefully wrapped the gift before placing it under the tree."],
68
+ "paragraph": ["On Sunday, Ana and her father visited the library. They each chose three books and sat near a sunny window. Ana read a story about animals while her father read about history. On the way home, they talked about their favourite parts. The visit became a happy weekly tradition."],
69
+ "story": ["Nina had never travelled by train before. Her aunt invited her to visit the coast for the day. Nina watched fields and small towns pass the window. When the sea appeared, she smiled with excitement. They built a sandcastle and ate lunch on the beach. Nina fell asleep on the journey home, remembering the wonderful day."]},
70
+ "intermediate": {
71
+ "sentence": ["Although the weather looked threatening, they decided to continue with the picnic. She had practised the violin for years before performing on stage. The experiment produced unexpected results that challenged several established theories."],
72
+ "paragraph": ["The restoration of the old theatre took nearly four years to complete. Workers uncovered layers of paint that revealed how the building had changed over time. Beneath the stage, they found programmes, handwritten notes, and an old leather glove. The architect said the project had changed her understanding of history. When the theatre reopened, every performance sold out."],
73
+ "story": ["For many years, the lighthouse at the edge of the peninsula had guided ships safely home. A young historian named Clara was asked to research its past. She discovered old logbooks and letters hidden behind a loose stone. The letters belonged to the former keeper's daughter, who had moved away decades earlier. Clara found the woman and returned the letters to her. Together, they read about a life shaped by storms, patience, and the sea."]},
74
+ "upper_intermediate": {
75
+ "sentence": ["The proposal, while ambitious in scope, failed to account for the considerable logistical challenges involved. The relationship between language and identity is more complex than introductory accounts suggest. Her reluctance to commit to a definitive position frustrated colleagues who valued clarity above nuance."],
76
+ "paragraph": ["The concept of deep time can be profoundly disorienting when considered carefully. Human civilisation occupies only a tiny fraction of Earth's long history. Mountains that appear permanent have risen and eroded repeatedly, while oceans have advanced across continents. Understanding these changes requires intellectual humility. It reminds us that the scale of human experience is small compared with the age of the world we inhabit."],
77
+ "story": ["Two neighbours had not spoken for eleven years after a disagreement about a boundary wall. Neither could remember exactly how the argument had begun, and the wall had since been rebuilt. During a winter storm, a tree fell across their shared fence. Removing it required them to coordinate their efforts. They met with tools and began clearing the branches together. At first, their conversation was awkward, but practical questions soon became ordinary conversation. The following Sunday, they shared coffee beside the repaired fence."]},
78
+ "advanced": {
79
+ "sentence": ["The epistemological assumptions underlying the study were never made explicit, undermining the credibility of its conclusions. Language does not merely describe experience; it shapes the categories through which experience becomes available to consciousness. What appears from a distance as consensus often dissolves into a negotiation between competing interests."],
80
+ "paragraph": ["The question of what we owe to future generations is philosophically formidable because those people do not yet exist to articulate their interests. We cannot consult them or know with precision what they will value, and the world they inherit may differ substantially from our own. Yet uncertainty does not automatically dissolve our obligations. It makes fulfilling them require moral imagination, careful judgement, and a willingness to consider consequences beyond the limits of immediate experience."],
81
+ "story": ["The translator spent twelve years working on a medieval chronicle written in a dialect for which no modern dictionary existed. She reconstructed its grammar from patterns within the text, comparing repeated phrases and unusual word endings. When the translation appeared, specialists identified several errors. She acknowledged each one in later editions and explained why her original decisions had seemed reasonable. The chronicle itself attracted only a small academic audience. However, her preface became widely read because it described uncertainty without pretending that every question had a simple answer. Students continued to study it as an example of intellectual honesty."]},
82
+ }
83
+
84
+ # --------------------------- Model globals ----------------------------------
85
+ device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
86
+ genai_dtype = torch.float16 if torch.cuda.is_available() else torch.float32
87
+ GENAI_MODEL_NAME = "Qwen/Qwen2.5-1.5B-Instruct"
88
+ GENAI_ARCHITECTURE = "causal"
89
+ genai_tokenizer = None
90
+ genai_model = None
91
+ speech_to_text = None
92
+ _hand_landmarker = None
93
+ model_status: dict[str, Any] = {"whisper": False, "text_generation": False, "gesture": False, "text_model": None, "errors": []}
94
+ generation_records: list[dict[str, Any]] = []
95
+ sessions: dict[str, dict[str, Any]] = {}
96
+
97
+ _CONTRACTIONS = {
98
+ "i'm": "i am", "i've": "i have", "i'll": "i will", "i'd": "i would",
99
+ "it's": "it is", "that's": "that is", "he's": "he is", "she's": "she is",
100
+ "we're": "we are", "they're": "they are", "you're": "you are",
101
+ "isn't": "is not", "aren't": "are not", "wasn't": "was not", "weren't": "were not",
102
+ "don't": "do not", "doesn't": "does not", "didn't": "did not", "can't": "cannot",
103
+ "couldn't": "could not", "won't": "will not", "wouldn't": "would not", "there's": "there is",
104
+ "hadn't": "had not", "haven't": "have not", "hasn't": "has not",
105
+ }
106
+
107
+ def normalize_text(text: str) -> str:
108
+ text = (text or "").lower().strip()
109
+ for contraction, expanded in _CONTRACTIONS.items():
110
+ text = text.replace(contraction, expanded)
111
+ return re.sub(r"\s+", " ", re.sub(r"[^a-z\s]", " ", text)).strip()
112
+
113
+ def normalize_word(word: str) -> str:
114
+ return re.sub(r"[^a-z]", "", word.lower())
115
+
116
+ def _count_sentences(text: str) -> int:
117
+ return len([x for x in re.split(r"(?<=[.!?])\s+", text.strip()) if x.strip()])
118
+
119
+ def _clean_passage(text: str, part: str) -> str:
120
+ text = re.sub(r"^\s*(passage|story|paragraph|sentences)\s*:\s*", "", str(text), flags=re.I)
121
+ text = text.strip().strip('"“”')
122
+ paragraphs = []
123
+ for para in re.split(r"\n\s*\n", text):
124
+ para = re.sub(r"\s+", " ", para).strip()
125
+ para = re.sub(r"^(?:[-*•]|\d+[.)])\s+", "", para)
126
+ if not para:
127
+ continue
128
+ sentences = [s.strip() for s in re.split(r"(?<=[.!?])\s+", para) if s.strip()]
129
+ para = " ".join(sentences)
130
+ if para and para[-1] not in ".!?":
131
+ para += "."
132
+ if para:
133
+ paragraphs.append(para)
134
+ if part == "sentence":
135
+ sentences = []
136
+ for para in paragraphs:
137
+ sentences.extend([s.strip() for s in re.split(r"(?<=[.!?])\s+", para) if s.strip()])
138
+ return " ".join(sentences[:NUM_SENTENCES])
139
+ if part == "paragraph":
140
+ return " ".join(paragraphs)
141
+ return "\n\n".join(paragraphs)
142
+
143
+ # --------------------------- Loading models ---------------------------------
144
+ def load_models() -> None:
145
+ global speech_to_text, genai_tokenizer, genai_model, GENAI_MODEL_NAME, GENAI_ARCHITECTURE, _hand_landmarker
146
+ if speech_to_text is None:
147
+ try:
148
+ speech_to_text = hf_pipeline(
149
+ "automatic-speech-recognition", model="openai/whisper-small",
150
+ device=0 if torch.cuda.is_available() else -1,
151
+ torch_dtype=torch.float16 if torch.cuda.is_available() else torch.float32,
152
+ chunk_length_s=30, stride_length_s=5,
153
+ generate_kwargs={"language": "english", "task": "transcribe"},
154
+ )
155
+ model_status["whisper"] = True
156
+ except Exception as exc:
157
+ model_status["errors"].append(f"Whisper load failed: {exc}")
158
+ if genai_model is None:
159
+ try:
160
+ GENAI_MODEL_NAME = "Qwen/Qwen2.5-1.5B-Instruct"
161
+ GENAI_ARCHITECTURE = "causal"
162
+ genai_tokenizer = AutoTokenizer.from_pretrained(GENAI_MODEL_NAME)
163
+ genai_model = AutoModelForCausalLM.from_pretrained(
164
+ GENAI_MODEL_NAME, torch_dtype=genai_dtype, low_cpu_mem_usage=True
165
+ ).to(device)
166
+ except Exception as primary_exc:
167
+ model_status["errors"].append(f"Qwen could not load; trying FLAN-T5-base: {primary_exc}")
168
+ try:
169
+ if genai_model is not None:
170
+ del genai_model
171
+ GENAI_MODEL_NAME = "google/flan-t5-base"
172
+ GENAI_ARCHITECTURE = "seq2seq"
173
+ genai_tokenizer = AutoTokenizer.from_pretrained(GENAI_MODEL_NAME)
174
+ genai_model = AutoModelForSeq2SeqLM.from_pretrained(
175
+ GENAI_MODEL_NAME, torch_dtype=genai_dtype, low_cpu_mem_usage=True
176
+ ).to(device)
177
+ except Exception as fallback_exc:
178
+ model_status["errors"].append(f"FLAN-T5 fallback failed: {fallback_exc}")
179
+ genai_model = None
180
+ genai_tokenizer = None
181
+ if genai_model is not None:
182
+ genai_model.eval()
183
+ model_status["text_generation"] = True
184
+ model_status["text_model"] = GENAI_MODEL_NAME
185
+ try:
186
+ import mediapipe as mp
187
+ from mediapipe.tasks import python as mp_python
188
+ from mediapipe.tasks.python import vision as mp_vision
189
+ model_path = Path("/content/hand_landmarker.task")
190
+ if not model_path.exists():
191
+ model_path.parent.mkdir(parents=True, exist_ok=True)
192
+ url = ("https://storage.googleapis.com/mediapipe-models/hand_landmarker/"
193
+ "hand_landmarker/float16/1/hand_landmarker.task")
194
+ urllib.request.urlretrieve(url, str(model_path))
195
+ options = mp_vision.HandLandmarkerOptions(
196
+ base_options=mp_python.BaseOptions(model_asset_path=str(model_path)),
197
+ num_hands=1, min_hand_detection_confidence=0.7,
198
+ min_hand_presence_confidence=0.7, min_tracking_confidence=0.7,
199
+ )
200
+ _hand_landmarker = mp_vision.HandLandmarker.create_from_options(options)
201
+ model_status["gesture"] = True
202
+ except Exception as exc:
203
+ model_status["errors"].append(f"MediaPipe gesture model unavailable: {exc}")
204
+
205
+ def _model_generate(prompt: str, max_new_tokens: int = 180) -> str:
206
+ if genai_model is None or genai_tokenizer is None:
207
+ raise RuntimeError("Text generation model is not available")
208
+ if GENAI_ARCHITECTURE == "causal":
209
+ messages = [
210
+ {"role": "system", "content": "You write clear, grammatical English reading passages. Output only the requested reading text."},
211
+ {"role": "user", "content": prompt},
212
+ ]
213
+ rendered = genai_tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
214
+ encoded = genai_tokenizer(rendered, return_tensors="pt", truncation=True, max_length=2048).to(device)
215
+ with torch.inference_mode():
216
+ output_ids = genai_model.generate(
217
+ **encoded, max_new_tokens=max_new_tokens, do_sample=True,
218
+ temperature=0.65, top_p=0.9, repetition_penalty=1.12,
219
+ pad_token_id=genai_tokenizer.eos_token_id,
220
+ )
221
+ new_ids = output_ids[0][encoded["input_ids"].shape[1]:]
222
+ return genai_tokenizer.decode(new_ids, skip_special_tokens=True).strip()
223
+ encoded = genai_tokenizer(prompt, return_tensors="pt", truncation=True, max_length=512).to(device)
224
+ with torch.inference_mode():
225
+ output_ids = genai_model.generate(
226
+ **encoded, max_new_tokens=max_new_tokens, do_sample=True,
227
+ temperature=0.7, top_p=0.9, repetition_penalty=1.12,
228
+ )
229
+ return genai_tokenizer.decode(output_ids[0], skip_special_tokens=True).strip()
230
+
231
+ def _generate_paragraph(level: str, target_words: int, context: str = "") -> str:
232
+ prompt = (
233
+ "Write ONE complete English paragraph for a reading-practice app. "
234
+ f"Difficulty: {level.replace('_', ' ')}. {LEVEL_GUIDANCE[level]} "
235
+ f"Length: about {target_words} words. Use at least four complete sentences, "
236
+ "one coherent topic, no title, bullets, or numbering. "
237
+ + (f"Story context: {context}. " if context else "") + "Return only the paragraph."
238
+ )
239
+ last = ""
240
+ for _ in range(3):
241
+ last = _clean_passage(_model_generate(prompt, max_new_tokens=min(512, max(180, int(target_words * 2.4)))), "paragraph")
242
+ if _count_sentences(last) >= 4 and len(last.split()) >= max(25, int(target_words * 0.65)):
243
+ return last
244
+ prompt += " Make the paragraph longer and include at least four complete sentences."
245
+ return last
246
+
247
+ def generate_passage(level: str, part: str) -> str:
248
+ started = time.time()
249
+ spec = LEVEL_SPECS[level]
250
+ used_fallback = False
251
+ min_words, max_words = spec["sentence_words"]
252
+ target_words = NUM_SENTENCES * ((min_words + max_words) // 2) if part == "sentence" else (
253
+ spec["paragraph_words"] if part == "paragraph" else spec["story_paragraphs"] * spec["story_words_per_paragraph"]
254
+ )
255
+ try:
256
+ if part == "sentence":
257
+ prompt = (
258
+ f"Write exactly three original English sentences for reading practice at {level.replace('_',' ')} level. "
259
+ f"{LEVEL_GUIDANCE[level]} Each sentence must contain {min_words}-{max_words} words. "
260
+ "Put one complete sentence per line. No title or explanation."
261
+ )
262
+ passage = ""
263
+ for _ in range(3):
264
+ candidate = _clean_passage(_model_generate(prompt, 160), "sentence")
265
+ sentences = [s.strip() for s in re.split(r"(?<=[.!?])\s+", candidate) if s.strip()]
266
+ if len(sentences) == 3 and all(min_words <= len(s.split()) <= max_words for s in sentences):
267
+ passage = " ".join(sentences)
268
+ break
269
+ prompt += f" Try again: exactly 3 sentences, each {min_words}-{max_words} words."
270
+ if not passage:
271
+ eligible = [x for x in FALLBACK[level]["sentence"][0].split(". ") if x.strip()]
272
+ passage = " ".join((eligible * 3)[:3])
273
+ if not passage.endswith("."):
274
+ passage += "."
275
+ used_fallback = True
276
+ elif part == "paragraph":
277
+ passage = _generate_paragraph(level, spec["paragraph_words"])
278
+ if _count_sentences(passage) < 4:
279
+ passage = FALLBACK[level]["paragraph"][0]
280
+ used_fallback = True
281
+ else:
282
+ paras = []
283
+ context = "Introduce the main character, setting, and situation."
284
+ for i in range(spec["story_paragraphs"]):
285
+ if i == spec["story_paragraphs"] - 1:
286
+ context = "Continue the same story and give it a clear, satisfying ending. Previous story: " + " ".join(paras)[-900:]
287
+ elif paras:
288
+ context = "Continue the same story without repeating. Previous story: " + " ".join(paras)[-900:]
289
+ para = _generate_paragraph(level, spec["story_words_per_paragraph"], context)
290
+ if _count_sentences(para) < 3:
291
+ raise ValueError("Story paragraph was too short")
292
+ paras.append(para)
293
+ passage = "\n\n".join(paras)
294
+ except Exception as exc:
295
+ used_fallback = True
296
+ passage = FALLBACK[level][part][0]
297
+ print(f"Passage generation fallback for {level}/{part}: {exc}")
298
+ if not passage.strip():
299
+ passage = FALLBACK[level][part][0]
300
+ used_fallback = True
301
+ paragraphs = [p for p in re.split(r"\n\s*\n", passage.strip()) if p.strip()]
302
+ observed = "story" if len(paragraphs) >= 2 else ("sentence" if _count_sentences(passage) <= 3 else "paragraph")
303
+ record = {
304
+ "level": level, "requested_format": part, "observed_format": observed,
305
+ "word_count": len(passage.split()), "sentence_count": _count_sentences(passage),
306
+ "paragraph_count": len(paragraphs), "target_words": target_words,
307
+ "generation_seconds": round(time.time() - started, 2), "used_fallback": used_fallback,
308
+ "generation_method": GENAI_MODEL_NAME if not used_fallback else "reviewed_fallback",
309
+ "length_compliant": len(passage.split()) >= target_words * 0.65,
310
+ "structure_compliant": (part == "sentence" and _count_sentences(passage) == 3) or (part == "paragraph" and len(paragraphs) == 1 and _count_sentences(passage) >= 4) or (part == "story" and len(paragraphs) >= 2),
311
+ }
312
+ generation_records.append(record)
313
+ return passage
314
+
315
+ # --------------------------- Speech evaluation ------------------------------
316
+ def word_similarity(a: str, b: str) -> float:
317
+ a, b = normalize_word(a), normalize_word(b)
318
+ if not a or not b:
319
+ return 0.0
320
+ if a == b:
321
+ return 1.0
322
+ return SequenceMatcher(None, a, b).ratio()
323
+
324
+ def build_events(expected_words: list[str], spoken_words: list[str]) -> list[dict[str, Any]]:
325
+ events = []
326
+ matcher = SequenceMatcher(None, expected_words, spoken_words, autojunk=False)
327
+ for tag, i1, i2, j1, j2 in matcher.get_opcodes():
328
+ if tag == "equal":
329
+ for w in expected_words[i1:i2]:
330
+ events.append({"word": w, "heard": w, "type": "correct", "similarity": 1.0, "attempted": True})
331
+ elif tag == "replace":
332
+ exp = expected_words[i1:i2]
333
+ said = spoken_words[j1:j2]
334
+ for i, word in enumerate(exp):
335
+ heard = said[i] if i < len(said) else ""
336
+ sim = word_similarity(word, heard)
337
+ kind = "correct" if sim >= 0.80 else ("hesitation" if sim >= 0.50 else ("mismatch" if heard else "skipped"))
338
+ events.append({"word": word, "heard": heard, "type": kind, "similarity": sim, "attempted": bool(heard)})
339
+ elif tag == "delete":
340
+ for word in expected_words[i1:i2]:
341
+ events.append({"word": word, "heard": "", "type": "skipped", "similarity": 0.0, "attempted": False})
342
+ # Extra spoken words are represented in WER; no expected word is assigned to them.
343
+ return events
344
+
345
+ def calculate_wer(reference: str, hypothesis: str) -> float:
346
+ ref = normalize_text(reference).split()
347
+ hyp = normalize_text(hypothesis).split()
348
+ if not ref:
349
+ return 0.0
350
+ prev = list(range(len(hyp) + 1))
351
+ for i, rw in enumerate(ref, 1):
352
+ curr = [i]
353
+ for j, hw in enumerate(hyp, 1):
354
+ curr.append(prev[j-1] if rw == hw else 1 + min(prev[j-1], prev[j], curr[j-1]))
355
+ prev = curr
356
+ return round(prev[-1] / len(ref) * 100, 2)
357
+
358
+ def evaluate(expected_words: list[str], events: list[dict[str, Any]], spoken_text: str, elapsed_secs: float) -> dict[str, Any]:
359
+ total = len(expected_words)
360
+ correct = sum(e["type"] == "correct" for e in events)
361
+ hes = sum(e["type"] == "hesitation" for e in events)
362
+ skipped = sum(e["type"] == "skipped" for e in events)
363
+ mismatches = sum(e["type"] == "mismatch" for e in events)
364
+ accuracy = correct / total * 100 if total else 0.0
365
+ sims = [e["similarity"] for e in events if e.get("heard")]
366
+ pronunciation = float(np.mean(sims)) * 100 if sims else 0.0
367
+ attempted = correct + mismatches + hes
368
+ completion = min(attempted / total * 100, 100.0) if total else 0.0
369
+ rate = len(normalize_text(spoken_text).split()) / (elapsed_secs / 60) if elapsed_secs > 0 else 0.0
370
+ overall = 0.50 * accuracy + 0.30 * pronunciation + 0.20 * completion
371
+ return {"word_accuracy": round(accuracy, 2), "pronunciation": round(pronunciation, 2), "completion": round(completion, 2), "hesitations": hes, "skipped": skipped, "mismatches": mismatches, "speaking_rate": round(rate, 2), "overall": round(overall, 2)}
372
+
373
+ # --------------------------- MediaPipe gesture detection --------------------
374
+ def _angle(a, b, c) -> float:
375
+ a, b, c = np.array([a.x, a.y]), np.array([b.x, b.y]), np.array([c.x, c.y])
376
+ ba, bc = a - b, c - b
377
+ cos = np.dot(ba, bc) / (np.linalg.norm(ba) * np.linalg.norm(bc) + 1e-8)
378
+ return float(np.degrees(np.arccos(np.clip(cos, -1.0, 1.0))))
379
+
380
+ def _finger_states(landmarks) -> list[int]:
381
+ wrist, index_mcp, pinky_mcp = landmarks[0], landmarks[5], landmarks[17]
382
+ palm_width = np.sqrt((index_mcp.x - pinky_mcp.x) ** 2 + (index_mcp.y - pinky_mcp.y) ** 2)
383
+ palm_cx = sum(landmarks[i].x for i in [0, 5, 9, 13, 17]) / 5
384
+ palm_cy = sum(landmarks[i].y for i in [0, 5, 9, 13, 17]) / 5
385
+ thumb_angle = _angle(landmarks[2], landmarks[3], landmarks[4])
386
+ thumb_dist = np.sqrt((landmarks[4].x - palm_cx) ** 2 + (landmarks[4].y - palm_cy) ** 2)
387
+ fingers = [1 if thumb_angle > 145 and thumb_dist > palm_width * 0.65 else 0]
388
+ for mcp, pip, tip in [(5, 6, 8), (9, 10, 12), (13, 14, 16), (17, 18, 20)]:
389
+ angle = _angle(landmarks[mcp], landmarks[pip], landmarks[tip])
390
+ tip_dist = np.sqrt((landmarks[tip].x - wrist.x) ** 2 + (landmarks[tip].y - wrist.y) ** 2)
391
+ pip_dist = np.sqrt((landmarks[pip].x - wrist.x) ** 2 + (landmarks[pip].y - wrist.y) ** 2)
392
+ fingers.append(1 if angle > 155 and tip_dist > pip_dist else 0)
393
+ return fingers
394
+
395
+ def _recognize_gesture(fingers: list[int], landmarks) -> str:
396
+ if fingers == [1, 1, 1, 1, 1]: return "LEVEL_5"
397
+ if fingers == [0, 1, 1, 1, 1]: return "LEVEL_4"
398
+ if fingers == [0, 1, 1, 1, 0]: return "LEVEL_3"
399
+ if fingers == [0, 1, 1, 0, 0]: return "LEVEL_2"
400
+ if fingers == [0, 1, 0, 0, 0]: return "LEVEL_1"
401
+ if fingers == [1, 0, 0, 0, 0]: return "HOME" if landmarks[4].y < landmarks[0].y else "PAUSE"
402
+ return "UNKNOWN"
403
+
404
+ def detect_gesture_from_bytes(raw: bytes) -> str:
405
+ if _hand_landmarker is None:
406
+ raise HTTPException(status_code=503, detail="Gesture detector is not available. Check /api/health.")
407
+ arr = np.frombuffer(raw, dtype=np.uint8)
408
+ frame = cv2.imdecode(arr, cv2.IMREAD_COLOR)
409
+ if frame is None:
410
+ raise HTTPException(status_code=400, detail="Could not decode the camera image.")
411
+ rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
412
+ import mediapipe as mp
413
+ mp_image = mp.Image(image_format=mp.ImageFormat.SRGB, data=rgb)
414
+ result = _hand_landmarker.detect(mp_image)
415
+ if not result.hand_landmarks:
416
+ return "UNKNOWN"
417
+ landmarks = result.hand_landmarks[0]
418
+ return _recognize_gesture(_finger_states(landmarks), landmarks)
419
+
420
+ # --------------------------- API application --------------------------------
421
+ app = FastAPI(title="RISE Reading Assistant API", version="1.0.0")
422
+ app.add_middleware(
423
+ CORSMiddleware, allow_origins=["*"], allow_credentials=False,
424
+ allow_methods=["*"], allow_headers=["*"],
425
+ )
426
+
427
+ @app.on_event("startup")
428
+ def _startup():
429
+ print("Loading RISE models. This may take several minutes on the first run...")
430
+ load_models()
431
+ print("RISE model status:", model_status)
432
+
433
+ @app.get("/api/health")
434
+ def health():
435
+ return {"ok": True, "service": "RISE", "models": model_status, "threshold": ADVANCE_THRESHOLD}
436
+
437
+ class StartRequest(BaseModel):
438
+ level: str = "beginner"
439
+ part: str = "sentence"
440
+
441
+ @app.post("/api/session/start")
442
+ def start_session(request: StartRequest):
443
+ if not model_status.get("text_generation"):
444
+ raise HTTPException(status_code=503, detail="Text generation model is not ready. Check /api/health and server logs.")
445
+ level = request.level if request.level in LEVELS else LEVELS[0]
446
+ part = request.part if request.part in PARTS else PARTS[0]
447
+ session_id = str(uuid.uuid4())
448
+ passage = generate_passage(level, part)
449
+ sessions[session_id] = {"id": session_id, "level": level, "part": part, "passage": passage, "history": [], "ended": False, "created_at": time.time()}
450
+ return {"session_id": session_id, "level": level, "part": part, "passage": passage, "threshold": ADVANCE_THRESHOLD, "model": GENAI_MODEL_NAME}
451
+
452
+ @app.get("/api/session/{session_id}")
453
+ def get_session(session_id: str):
454
+ state = sessions.get(session_id)
455
+ if not state:
456
+ raise HTTPException(status_code=404, detail="Session not found. Start a new session.")
457
+ return {k: v for k, v in state.items() if k != "history"} | {"attempts": len(state["history"])}
458
+
459
+ @app.post("/api/session/{session_id}/evaluate")
460
+ async def evaluate_audio(session_id: str, audio: UploadFile = File(...), elapsed_seconds: float = Form(0.0)):
461
+ state = sessions.get(session_id)
462
+ if not state:
463
+ raise HTTPException(status_code=404, detail="Session not found. Start a new session.")
464
+ if state["ended"]:
465
+ raise HTTPException(status_code=400, detail="This session has ended. Start a new session.")
466
+ if not model_status.get("whisper") or speech_to_text is None:
467
+ raise HTTPException(status_code=503, detail="Whisper is not ready. Check /api/health.")
468
+ raw = await audio.read()
469
+ if not raw:
470
+ raise HTTPException(status_code=400, detail="The uploaded audio is empty.")
471
+ suffix = ".webm" if "webm" in (audio.content_type or "") or (audio.filename or "").endswith(".webm") else (Path(audio.filename or "recording.wav").suffix or ".wav")
472
+ try:
473
+ with tempfile.TemporaryDirectory() as tmp:
474
+ in_path, wav_path = Path(tmp) / f"input{suffix}", Path(tmp) / "audio.wav"
475
+ in_path.write_bytes(raw)
476
+ proc = subprocess.run(
477
+ ["ffmpeg", "-y", "-loglevel", "error", "-i", str(in_path), "-ar", str(SAMPLE_RATE), "-ac", "1", "-sample_fmt", "s16", str(wav_path)],
478
+ capture_output=True, text=True, timeout=90,
479
+ )
480
+ if proc.returncode != 0 or not wav_path.exists():
481
+ raise ValueError(proc.stderr[-800:] or "ffmpeg could not decode the audio")
482
+ samples, sr = sf.read(str(wav_path), dtype="float32")
483
+ except Exception as exc:
484
+ raise HTTPException(status_code=400, detail=f"Could not decode recording. Please record again. Details: {str(exc)[:300]}")
485
+ if samples.ndim > 1:
486
+ samples = samples.mean(axis=1)
487
+ if len(samples) < SAMPLE_RATE // 2:
488
+ raise HTTPException(status_code=400, detail="Recording is too short. Please read aloud for at least half a second.")
489
+ peak = float(np.max(np.abs(samples))) if len(samples) else 0.0
490
+ if peak > 0.001:
491
+ samples = samples * min(0.7 / peak, 3.0)
492
+ duration = elapsed_seconds if elapsed_seconds > 0 else len(samples) / SAMPLE_RATE
493
+ try:
494
+ transcript = speech_to_text({"array": samples, "sampling_rate": SAMPLE_RATE})["text"].strip()
495
+ except Exception as exc:
496
+ raise HTTPException(status_code=500, detail=f"Whisper transcription failed: {str(exc)[:250]}")
497
+ if not normalize_text(transcript):
498
+ raise HTTPException(status_code=400, detail="No speech was detected. Check microphone permissions and try again.")
499
+ expected_words = normalize_text(state["passage"]).split()
500
+ spoken_words = normalize_text(transcript).split()
501
+ events = build_events(expected_words, spoken_words)
502
+ metrics = evaluate(expected_words, events, transcript, duration)
503
+ wer = calculate_wer(state["passage"], transcript)
504
+ pronunciation_words = list(dict.fromkeys(e["word"] for e in events if e["type"] in {"hesitation", "mismatch", "skipped"}))
505
+ attempt = {"level": state["level"], "part": state["part"], "transcript": transcript, "metrics": metrics, "wer": wer, "timestamp": time.time()}
506
+ state["history"].append(attempt)
507
+ passed = metrics["overall"] >= ADVANCE_THRESHOLD
508
+ old_level, old_part = state["level"], state["part"]
509
+ complete = False
510
+ if passed:
511
+ level_index, part_index = LEVELS.index(old_level), PARTS.index(old_part)
512
+ if part_index + 1 < len(PARTS):
513
+ state["part"] = PARTS[part_index + 1]
514
+ state["passage"] = generate_passage(state["level"], state["part"])
515
+ message = f"Passed! Moving to {state['part']} at {state['level'].replace('_', ' ')} level."
516
+ elif level_index + 1 < len(LEVELS):
517
+ state["level"] = LEVELS[level_index + 1]
518
+ state["part"] = PARTS[0]
519
+ state["passage"] = generate_passage(state["level"], state["part"])
520
+ message = f"Level complete! Moving to {state['level'].replace('_', ' ')}."
521
+ else:
522
+ state["ended"] = True
523
+ complete = True
524
+ message = "Congratulations! You completed all five levels."
525
+ else:
526
+ # Keep the same generated passage so the learner can retry the exact same phase.
527
+ message = f"Score below {ADVANCE_THRESHOLD:.0f}%. Try the same passage again."
528
+ return {
529
+ "session_id": session_id, "level": old_level, "part": old_part,
530
+ "transcript": transcript, "metrics": metrics, "wer": wer,
531
+ "events": events, "pronunciation_words": pronunciation_words,
532
+ "passed": passed, "threshold": ADVANCE_THRESHOLD, "message": message,
533
+ "next_level": state["level"], "next_part": state["part"],
534
+ "next_passage": state["passage"] if not complete else None,
535
+ "completed": complete, "ended": state["ended"],
536
+ }
537
+
538
+ @app.get("/api/pronounce/{word}")
539
+ def pronounce(word: str):
540
+ word = re.sub(r"[^A-Za-z'-]", "", word)[:60]
541
+ if not word:
542
+ raise HTTPException(status_code=400, detail="Provide a word to pronounce.")
543
+ buf = io.BytesIO()
544
+ try:
545
+ gTTS(text=word, lang="en", slow=True).write_to_fp(buf)
546
+ except Exception as exc:
547
+ raise HTTPException(status_code=502, detail=f"Text-to-speech could not generate audio: {str(exc)[:200]}")
548
+ buf.seek(0)
549
+ return StreamingResponse(buf, media_type="audio/mpeg", headers={"Cache-Control": "no-store"})
550
+
551
+ @app.post("/api/session/{session_id}/gesture")
552
+ async def gesture(session_id: str, image: UploadFile = File(...)):
553
+ state = sessions.get(session_id)
554
+ if not state:
555
+ raise HTTPException(status_code=404, detail="Session not found. Start a new session.")
556
+ raw = await image.read()
557
+ gesture_name = detect_gesture_from_bytes(raw)
558
+ action = "none"
559
+ if gesture_name == "HOME":
560
+ state["level"], state["part"] = LEVELS[0], PARTS[0]
561
+ state["passage"] = generate_passage(state["level"], state["part"])
562
+ state["ended"] = False
563
+ action = "home"
564
+ elif gesture_name == "PAUSE":
565
+ state["ended"] = True
566
+ action = "pause"
567
+ elif gesture_name.startswith("LEVEL_"):
568
+ level_index = int(gesture_name.split("_")[1]) - 1
569
+ state["level"], state["part"] = LEVELS[level_index], PARTS[0]
570
+ state["passage"] = generate_passage(state["level"], state["part"])
571
+ state["ended"] = False
572
+ action = "jump_level"
573
+ return {"gesture": gesture_name, "action": action, "level": state["level"], "part": state["part"], "passage": state["passage"] if not state["ended"] else None, "ended": state["ended"]}
574
+
575
+ @app.post("/api/session/{session_id}/finish")
576
+ def finish_session(session_id: str):
577
+ state = sessions.get(session_id)
578
+ if not state:
579
+ raise HTTPException(status_code=404, detail="Session not found. Start a new session.")
580
+ state["ended"] = True
581
+ history = state["history"]
582
+ formats = ["sentence", "paragraph", "story"]
583
+ matrix = {actual: {predicted: 0 for predicted in formats} for actual in formats}
584
+ for record in generation_records:
585
+ actual = record.get("requested_format")
586
+ predicted = record.get("observed_format")
587
+ if actual in matrix and predicted in matrix[actual]:
588
+ matrix[actual][predicted] += 1
589
+ format_matches = sum(matrix[f][f] for f in formats)
590
+ generation_summary = {
591
+ "passages_generated": len(generation_records),
592
+ "format_confusion_matrix_rows_requested_columns_detected": matrix,
593
+ "heuristic_format_match_rate_percent": round(100 * format_matches / len(generation_records), 2) if generation_records else 0,
594
+ "fallback_rate_percent": round(100 * sum(bool(x["used_fallback"]) for x in generation_records) / len(generation_records), 2) if generation_records else 0,
595
+ "length_compliance_percent": round(100 * sum(bool(x["length_compliant"]) for x in generation_records) / len(generation_records), 2) if generation_records else 0,
596
+ "structure_compliance_percent": round(100 * sum(bool(x["structure_compliant"]) for x in generation_records) / len(generation_records), 2) if generation_records else 0,
597
+ "mean_generation_seconds": round(float(np.mean([x["generation_seconds"] for x in generation_records])), 2) if generation_records else 0,
598
+ "records": generation_records[-100:],
599
+ "note": "Format detection is heuristic; these automatic checks do not establish grammar, coherence, factuality, or true reading-level suitability."
600
+ }
601
+ return {
602
+ "session_id": session_id, "ended": True, "level": state["level"], "part": state["part"],
603
+ "attempts": len(history),
604
+ "average_word_accuracy": round(float(np.mean([h["metrics"]["word_accuracy"] for h in history])), 2) if history else 0,
605
+ "average_overall": round(float(np.mean([h["metrics"]["overall"] for h in history])), 2) if history else 0,
606
+ "average_wer": round(float(np.mean([h["wer"] for h in history])), 2) if history else 0,
607
+ "history": history,
608
+ "text_generation": generation_summary,
609
+ }
610
+
611
+ # Run with: uvicorn rise_colab_api:app --host 0.0.0.0 --port 8000