# Setup and operation

Tested on: **Python 3.13.14, Linux x86_64, CPU-only (2 vCPU, ~2 GB RAM)** with the
pinned versions in `requirements.txt`. TensorFlow 2.21 + Keras 3, MediaPipe 1.0.1
(Tasks API only — that release has **no** `mediapipe.solutions`).

---

## 1. Install

```bash
git clone <your-fork> isl-recognition && cd isl-recognition
python -m venv .venv && source .venv/bin/activate      # Windows: .venv\Scripts\activate
pip install --upgrade pip
pip install -r requirements.txt
python run.py doctor          # environment + asset self-check
```

Everything after this is driven by `python run.py <command>`; each command forwards
extra flags to the underlying script (`python run.py train --model gru --epochs 40`).

## 2. Get the data (required for audit/training)

The project trains on AI4Bharat's released **INCLUDE keypoint archive** (≈0.66 GB), not
on the 56 GB raw video release.

```bash
# 1) official metadata (a few hundred KB) -> datasets/metadata/
python run.py metadata

# 2) keypoints.  Download Pose_Signs.zip from the OpenHands/INCLUDE Zenodo record and
#    place the extracted archive where the loader looks, or point the env var at it:
#      https://zenodo.org/record/6674324/files/INCLUDE.zip      (contains Pose_Signs.zip)
export ISL_KEYPOINTS_PATH=/path/to/Pose_Signs_fresh.zip
python run.py info            # the "keypoint archive" line must say OK
```

The loader accepts either the `.zip` archive or an extracted `Pose_Signs/` folder and
records the resolved path in `docs/dataset_audit.md`.

MediaPipe Task bundles (needed only for webcam/video input, not for the dataset):

```bash
python scripts/download_models.py           # holistic + pose + hand, with checksums
```

## 3. Reproduce the study (the whole pipeline)

```bash
python run.py audit           # PHASE 2 -> docs/dataset_audit.md, results/dataset_audit_summary.json
python run.py preprocess      # PHASE 4 -> datasets/processed/sequences_session_disjoint.npz (~37 MB)
python run.py train           # PHASES 5-7 -> models/{lstm,gru,gru_mha}/  (~2-3 min/model on 2 vCPU)
python run.py evaluate        # PHASE 7 -> results/summary_metrics.csv, per_class_comparison.csv, plots
python run.py robustness      # PHASE 8 -> results/robustness_results.csv + degradation plot
python run.py threshold       #         -> results/threshold_sweep.csv + plot
```

Both split protocols are supported everywhere:

```bash
python run.py preprocess --protocol official
python run.py evaluate   --protocol official     # -> *_official.csv / *_official.png
python run.py robustness --protocol official
```

Environment overrides (all optional, `.env` is read automatically):

```bash
ISL_SEQUENCE_LENGTH=40 ISL_CONFIDENCE_THRESHOLD=0.8 ISL_SPLIT_PROTOCOL=official python run.py train
```

## 4. Run the system

```bash
# 1) API and browser dashboard (http://localhost:8000; API docs at /docs)
python run.py api
#    or: uvicorn backend.main:app --host 0.0.0.0 --port 8000

# 2) Live webcam recognition (local machine, camera required)
python run.py realtime --model gru_mha --threshold 0.70
#    keys: q quit | s speech | a accept | c correct | r reject | 1/2/3 model | x clear | b backspace

# 3) Offline / batch
python run.py batch --from-dataset --split test --limit 20
python run.py batch --video uploads/hello.mp4
python run.py batch --video-dir uploads/ --recursive
```

## 5. Tests

```bash
python run.py test                 # or: python -m pytest tests -v
python -m pytest tests/test_api.py -q
```

Tests that need the dataset or trained models **skip with an explicit reason** when
the artefact is absent, so the suite is meaningful on a fresh checkout and complete on
a full one. No test writes into the real logs (audit paths are redirected to a temp
dir) and no test writes into `models/personalized/`.

## 6. Docker

```bash
docker compose up --build          # API + browser dashboard :8000
docker compose run --rm api python training/audit_dataset.py
docker compose run --rm api python -m pytest tests -q
```

Mount `models/`, `datasets/`, `results/`, `logs/` (compose does this) so artefacts and
audit data live on the host.

### Webcam limitations in Docker

A container normally has no `/dev/video0`, and a browser camera belongs to the host
machine, so **local webcam recognition is not available inside Docker**. Supported
alternatives:

1. **Run `backend/inference/realtime.py` on the host** against the same checkout — full camera
   rate, the intended way to demo real-time behaviour.
2. **Use the browser dashboard** at `http://localhost:8000` for video upload,
   dataset demos or browser-camera capture. Browser capture and network transfer add
   latency compared with the local real-time camera loop.
3. **Run the API in Docker** for upload/dataset workflows. The Python real-time camera
   loop still runs on the host because it needs direct camera access and local MediaPipe.

## 7. Why the webcam does not work

Run the dedicated check first — it measures the two things a webcam needs (a capture
device and, for the preview window, a display) and prints the matching fix:

```bash
python run.py camera          # or: python scripts/camera_check.py --indices 0 1 2 3
```

Measured example from a headless container (this is what "not working" usually is):

```
platform            : Linux 6.1.158+
capture devices     : none (/dev/video* absent)
DISPLAY             : <unset>
OpenCV GUI backend  : Qt5
preview window      : False  (Available platform plugins are: xcb.)
  [--   ] index 0: opened=False readable=False ... device could not be opened
 * No /dev/video* device exists: this machine has no camera attached
 * No usable display: cv2.imshow here aborts the process -> use --no-window
```

| symptom | cause | fix |
|---|---|---|
| `no /dev/video*` / `device nodes: none` | the machine has no camera: server, container, WSL, CI, cloud sandbox, or VM without USB passthrough | use the browser dashboard upload/dataset tabs, `backend/inference/batch.py`, or run on a desktop machine |
| `opened=False` on every index although a camera is attached | camera already in use by another app (Zoom/Teams/browser tab) | close the other app, then re-run |
| same, on macOS | the *terminal application* was never granted camera access | System Settings → Privacy & Security → Camera → enable Terminal/iTerm/VS Code, then restart it |
| same, on Windows | camera access blocked for desktop apps | Settings → Privacy & security → Camera → allow desktop apps |
| index 0 fails, another index works | laptop has an internal + external camera, or a virtual camera is registered first | `python run.py realtime --camera 1` (the script already falls back through indices 0-3) |
| WSL2: no device at all | WSL does not forward USB cameras by default | forward it: `usbipd list` → `usbipd attach --wsl --busid <id>`, then check `/dev/video0` |
| Docker: no device | containers do not get `/dev/video0` automatically | `docker run --device /dev/video0 ...` plus `--group-add video`, and run with `--no-window` (no X server) — see §6 |
| process **dies with no Python traceback** after printing `qt.qpa.xcb: could not connect to display` | `cv2.imshow` with no display aborts the process in the C++ layer | this is handled now: `realtime.py` probes the GUI in a subprocess and continues windowless (`--no-window` forces it). If you see it, you are running an older copy |
| `cv2.error: The function is not implemented ...` on `imshow` | OpenCV was installed as `opencv-python-headless` (the pinned default, which has no GUI) | `pip install opencv-python` (non-headless) for a desktop preview window; headless is fine for uploads, batch and dataset work |
| camera works but every sign is `UNCERTAIN` | detector is fine, model confidence is low (that is the gate working) | check the hand-detection rates in the UI/HUD, hold the sign for the full buffer, and tune `--threshold` after reading `docs/evaluation.md` §5 |

Reminder: the hosted/sandbox preview you may be looking at has **no camera device and
no display**, so real-time video cannot work there by construction — that is why the
live tab buffers browser captures and why `realtime.py` prints fallbacks instead of a
traceback. Use `python run.py batch --from-dataset`, a video upload, or a desktop
machine for camera-rate recognition.

## 8. Troubleshooting

| symptom | cause / fix |
|---|---|
| `keypoint archive ... --` in `run.py info` | keypoints not found; set `ISL_KEYPOINTS_PATH` (see §2) |
| `FileNotFoundError: trained model not found` | train first: `python run.py train` (or `--model gru_mha`) |
| `/predict` returns 503 | same cause; `/health` reports `model_loaded=false` with the reason |
| MediaPipe prints `cuInit`/`oneDNN`/`absl` noise | harmless on CPU builds; the scripts filter it in logging |
| `Could not open camera index 0` | headless/container environment — see §6 and §7 |
| TTS says unavailable | no `gtts`/`pyttsx3` or no audio player; recognition still works (`TTS_ENGINE=none` disables it explicitly) |
| frontend shows "Backend not reachable" | start the API first (`python run.py api`) — the UI never fakes a prediction |
| Docker build is slow | TensorFlow is a large wheel; the layer cache makes rebuilds cheap |

## 9. Housekeeping

* `logs/predictions.jsonl` and `logs/feedback/feedback.jsonl` grow append-only; delete
  them to reset the error-review view (or use `backend.inference.audit.clear_logs()`).
* Evaluation results, trained model weights, the processed session-disjoint tensor
  and MediaPipe task bundles are kept in the project for a reproducible demo.
* Raw recordings and uploaded videos are not included in the repository; uploads are
  never retained unless `ISL_PERSIST_UPLOADS=true`.