docs/setup.md

Setup and operation

Tested on: Python 3.13.14, Linux x86_64, CPU-only (2 vCPU, ~2 GB RAM) with the pinned versions in requirements.txt. TensorFlow 2.21 + Keras 3, MediaPipe 1.0.1 (Tasks API only — that release has no mediapipe.solutions).

---

1. Install

git clone <your-fork> isl-recognition && cd isl-recognition
python -m venv .venv && source .venv/bin/activate      # Windows: .venv\Scripts\activate
pip install --upgrade pip
pip install -r requirements.txt
python run.py doctor          # environment + asset self-check

Everything after this is driven by python run.py <command>; each command forwards extra flags to the underlying script (python run.py train --model gru --epochs 40).

2. Get the data (required for audit/training)

The project trains on AI4Bharat's released INCLUDE keypoint archive (≈0.66 GB), not on the 56 GB raw video release.

# 1) official metadata (a few hundred KB) -> datasets/metadata/
python run.py metadata

# 2) keypoints.  Download Pose_Signs.zip from the OpenHands/INCLUDE Zenodo record and
#    place the extracted archive where the loader looks, or point the env var at it:
#      https://zenodo.org/record/6674324/files/INCLUDE.zip      (contains Pose_Signs.zip)
export ISL_KEYPOINTS_PATH=/path/to/Pose_Signs_fresh.zip
python run.py info            # the "keypoint archive" line must say OK

The loader accepts either the .zip archive or an extracted Pose_Signs/ folder and records the resolved path in docs/dataset_audit.md.

MediaPipe Task bundles (needed only for webcam/video input, not for the dataset):

python scripts/download_models.py           # holistic + pose + hand, with checksums

3. Reproduce the study (the whole pipeline)

python run.py audit           # PHASE 2 -> docs/dataset_audit.md, results/dataset_audit_summary.json
python run.py preprocess      # PHASE 4 -> datasets/processed/sequences_session_disjoint.npz (~37 MB)
python run.py train           # PHASES 5-7 -> models/{lstm,gru,gru_mha}/  (~2-3 min/model on 2 vCPU)
python run.py evaluate        # PHASE 7 -> results/summary_metrics.csv, per_class_comparison.csv, plots
python run.py robustness      # PHASE 8 -> results/robustness_results.csv + degradation plot
python run.py threshold       #         -> results/threshold_sweep.csv + plot

Both split protocols are supported everywhere:

python run.py preprocess --protocol official
python run.py evaluate   --protocol official     # -> *_official.csv / *_official.png
python run.py robustness --protocol official

Environment overrides (all optional, .env is read automatically):

ISL_SEQUENCE_LENGTH=40 ISL_CONFIDENCE_THRESHOLD=0.8 ISL_SPLIT_PROTOCOL=official python run.py train

4. Run the system

# 1) API and browser dashboard (http://localhost:8000; API docs at /docs)
python run.py api
#    or: uvicorn backend.main:app --host 0.0.0.0 --port 8000

# 2) Live webcam recognition (local machine, camera required)
python run.py realtime --model gru_mha --threshold 0.70
#    keys: q quit | s speech | a accept | c correct | r reject | 1/2/3 model | x clear | b backspace

# 3) Offline / batch
python run.py batch --from-dataset --split test --limit 20
python run.py batch --video uploads/hello.mp4
python run.py batch --video-dir uploads/ --recursive

5. Tests

python run.py test                 # or: python -m pytest tests -v
python -m pytest tests/test_api.py -q

Tests that need the dataset or trained models skip with an explicit reason when the artefact is absent, so the suite is meaningful on a fresh checkout and complete on a full one. No test writes into the real logs (audit paths are redirected to a temp dir) and no test writes into models/personalized/.

6. Docker

docker compose up --build          # API + browser dashboard :8000
docker compose run --rm api python training/audit_dataset.py
docker compose run --rm api python -m pytest tests -q

Mount models/, datasets/, results/, logs/ (compose does this) so artefacts and audit data live on the host.

Webcam limitations in Docker

A container normally has no /dev/video0, and a browser camera belongs to the host machine, so local webcam recognition is not available inside Docker. Supported alternatives:

1. Run backend/inference/realtime.py on the host against the same checkout — full camera rate, the intended way to demo real-time behaviour.
2. Use the browser dashboard at http://localhost:8000 for video upload, dataset demos or browser-camera capture. Browser capture and network transfer add latency compared with the local real-time camera loop.
3. Run the API in Docker for upload/dataset workflows. The Python real-time camera loop still runs on the host because it needs direct camera access and local MediaPipe.

7. Why the webcam does not work

Run the dedicated check first — it measures the two things a webcam needs (a capture device and, for the preview window, a display) and prints the matching fix:

python run.py camera          # or: python scripts/camera_check.py --indices 0 1 2 3

Measured example from a headless container (this is what "not working" usually is):

platform            : Linux 6.1.158+
capture devices     : none (/dev/video* absent)
DISPLAY             : <unset>
OpenCV GUI backend  : Qt5
preview window      : False  (Available platform plugins are: xcb.)
  [--   ] index 0: opened=False readable=False ... device could not be opened
 * No /dev/video* device exists: this machine has no camera attached
 * No usable display: cv2.imshow here aborts the process -> use --no-window

symptom	cause	fix
no /dev/video* / device nodes: none	the machine has no camera: server, container, WSL, CI, cloud sandbox, or VM without USB passthrough	use the browser dashboard upload/dataset tabs, backend/inference/batch.py, or run on a desktop machine
opened=False on every index although a camera is attached	camera already in use by another app (Zoom/Teams/browser tab)	close the other app, then re-run
same, on macOS	the terminal application was never granted camera access	System Settings → Privacy & Security → Camera → enable Terminal/iTerm/VS Code, then restart it
same, on Windows	camera access blocked for desktop apps	Settings → Privacy & security → Camera → allow desktop apps
index 0 fails, another index works	laptop has an internal + external camera, or a virtual camera is registered first	python run.py realtime --camera 1 (the script already falls back through indices 0-3)
WSL2: no device at all	WSL does not forward USB cameras by default	forward it: usbipd list → usbipd attach --wsl --busid <id>, then check /dev/video0
Docker: no device	containers do not get /dev/video0 automatically	docker run --device /dev/video0 ... plus --group-add video, and run with --no-window (no X server) — see §6
process dies with no Python traceback after printing qt.qpa.xcb: could not connect to display	cv2.imshow with no display aborts the process in the C++ layer	this is handled now: realtime.py probes the GUI in a subprocess and continues windowless (--no-window forces it). If you see it, you are running an older copy
cv2.error: The function is not implemented ... on imshow	OpenCV was installed as opencv-python-headless (the pinned default, which has no GUI)	pip install opencv-python (non-headless) for a desktop preview window; headless is fine for uploads, batch and dataset work
camera works but every sign is UNCERTAIN	detector is fine, model confidence is low (that is the gate working)	check the hand-detection rates in the UI/HUD, hold the sign for the full buffer, and tune --threshold after reading docs/evaluation.md §5

Reminder: the hosted/sandbox preview you may be looking at has no camera device and no display, so real-time video cannot work there by construction — that is why the live tab buffers browser captures and why realtime.py prints fallbacks instead of a traceback. Use python run.py batch --from-dataset, a video upload, or a desktop machine for camera-rate recognition.

8. Troubleshooting

symptom	cause / fix
keypoint archive ... -- in run.py info	keypoints not found; set ISL_KEYPOINTS_PATH (see §2)
FileNotFoundError: trained model not found	train first: python run.py train (or --model gru_mha)
/predict returns 503	same cause; /health reports model_loaded=false with the reason
MediaPipe prints cuInit/oneDNN/absl noise	harmless on CPU builds; the scripts filter it in logging
Could not open camera index 0	headless/container environment — see §6 and §7
TTS says unavailable	no gtts/pyttsx3 or no audio player; recognition still works (TTS_ENGINE=none disables it explicitly)
frontend shows "Backend not reachable"	start the API first (python run.py api) — the UI never fakes a prediction
Docker build is slow	TensorFlow is a large wheel; the layer cache makes rebuilds cheap

9. Housekeeping

• logs/predictions.jsonl and logs/feedback/feedback.jsonl grow append-only; delete them to reset the error-review view (or use backend.inference.audit.clear_logs()).
• Evaluation results, trained model weights, the processed session-disjoint tensor and MediaPipe task bundles are kept in the project for a reproducible demo.
• Raw recordings and uploaded videos are not included in the repository; uploads are never retained unless ISL_PERSIST_UPLOADS=true.