docs/evaluation.md

# Evaluation

All numbers in this document were produced by this repository on this machine
(Python 3.13.14, TensorFlow 2.21, CPU-only, 2 vCPU) and can be regenerated with the
commands at the end of each section. Nothing is estimated, extrapolated or copied
from another paper.

---

## 1. Evaluation protocol

| item | value |
|---|---|
| dataset | AI4Bharat **INCLUDE-50** (release of pre-extracted MediaPipe keypoints) |
| classes | 50 (frozen from the official metadata, `datasets/metadata/vocabulary.json`) |
| clips used | 958 (the full INCLUDE-50 pool) |
| primary split | `session_disjoint` — train 652 / val 152 / test 154, seed 42 |
| secondary split | `official` — train 686 / val 80 / test 192 (AI4Bharat's own train/test files) |
| input | 30 frames × 225 features (75 landmarks × x,y,z, shoulder-normalised) |
| fairness contract | identical data, split, seed, class mapping, sequence length, augmentation, callbacks and test set for all three models; no test-time augmentation, no per-model tuning |
| metrics | top-1 accuracy, macro/weighted precision/recall/F1, per-class P/R/F1, confusion matrices, parameter count, single-clip latency, throughput |

### Why `session_disjoint` is the primary protocol

AI4Bharat's official train/test files reuse recording sessions across the split:
measured on this dataset, **199 of 299 recording clusters span train and test**
(167 near-duplicate clip pairs), so the official test set is partly seen data. The
official split is *not* signer-disjoint either — INCLUDE publishes no signer ids at
all, so a signer-disjoint evaluation is impossible with this release.

`session_disjoint` therefore pools the full INCLUDE-50 pool and allocates **whole
recording clusters** (clips whose file ids lie within a gap of 10 in the same class)
to a single split: **0 of 299 clusters span a split**, with the class distribution
kept stratified (all 50 classes in every split; 3–4 test clips per class). It is the
strongest reproducible substitute for signer-disjoint evaluation available from this
metadata. Both protocols are reported everywhere, and the difference between them is
itself a result (see §3).

---

## 2. Headline results — primary protocol (`session_disjoint`, test split, 154 clips)

| model | params | accuracy | macro P | macro R | **macro F1** | weighted F1 | latency (ms) | FPS |
|---|---|---|---|---|---|---|---|---|
| LSTM(128→64) | 245,426 | 0.6623 | 0.6019 | 0.6567 | 0.5969 | 0.6025 | 2.76 | 362.4 |
| GRU(128→64) | 188,338 | 0.6299 | 0.5621 | 0.6267 | 0.5588 | 0.5643 | 2.43 | 412.2 |
| **GRU+MHA (proposed)** | 263,026 | **0.7727** | **0.7214** | **0.7667** | **0.7182** | **0.7255** | 2.73 | 366.4 |

GRU+MHA has the highest accuracy and macro F1 in this snapshot; GRU has the lowest
model-only latency. The timing above is a warmed, batch-1 model call only; it excludes
MediaPipe and the rest of the live request. The live API now reports separate capture,
MediaPipe, preprocessing, buffer, model, gate, smoothing, server and browser timings.

Use `results/per_class_comparison.csv` for the authoritative current per-sign precision,
recall, within-sign accuracy, F1, support and measured model latency. The dashboard
shows the same per-sign comparison for LSTM, GRU and GRU+MHA. Each class has only 3–4
test examples, so these values are directional and one error changes recall sharply.

Files: `results/summary_metrics.csv`, `results/per_class_comparison.csv`,
`results/model_comparison.png`, `results/latency_comparison.png`,
`results/confusion_{lstm,gru,gru_mha}.png`, `results/evaluation_summary_session_disjoint.json`.

```bash
python training/evaluate_models.py --protocol session_disjoint
```

---

## 3. The same models on the official (leaking) split

| model | accuracy | macro F1 | weighted F1 |
|---|---|---|---|
| LSTM | generated by `run.py evaluate` | generated by `run.py evaluate` | generated by `run.py evaluate` |
| GRU | generated by `run.py evaluate` | generated by `run.py evaluate` | generated by `run.py evaluate` |
| GRU+MHA | generated by `run.py evaluate` | generated by `run.py evaluate` | generated by `run.py evaluate` |

| protocol | test clips | accuracy (GRU+MHA) | macro F1 |
|---|---|---|---|
| session_disjoint (leak-free) | 154 | 0.7727 | 0.7182 |
| official | Not evaluated in the current results bundle | — | — |
| **difference** |  | Not available | Not available |

Do not infer a protocol gap from the earlier results: this checkout's current result
bundle contains the session-disjoint evaluation only. Generate the official protocol
before comparing it. INCLUDE-50 does not publish signer ids, so this still does not
measure signer-independent accuracy.

```bash
python training/evaluate_models.py --protocol official
```

---

## 4. Per-class behaviour and failure analysis (proposed model, primary protocol)

* Per-class precision, recall, F1, accuracy, support and model-only latency are in `results/per_class_comparison.csv`; inspect that generated file rather than relying on a hard-coded list in prose.
* The test set has only 3–4 clips per class. A single flipped clip can move recall by 0.25–0.33, so class metrics are directional, not production guarantees.

---

## 5. Confidence gate (why 0.70 is a starting point, not an optimum)

The gate converts low-confidence outputs into the `UNCERTAIN` state. Sweeping it on
the validation split (`results/threshold_sweep.csv`, `results/threshold_sweep.png`)
measures the coverage/accuracy trade-off; the shipped default was chosen
conservatively *before* this sweep and is deliberately configurable
(`ISL_CONFIDENCE_THRESHOLD`, or per profile within a ±0.2 band).

Proposed model, validation split (152 clips):

| threshold | coverage | selective accuracy | pessimistic accuracy* |
|---|---|---|---|
| 0.50 | 0.803 | 0.8279 | 0.6645 |
| 0.60 | 0.664 | 0.8317 | 0.5526 |
| **0.70 (shipped)** | **0.579** | **0.8523** | **0.4934** |
| 0.75 | 0.539 | 0.8659 | 0.4671 |
| 0.85 | 0.441 | 0.8806 | 0.3882 |
| 0.95 | 0.118 | 1.0000 | 0.1184 |

\* pessimistic accuracy counts every gated-out clip as wrong — i.e. the accuracy if
the user never re-attempts an uncertain sign.

Reading: at 0.70 the system commits to ~58% of validation clips and is right ~85% of the time on
those; lowering to 0.50 raises coverage to ~80% but drops selective accuracy to ~83%;
raising to 0.85 commits to ~44% of clips. **0.70 sits on the conservative side of
the knee and is not claimed to be optimal for any particular user or deployment** —
tune it with this sweep and document your own choice.

```bash
python training/tune_threshold.py            # val split, both protocols
```

---

## 6. Robustness (real models, perturbed real test tensors)

Accuracy under each perturbation (14 conditions × 3 models; full table in
`results/robustness_results.csv`, plots in `results/robustness_degradation.png`).

Primary protocol (`session_disjoint`, 154 clips):

| condition | LSTM | GRU | **GRU+MHA** | Δ (proposed) |
|---|---|---|---|---|
| clean | 0.6623 | 0.6299 | **0.7727** | — |
| landmark dropout 10% | 0.6753 | 0.6429 | **0.7532** | −0.0195 |
| landmark dropout 30% | 0.5844 | 0.5584 | **0.7208** | −0.0519 |
| left hand removed | 0.2273 | 0.2208 | **0.5325** | −0.2403 |
| right hand removed | 0.1753 | 0.1364 | **0.2597** | −0.5130 |
| both hands removed | 0.0390 | 0.0649 | **0.1169** | −0.6558 |
| Gaussian jitter σ=0.10 | 0.6883 | 0.6299 | **0.7662** | −0.0065 |
| Gaussian jitter σ=0.25 | 0.6818 | 0.5844 | **0.7338** | −0.0390 |
| Gaussian jitter σ=0.50 | 0.5844 | 0.5130 | **0.6818** | −0.0909 |
| frame drop 20% | 0.6494 | 0.6234 | **0.7338** | −0.0390 |
| frame drop 40% | 0.6429 | 0.5844 | **0.7468** | −0.0260 |
| frame drop 60% | 0.5584 | 0.5260 | **0.6883** | −0.0844 |
| uniform 2× subsampling | 0.6688 | 0.6364 | **0.7532** | −0.0195 |
| presence mask zeroed | 0.6623 | 0.6299 | **0.7727** | 0.0000 (coordinate-only input) |

Findings:

1. **Hands carry the signal.** Removing both hands collapses accuracy to 0.03–0.12
   for every model; removing one hand costs 0.31–0.53. The proposed model degrades
   least (left hand −0.31 vs −0.47 LSTM / −0.47 GRU). Practical consequence: the
   hand-detection rate of the extractor, not the network, is the main risk in
   deployment.
2. **Temporal perturbations are tolerated in this test**: dropping 40% of frames costs
   the proposed model about 0.026 accuracy; 60% costs about 0.084.
3. **Coordinate noise is tolerated up to σ≈0.25 shoulder-widths**; at σ=0.5 it still
   beats both baselines.
4. `mask_missing` is a no-op because this build trains with `MASK_ENABLED=False`
   (coordinates only). The condition is kept so the experiment is meaningful when the
   mask build is enabled.
5. In the *pessimistic* sense the gate helps: under `both_hands_missing` 142 of 154
   clips fall below the 0.70 gate for the proposed model, so a live user would see
   `UNCERTAIN` rather than a confidently wrong sign in most cases. That is the
   intended behaviour of the confidence-aware design.

The current results bundle does not include the robustness experiment on the official
split; do not infer its ranking from the session-disjoint table.

```bash
python training/robustness.py --protocol session_disjoint
python training/robustness.py --protocol official
```

---

## 7. Deployment-facing numbers

| quantity | measured |
|---|---|
| warmed model latency (30 frames, batch 1) | 2.76 / 2.43 / 2.73 ms for LSTM / GRU / GRU+MHA |
| measured model throughput, batch 1 | 362 / 412 / 366 sequences/s respectively |
| end-to-end live path | MediaPipe/capture/API/browser work is additional and reported separately by the live stage telemetry; do not present model-only timing as full response latency |
| model size on disk | 2.3–3.2 MB (`.keras`) |
| end-to-end live path | MediaPipe extraction dominates on CPU; the model is ~60 ms of a much longer frame budget — the buffer, not the network, limits the emitted-sign rate |

Real-time behaviour (buffer → gate → smoothing → caption) is exercised by
`python backend/inference/realtime.py`; the smoothing/gate logic is unit-tested in
`tests/test_inference.py`.

---

## 8. What is *not* measured

Stated plainly so that the numbers above are not over-read:

* **No live-camera accuracy study.** Every accuracy figure comes from INCLUDE
  keypoints. A webcam introduces different frame rates, compression, lighting,
  camera geometry and detector behaviour. The geometry calibration
  (`X_AXIS_SCALE`, see `docs/dataset_audit.md`) aligns the two coordinate systems,
  but live accuracy was not measured and may differ.
* **No unseen-signer or dialect study.** INCLUDE publishes no signer ids, so
  signer-generalisation cannot be quantified here.
* **No continuous-stream or sentence-level evaluation.** The system recognises
  isolated signs; segmentation into sentences/grammar was not attempted.
* **No user study** of the override/personalisation workflow; the audit log and the
  error-review UI make such a study possible, they do not constitute one.
* **No claim that 0.70 is optimal**, that the 30-frame window is optimal, or that a
  particular augmentation setting is best — only that all three models were treated
  identically and that the sweeps are reproducible.