| # Evaluation |
|
|
| All numbers in this document were produced by this repository on this machine |
| (Python 3.13.14, TensorFlow 2.21, CPU-only, 2 vCPU) and can be regenerated with the |
| commands at the end of each section. Nothing is estimated, extrapolated or copied |
| from another paper. |
|
|
| --- |
|
|
| ## 1. Evaluation protocol |
|
|
| | item | value | |
| |---|---| |
| | dataset | AI4Bharat **INCLUDE-50** (release of pre-extracted MediaPipe keypoints) | |
| | classes | 50 (frozen from the official metadata, `datasets/metadata/vocabulary.json`) | |
| | clips used | 958 (the full INCLUDE-50 pool) | |
| | primary split | `session_disjoint` β train 652 / val 152 / test 154, seed 42 | |
| | secondary split | `official` β train 686 / val 80 / test 192 (AI4Bharat's own train/test files) | |
| | input | 30 frames Γ 225 features (75 landmarks Γ x,y,z, shoulder-normalised) | |
| | fairness contract | identical data, split, seed, class mapping, sequence length, augmentation, callbacks and test set for all three models; no test-time augmentation, no per-model tuning | |
| | metrics | top-1 accuracy, macro/weighted precision/recall/F1, per-class P/R/F1, confusion matrices, parameter count, single-clip latency, throughput | |
|
|
| ### Why `session_disjoint` is the primary protocol |
| |
| AI4Bharat's official train/test files reuse recording sessions across the split: |
| measured on this dataset, **199 of 299 recording clusters span train and test** |
| (167 near-duplicate clip pairs), so the official test set is partly seen data. The |
| official split is *not* signer-disjoint either β INCLUDE publishes no signer ids at |
| all, so a signer-disjoint evaluation is impossible with this release. |
| |
| `session_disjoint` therefore pools the full INCLUDE-50 pool and allocates **whole |
| recording clusters** (clips whose file ids lie within a gap of 10 in the same class) |
| to a single split: **0 of 299 clusters span a split**, with the class distribution |
| kept stratified (all 50 classes in every split; 3β4 test clips per class). It is the |
| strongest reproducible substitute for signer-disjoint evaluation available from this |
| metadata. Both protocols are reported everywhere, and the difference between them is |
| itself a result (see Β§3). |
|
|
| --- |
|
|
| ## 2. Headline results β primary protocol (`session_disjoint`, test split, 154 clips) |
| |
| | model | params | accuracy | macro P | macro R | **macro F1** | weighted F1 | latency (ms) | FPS | |
| |---|---|---|---|---|---|---|---|---| |
| | LSTM(128β64) | 245,426 | 0.6623 | 0.6019 | 0.6567 | 0.5969 | 0.6025 | 2.76 | 362.4 | |
| | GRU(128β64) | 188,338 | 0.6299 | 0.5621 | 0.6267 | 0.5588 | 0.5643 | 2.43 | 412.2 | |
| | **GRU+MHA (proposed)** | 263,026 | **0.7727** | **0.7214** | **0.7667** | **0.7182** | **0.7255** | 2.73 | 366.4 | |
| |
| GRU+MHA has the highest accuracy and macro F1 in this snapshot; GRU has the lowest |
| model-only latency. The timing above is a warmed, batch-1 model call only; it excludes |
| MediaPipe and the rest of the live request. The live API now reports separate capture, |
| MediaPipe, preprocessing, buffer, model, gate, smoothing, server and browser timings. |
| |
| Use `results/per_class_comparison.csv` for the authoritative current per-sign precision, |
| recall, within-sign accuracy, F1, support and measured model latency. The dashboard |
| shows the same per-sign comparison for LSTM, GRU and GRU+MHA. Each class has only 3β4 |
| test examples, so these values are directional and one error changes recall sharply. |
| |
| Files: `results/summary_metrics.csv`, `results/per_class_comparison.csv`, |
| `results/model_comparison.png`, `results/latency_comparison.png`, |
| `results/confusion_{lstm,gru,gru_mha}.png`, `results/evaluation_summary_session_disjoint.json`. |
|
|
| ```bash |
| python training/evaluate_models.py --protocol session_disjoint |
| ``` |
|
|
| --- |
|
|
| ## 3. The same models on the official (leaking) split |
|
|
| | model | accuracy | macro F1 | weighted F1 | |
| |---|---|---|---| |
| | LSTM | generated by `run.py evaluate` | generated by `run.py evaluate` | generated by `run.py evaluate` | |
| | GRU | generated by `run.py evaluate` | generated by `run.py evaluate` | generated by `run.py evaluate` | |
| | GRU+MHA | generated by `run.py evaluate` | generated by `run.py evaluate` | generated by `run.py evaluate` | |
|
|
| | protocol | test clips | accuracy (GRU+MHA) | macro F1 | |
| |---|---|---|---| |
| | session_disjoint (leak-free) | 154 | 0.7727 | 0.7182 | |
| | official | Not evaluated in the current results bundle | β | β | |
| | **difference** | | Not available | Not available | |
| |
| Do not infer a protocol gap from the earlier results: this checkout's current result |
| bundle contains the session-disjoint evaluation only. Generate the official protocol |
| before comparing it. INCLUDE-50 does not publish signer ids, so this still does not |
| measure signer-independent accuracy. |
| |
| ```bash |
| python training/evaluate_models.py --protocol official |
| ``` |
| |
| --- |
| |
| ## 4. Per-class behaviour and failure analysis (proposed model, primary protocol) |
| |
| * Per-class precision, recall, F1, accuracy, support and model-only latency are in `results/per_class_comparison.csv`; inspect that generated file rather than relying on a hard-coded list in prose. |
| * The test set has only 3β4 clips per class. A single flipped clip can move recall by 0.25β0.33, so class metrics are directional, not production guarantees. |
| |
| --- |
| |
| ## 5. Confidence gate (why 0.70 is a starting point, not an optimum) |
| |
| The gate converts low-confidence outputs into the `UNCERTAIN` state. Sweeping it on |
| the validation split (`results/threshold_sweep.csv`, `results/threshold_sweep.png`) |
| measures the coverage/accuracy trade-off; the shipped default was chosen |
| conservatively *before* this sweep and is deliberately configurable |
| (`ISL_CONFIDENCE_THRESHOLD`, or per profile within a Β±0.2 band). |
| |
| Proposed model, validation split (152 clips): |
| |
| | threshold | coverage | selective accuracy | pessimistic accuracy* | |
| |---|---|---|---| |
| | 0.50 | 0.803 | 0.8279 | 0.6645 | |
| | 0.60 | 0.664 | 0.8317 | 0.5526 | |
| | **0.70 (shipped)** | **0.579** | **0.8523** | **0.4934** | |
| | 0.75 | 0.539 | 0.8659 | 0.4671 | |
| | 0.85 | 0.441 | 0.8806 | 0.3882 | |
| | 0.95 | 0.118 | 1.0000 | 0.1184 | |
| |
| \* pessimistic accuracy counts every gated-out clip as wrong β i.e. the accuracy if |
| the user never re-attempts an uncertain sign. |
| |
| Reading: at 0.70 the system commits to ~58% of validation clips and is right ~85% of the time on |
| those; lowering to 0.50 raises coverage to ~80% but drops selective accuracy to ~83%; |
| raising to 0.85 commits to ~44% of clips. **0.70 sits on the conservative side of |
| the knee and is not claimed to be optimal for any particular user or deployment** β |
| tune it with this sweep and document your own choice. |
| |
| ```bash |
| python training/tune_threshold.py # val split, both protocols |
| ``` |
| |
| --- |
| |
| ## 6. Robustness (real models, perturbed real test tensors) |
| |
| Accuracy under each perturbation (14 conditions Γ 3 models; full table in |
| `results/robustness_results.csv`, plots in `results/robustness_degradation.png`). |
| |
| Primary protocol (`session_disjoint`, 154 clips): |
|
|
| | condition | LSTM | GRU | **GRU+MHA** | Ξ (proposed) | |
| |---|---|---|---|---| |
| | clean | 0.6623 | 0.6299 | **0.7727** | β | |
| | landmark dropout 10% | 0.6753 | 0.6429 | **0.7532** | β0.0195 | |
| | landmark dropout 30% | 0.5844 | 0.5584 | **0.7208** | β0.0519 | |
| | left hand removed | 0.2273 | 0.2208 | **0.5325** | β0.2403 | |
| | right hand removed | 0.1753 | 0.1364 | **0.2597** | β0.5130 | |
| | both hands removed | 0.0390 | 0.0649 | **0.1169** | β0.6558 | |
| | Gaussian jitter Ο=0.10 | 0.6883 | 0.6299 | **0.7662** | β0.0065 | |
| | Gaussian jitter Ο=0.25 | 0.6818 | 0.5844 | **0.7338** | β0.0390 | |
| | Gaussian jitter Ο=0.50 | 0.5844 | 0.5130 | **0.6818** | β0.0909 | |
| | frame drop 20% | 0.6494 | 0.6234 | **0.7338** | β0.0390 | |
| | frame drop 40% | 0.6429 | 0.5844 | **0.7468** | β0.0260 | |
| | frame drop 60% | 0.5584 | 0.5260 | **0.6883** | β0.0844 | |
| | uniform 2Γ subsampling | 0.6688 | 0.6364 | **0.7532** | β0.0195 | |
| | presence mask zeroed | 0.6623 | 0.6299 | **0.7727** | 0.0000 (coordinate-only input) | |
|
|
| Findings: |
|
|
| 1. **Hands carry the signal.** Removing both hands collapses accuracy to 0.03β0.12 |
| for every model; removing one hand costs 0.31β0.53. The proposed model degrades |
| least (left hand β0.31 vs β0.47 LSTM / β0.47 GRU). Practical consequence: the |
| hand-detection rate of the extractor, not the network, is the main risk in |
| deployment. |
| 2. **Temporal perturbations are tolerated in this test**: dropping 40% of frames costs |
| the proposed model about 0.026 accuracy; 60% costs about 0.084. |
| 3. **Coordinate noise is tolerated up to Οβ0.25 shoulder-widths**; at Ο=0.5 it still |
| beats both baselines. |
| 4. `mask_missing` is a no-op because this build trains with `MASK_ENABLED=False` |
| (coordinates only). The condition is kept so the experiment is meaningful when the |
| mask build is enabled. |
| 5. In the *pessimistic* sense the gate helps: under `both_hands_missing` 142 of 154 |
| clips fall below the 0.70 gate for the proposed model, so a live user would see |
| `UNCERTAIN` rather than a confidently wrong sign in most cases. That is the |
| intended behaviour of the confidence-aware design. |
|
|
| The current results bundle does not include the robustness experiment on the official |
| split; do not infer its ranking from the session-disjoint table. |
|
|
| ```bash |
| python training/robustness.py --protocol session_disjoint |
| python training/robustness.py --protocol official |
| ``` |
|
|
| --- |
|
|
| ## 7. Deployment-facing numbers |
|
|
| | quantity | measured | |
| |---|---| |
| | warmed model latency (30 frames, batch 1) | 2.76 / 2.43 / 2.73 ms for LSTM / GRU / GRU+MHA | |
| | measured model throughput, batch 1 | 362 / 412 / 366 sequences/s respectively | |
| | end-to-end live path | MediaPipe/capture/API/browser work is additional and reported separately by the live stage telemetry; do not present model-only timing as full response latency | |
| | model size on disk | 2.3β3.2 MB (`.keras`) | |
| | end-to-end live path | MediaPipe extraction dominates on CPU; the model is ~60 ms of a much longer frame budget β the buffer, not the network, limits the emitted-sign rate | |
|
|
| Real-time behaviour (buffer β gate β smoothing β caption) is exercised by |
| `python backend/inference/realtime.py`; the smoothing/gate logic is unit-tested in |
| `tests/test_inference.py`. |
|
|
| --- |
|
|
| ## 8. What is *not* measured |
|
|
| Stated plainly so that the numbers above are not over-read: |
|
|
| * **No live-camera accuracy study.** Every accuracy figure comes from INCLUDE |
| keypoints. A webcam introduces different frame rates, compression, lighting, |
| camera geometry and detector behaviour. The geometry calibration |
| (`X_AXIS_SCALE`, see `docs/dataset_audit.md`) aligns the two coordinate systems, |
| but live accuracy was not measured and may differ. |
| * **No unseen-signer or dialect study.** INCLUDE publishes no signer ids, so |
| signer-generalisation cannot be quantified here. |
| * **No continuous-stream or sentence-level evaluation.** The system recognises |
| isolated signs; segmentation into sentences/grammar was not attempted. |
| * **No user study** of the override/personalisation workflow; the audit log and the |
| error-review UI make such a study possible, they do not constitute one. |
| * **No claim that 0.70 is optimal**, that the 30-frame window is optimal, or that a |
| particular augmentation setting is best β only that all three models were treated |
| identically and that the sweeps are reproducible. |
|
|
