docs/evaluation.md

Evaluation

All numbers in this document were produced by this repository on this machine (Python 3.13.14, TensorFlow 2.21, CPU-only, 2 vCPU) and can be regenerated with the commands at the end of each section. Nothing is estimated, extrapolated or copied from another paper.

---

1. Evaluation protocol

item	value
dataset	AI4Bharat INCLUDE-50 (release of pre-extracted MediaPipe keypoints)
classes	50 (frozen from the official metadata, datasets/metadata/vocabulary.json)
clips used	958 (the full INCLUDE-50 pool)
primary split	session_disjoint — train 652 / val 152 / test 154, seed 42
secondary split	official — train 686 / val 80 / test 192 (AI4Bharat's own train/test files)
input	30 frames × 225 features (75 landmarks × x,y,z, shoulder-normalised)
fairness contract	identical data, split, seed, class mapping, sequence length, augmentation, callbacks and test set for all three models; no test-time augmentation, no per-model tuning
metrics	top-1 accuracy, macro/weighted precision/recall/F1, per-class P/R/F1, confusion matrices, parameter count, single-clip latency, throughput

Why session_disjoint is the primary protocol

AI4Bharat's official train/test files reuse recording sessions across the split: measured on this dataset, 199 of 299 recording clusters span train and test (167 near-duplicate clip pairs), so the official test set is partly seen data. The official split is not signer-disjoint either — INCLUDE publishes no signer ids at all, so a signer-disjoint evaluation is impossible with this release.

session_disjoint therefore pools the full INCLUDE-50 pool and allocates whole recording clusters (clips whose file ids lie within a gap of 10 in the same class) to a single split: 0 of 299 clusters span a split, with the class distribution kept stratified (all 50 classes in every split; 3–4 test clips per class). It is the strongest reproducible substitute for signer-disjoint evaluation available from this metadata. Both protocols are reported everywhere, and the difference between them is itself a result (see §3).

---

2. Headline results — primary protocol (session_disjoint, test split, 154 clips)

model	params	accuracy	macro P	macro R	macro F1	weighted F1	latency (ms)	FPS
LSTM(128→64)	245,426	0.6623	0.6019	0.6567	0.5969	0.6025	2.76	362.4
GRU(128→64)	188,338	0.6299	0.5621	0.6267	0.5588	0.5643	2.43	412.2
GRU+MHA (proposed)	263,026	0.7727	0.7214	0.7667	0.7182	0.7255	2.73	366.4

GRU+MHA has the highest accuracy and macro F1 in this snapshot; GRU has the lowest model-only latency. The timing above is a warmed, batch-1 model call only; it excludes MediaPipe and the rest of the live request. The live API now reports separate capture, MediaPipe, preprocessing, buffer, model, gate, smoothing, server and browser timings.

Use results/per_class_comparison.csv for the authoritative current per-sign precision, recall, within-sign accuracy, F1, support and measured model latency. The dashboard shows the same per-sign comparison for LSTM, GRU and GRU+MHA. Each class has only 3–4 test examples, so these values are directional and one error changes recall sharply.

Files: results/summary_metrics.csv, results/per_class_comparison.csv, results/model_comparison.png, results/latency_comparison.png, results/confusion_{lstm,gru,gru_mha}.png, results/evaluation_summary_session_disjoint.json.

python training/evaluate_models.py --protocol session_disjoint

---

3. The same models on the official (leaking) split

model	accuracy	macro F1	weighted F1
LSTM	generated by run.py evaluate	generated by run.py evaluate	generated by run.py evaluate
GRU	generated by run.py evaluate	generated by run.py evaluate	generated by run.py evaluate
GRU+MHA	generated by run.py evaluate	generated by run.py evaluate	generated by run.py evaluate

protocol	test clips	accuracy (GRU+MHA)	macro F1
session_disjoint (leak-free)	154	0.7727	0.7182
official	Not evaluated in the current results bundle	—	—
difference		Not available	Not available

Do not infer a protocol gap from the earlier results: this checkout's current result bundle contains the session-disjoint evaluation only. Generate the official protocol before comparing it. INCLUDE-50 does not publish signer ids, so this still does not measure signer-independent accuracy.

python training/evaluate_models.py --protocol official

---

4. Per-class behaviour and failure analysis (proposed model, primary protocol)

• Per-class precision, recall, F1, accuracy, support and model-only latency are in results/per_class_comparison.csv; inspect that generated file rather than relying on a hard-coded list in prose.
• The test set has only 3–4 clips per class. A single flipped clip can move recall by 0.25–0.33, so class metrics are directional, not production guarantees.

---

5. Confidence gate (why 0.70 is a starting point, not an optimum)

The gate converts low-confidence outputs into the UNCERTAIN state. Sweeping it on the validation split (results/threshold_sweep.csv, results/threshold_sweep.png) measures the coverage/accuracy trade-off; the shipped default was chosen conservatively before this sweep and is deliberately configurable (ISL_CONFIDENCE_THRESHOLD, or per profile within a ±0.2 band).

Proposed model, validation split (152 clips):

threshold	coverage	selective accuracy	pessimistic accuracy*
0.50	0.803	0.8279	0.6645
0.60	0.664	0.8317	0.5526
0.70 (shipped)	0.579	0.8523	0.4934
0.75	0.539	0.8659	0.4671
0.85	0.441	0.8806	0.3882
0.95	0.118	1.0000	0.1184

* pessimistic accuracy counts every gated-out clip as wrong — i.e. the accuracy if the user never re-attempts an uncertain sign.

Reading: at 0.70 the system commits to ~58% of validation clips and is right ~85% of the time on those; lowering to 0.50 raises coverage to ~80% but drops selective accuracy to ~83%; raising to 0.85 commits to ~44% of clips. 0.70 sits on the conservative side of the knee and is not claimed to be optimal for any particular user or deployment — tune it with this sweep and document your own choice.

python training/tune_threshold.py            # val split, both protocols

---

6. Robustness (real models, perturbed real test tensors)

Accuracy under each perturbation (14 conditions × 3 models; full table in results/robustness_results.csv, plots in results/robustness_degradation.png).

Primary protocol (session_disjoint, 154 clips):

condition	LSTM	GRU	GRU+MHA	Δ (proposed)
clean	0.6623	0.6299	0.7727	—
landmark dropout 10%	0.6753	0.6429	0.7532	−0.0195
landmark dropout 30%	0.5844	0.5584	0.7208	−0.0519
left hand removed	0.2273	0.2208	0.5325	−0.2403
right hand removed	0.1753	0.1364	0.2597	−0.5130
both hands removed	0.0390	0.0649	0.1169	−0.6558
Gaussian jitter σ=0.10	0.6883	0.6299	0.7662	−0.0065
Gaussian jitter σ=0.25	0.6818	0.5844	0.7338	−0.0390
Gaussian jitter σ=0.50	0.5844	0.5130	0.6818	−0.0909
frame drop 20%	0.6494	0.6234	0.7338	−0.0390
frame drop 40%	0.6429	0.5844	0.7468	−0.0260
frame drop 60%	0.5584	0.5260	0.6883	−0.0844
uniform 2× subsampling	0.6688	0.6364	0.7532	−0.0195
presence mask zeroed	0.6623	0.6299	0.7727	0.0000 (coordinate-only input)

Findings:

1. Hands carry the signal. Removing both hands collapses accuracy to 0.03–0.12 for every model; removing one hand costs 0.31–0.53. The proposed model degrades least (left hand −0.31 vs −0.47 LSTM / −0.47 GRU). Practical consequence: the hand-detection rate of the extractor, not the network, is the main risk in deployment.
2. Temporal perturbations are tolerated in this test: dropping 40% of frames costs the proposed model about 0.026 accuracy; 60% costs about 0.084.
3. Coordinate noise is tolerated up to σ≈0.25 shoulder-widths; at σ=0.5 it still beats both baselines.
4. mask_missing is a no-op because this build trains with MASK_ENABLED=False (coordinates only). The condition is kept so the experiment is meaningful when the mask build is enabled.
5. In the pessimistic sense the gate helps: under both_hands_missing 142 of 154 clips fall below the 0.70 gate for the proposed model, so a live user would see UNCERTAIN rather than a confidently wrong sign in most cases. That is the intended behaviour of the confidence-aware design.

The current results bundle does not include the robustness experiment on the official split; do not infer its ranking from the session-disjoint table.

python training/robustness.py --protocol session_disjoint
python training/robustness.py --protocol official

---

7. Deployment-facing numbers

quantity	measured
warmed model latency (30 frames, batch 1)	2.76 / 2.43 / 2.73 ms for LSTM / GRU / GRU+MHA
measured model throughput, batch 1	362 / 412 / 366 sequences/s respectively
end-to-end live path	MediaPipe/capture/API/browser work is additional and reported separately by the live stage telemetry; do not present model-only timing as full response latency
model size on disk	2.3–3.2 MB (.keras)
end-to-end live path	MediaPipe extraction dominates on CPU; the model is ~60 ms of a much longer frame budget — the buffer, not the network, limits the emitted-sign rate

Real-time behaviour (buffer → gate → smoothing → caption) is exercised by python backend/inference/realtime.py; the smoothing/gate logic is unit-tested in tests/test_inference.py.

---

8. What is not measured

Stated plainly so that the numbers above are not over-read:

• No live-camera accuracy study. Every accuracy figure comes from INCLUDE keypoints. A webcam introduces different frame rates, compression, lighting, camera geometry and detector behaviour. The geometry calibration (X_AXIS_SCALE, see docs/dataset_audit.md) aligns the two coordinate systems, but live accuracy was not measured and may differ.
• No unseen-signer or dialect study. INCLUDE publishes no signer ids, so signer-generalisation cannot be quantified here.
• No continuous-stream or sentence-level evaluation. The system recognises isolated signs; segmentation into sentences/grammar was not attempted.
• No user study of the override/personalisation workflow; the audit log and the error-review UI make such a study possible, they do not constitute one.
• No claim that 0.70 is optimal, that the 30-frame window is optimal, or that a particular augmentation setting is best — only that all three models were treated identically and that the sweeps are reproducible.