| # Evaluation artefacts |
|
|
| This folder documents **how** the project is evaluated; the measured outputs live in |
| `results/` (so that they stay next to the plots and CSVs the app serves). |
|
|
| ## Protocol in one page |
|
|
| 1. **Split first, everything else second.** The split is fixed and cached before any |
| model sees data (`datasets/metadata/index_<protocol>.csv`). |
| * `session_disjoint` (primary): pools the full INCLUDE-50 pool, allocates whole |
| recording clusters to one split → 652 train / 152 val / **154 test**, 0 of 299 |
| clusters spanning a split, all 50 classes in every split. |
| * `official`: AI4Bharat's own train/test files + a seeded, class-stratified, |
| session-aware validation carve-out → 686 / 80 / 192; **199 of 299 clusters span |
| the split**, so it is reported as leaking. |
| 2. **One preprocessing for everyone.** `training/preprocess.py` writes |
| `sequences_<protocol>.npz`; the three training scripts read the same file. |
| 3. **Identical training conditions.** Same seed, augmentation, callbacks, epochs, |
| batch size, loss, label smoothing. Each run writes |
| `class_mapping.json`, `preprocessing_config.json`, `training_history.json`. |
| 4. **Frozen evaluation.** `training/evaluate_models.py` scores `X_test` with no |
| test-time augmentation and computes accuracy, macro/weighted P/R/F1, per-class |
| metrics, confusion matrices, parameter counts and single-clip latency. |
| 5. **Perturbation study.** `training/robustness.py` applies 14 conditions to the same |
| frozen tensors and pushes them through the real models. |
| 6. **Gate study.** `training/tune_threshold.py` sweeps the confidence threshold and |
| reports coverage vs selective accuracy (the shipped 0.70 is a starting point, not a |
| proven optimum). |
| |
| ## Where each number comes from |
|
|
| | question | artefact | command | |
| |---|---|---| |
| | how good is each model on the leak-free split? | `results/summary_metrics.csv`, `results/per_class_comparison.csv`, `results/confusion_*.png`, `results/model_comparison.png` | `python training/evaluate_models.py` | |
| | how much does the official split inflate? | `results/summary_metrics_official.csv`, `results/evaluation_summary_official.json` | `python training/evaluate_models.py --protocol official` | |
| | which classes fail, and how? | `results/per_class_comparison.csv`, `evaluation_summary_<protocol>.json` (top confusions, weakest/strongest classes) | same as above | |
| | how fast is inference? | `results/latency_comparison.png`, `latency_ms_*`, `throughput_fps` in `summary_metrics.csv` | same as above | |
| | what breaks the models? | `results/robustness_results.csv`, `results/robustness_degradation.png` | `python training/robustness.py` (both protocols) | |
| | what does the confidence gate cost? | `results/threshold_sweep.csv`, `results/threshold_sweep.png` | `python training/tune_threshold.py` | |
| | how live behaviour is verified | `tests/test_inference.py` (gate, smoothing, stability, duplicate suppression, TTS-failure safety) | `python -m pytest tests/test_inference.py -v` | |
|
|
| ## Reporting rules used in this project |
|
|
| * Every number is produced by a run in this repository; nothing is copied from the |
| literature or estimated. Where the original paper reports ≈94.5% for INCLUDE-50, this |
| README states only the difference between the two protocols *measured here*. |
| * Baseline and proposed model are named explicitly; the comparison table is the |
| headline result. |
| * Failures are reported as prominently as successes (13 of 50 classes have zero recall |
| on the leak-free split; that list is in `docs/evaluation.md` §4). |
| * No claim is made about live-camera accuracy, unseen signers or continuous signing — |
| those were not measured (`docs/evaluation.md` §8). |
|
|
