# Evaluation artefacts

This folder documents **how** the project is evaluated; the measured outputs live in
`results/` (so that they stay next to the plots and CSVs the app serves).

## Protocol in one page

1. **Split first, everything else second.** The split is fixed and cached before any
   model sees data (`datasets/metadata/index_<protocol>.csv`).
   * `session_disjoint` (primary): pools the full INCLUDE-50 pool, allocates whole
     recording clusters to one split → 652 train / 152 val / **154 test**, 0 of 299
     clusters spanning a split, all 50 classes in every split.
   * `official`: AI4Bharat's own train/test files + a seeded, class-stratified,
     session-aware validation carve-out → 686 / 80 / 192; **199 of 299 clusters span
     the split**, so it is reported as leaking.
2. **One preprocessing for everyone.** `training/preprocess.py` writes
   `sequences_<protocol>.npz`; the three training scripts read the same file.
3. **Identical training conditions.** Same seed, augmentation, callbacks, epochs,
   batch size, loss, label smoothing. Each run writes
   `class_mapping.json`, `preprocessing_config.json`, `training_history.json`.
4. **Frozen evaluation.** `training/evaluate_models.py` scores `X_test` with no
   test-time augmentation and computes accuracy, macro/weighted P/R/F1, per-class
   metrics, confusion matrices, parameter counts and single-clip latency.
5. **Perturbation study.** `training/robustness.py` applies 14 conditions to the same
   frozen tensors and pushes them through the real models.
6. **Gate study.** `training/tune_threshold.py` sweeps the confidence threshold and
   reports coverage vs selective accuracy (the shipped 0.70 is a starting point, not a
   proven optimum).

## Where each number comes from

| question | artefact | command |
|---|---|---|
| how good is each model on the leak-free split? | `results/summary_metrics.csv`, `results/per_class_comparison.csv`, `results/confusion_*.png`, `results/model_comparison.png` | `python training/evaluate_models.py` |
| how much does the official split inflate? | `results/summary_metrics_official.csv`, `results/evaluation_summary_official.json` | `python training/evaluate_models.py --protocol official` |
| which classes fail, and how? | `results/per_class_comparison.csv`, `evaluation_summary_<protocol>.json` (top confusions, weakest/strongest classes) | same as above |
| how fast is inference? | `results/latency_comparison.png`, `latency_ms_*`, `throughput_fps` in `summary_metrics.csv` | same as above |
| what breaks the models? | `results/robustness_results.csv`, `results/robustness_degradation.png` | `python training/robustness.py` (both protocols) |
| what does the confidence gate cost? | `results/threshold_sweep.csv`, `results/threshold_sweep.png` | `python training/tune_threshold.py` |
| how live behaviour is verified | `tests/test_inference.py` (gate, smoothing, stability, duplicate suppression, TTS-failure safety) | `python -m pytest tests/test_inference.py -v` |

## Reporting rules used in this project

* Every number is produced by a run in this repository; nothing is copied from the
  literature or estimated. Where the original paper reports ≈94.5% for INCLUDE-50, this
  README states only the difference between the two protocols *measured here*.
* Baseline and proposed model are named explicitly; the comparison table is the
  headline result.
* Failures are reported as prominently as successes (13 of 50 classes have zero recall
  on the leak-free split; that list is in `docs/evaluation.md` §4).
* No claim is made about live-camera accuracy, unseen signers or continuous signing —
  those were not measured (`docs/evaluation.md` §8).