File size: 3,701 Bytes
e2d95ed | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 | # Evaluation artefacts
This folder documents **how** the project is evaluated; the measured outputs live in
`results/` (so that they stay next to the plots and CSVs the app serves).
## Protocol in one page
1. **Split first, everything else second.** The split is fixed and cached before any
model sees data (`datasets/metadata/index_<protocol>.csv`).
* `session_disjoint` (primary): pools the full INCLUDE-50 pool, allocates whole
recording clusters to one split → 652 train / 152 val / **154 test**, 0 of 299
clusters spanning a split, all 50 classes in every split.
* `official`: AI4Bharat's own train/test files + a seeded, class-stratified,
session-aware validation carve-out → 686 / 80 / 192; **199 of 299 clusters span
the split**, so it is reported as leaking.
2. **One preprocessing for everyone.** `training/preprocess.py` writes
`sequences_<protocol>.npz`; the three training scripts read the same file.
3. **Identical training conditions.** Same seed, augmentation, callbacks, epochs,
batch size, loss, label smoothing. Each run writes
`class_mapping.json`, `preprocessing_config.json`, `training_history.json`.
4. **Frozen evaluation.** `training/evaluate_models.py` scores `X_test` with no
test-time augmentation and computes accuracy, macro/weighted P/R/F1, per-class
metrics, confusion matrices, parameter counts and single-clip latency.
5. **Perturbation study.** `training/robustness.py` applies 14 conditions to the same
frozen tensors and pushes them through the real models.
6. **Gate study.** `training/tune_threshold.py` sweeps the confidence threshold and
reports coverage vs selective accuracy (the shipped 0.70 is a starting point, not a
proven optimum).
## Where each number comes from
| question | artefact | command |
|---|---|---|
| how good is each model on the leak-free split? | `results/summary_metrics.csv`, `results/per_class_comparison.csv`, `results/confusion_*.png`, `results/model_comparison.png` | `python training/evaluate_models.py` |
| how much does the official split inflate? | `results/summary_metrics_official.csv`, `results/evaluation_summary_official.json` | `python training/evaluate_models.py --protocol official` |
| which classes fail, and how? | `results/per_class_comparison.csv`, `evaluation_summary_<protocol>.json` (top confusions, weakest/strongest classes) | same as above |
| how fast is inference? | `results/latency_comparison.png`, `latency_ms_*`, `throughput_fps` in `summary_metrics.csv` | same as above |
| what breaks the models? | `results/robustness_results.csv`, `results/robustness_degradation.png` | `python training/robustness.py` (both protocols) |
| what does the confidence gate cost? | `results/threshold_sweep.csv`, `results/threshold_sweep.png` | `python training/tune_threshold.py` |
| how live behaviour is verified | `tests/test_inference.py` (gate, smoothing, stability, duplicate suppression, TTS-failure safety) | `python -m pytest tests/test_inference.py -v` |
## Reporting rules used in this project
* Every number is produced by a run in this repository; nothing is copied from the
literature or estimated. Where the original paper reports ≈94.5% for INCLUDE-50, this
README states only the difference between the two protocols *measured here*.
* Baseline and proposed model are named explicitly; the comparison table is the
headline result.
* Failures are reported as prominently as successes (13 of 50 classes have zero recall
on the leak-free split; that list is in `docs/evaluation.md` §4).
* No claim is made about live-camera accuracy, unseen signers or continuous signing —
those were not measured (`docs/evaluation.md` §8).
|