docs/limitations.md

Limitations

Every item below is a measured property of this project or of the INCLUDE-50 dataset, not a hypothetical. Where a number appears it was produced by training/audit_dataset.py and is reproducible from results/dataset_audit_summary.json.

1. Scope

• This is an isolated sign/command recognition prototype. It classifies one sign at a time from a short clip. It is not a sign-language translator: it does not model ISL grammar, sentence structure, fingerspelling, non-manual markers (facial expression, mouth morphemes, head tilt) or spatial referencing.
• The system recognises exactly the classes present in the trained model's vocabulary (INCLUDE-50: 50 signs). Anything outside that vocabulary either produces a wrong label or should be caught by the confidence gate - it cannot produce a sensible "I don't know that sign" answer beyond the UNCERTAIN state.
• This is not a medical, legal, emergency or accessibility-certified system, and must not be used as the sole channel of communication.

2. Dataset limitations (measured)

limitation	measurement
Signer metadata absent	INCLUDE publishes no signer ids, so signer-disjoint evaluation is impossible. Restated in docs/dataset_audit.md §8.
Session leakage in the official split	199 of 299 recording clusters (bursts of consecutive file ids) span more than one side of the official train/test split; 63% of clips share an id-block with the other side. Benchmark numbers computed on the official split are therefore optimistic.
Our mitigation	The project's primary protocol (session_disjoint) keeps whole recording clusters on one side: 0 of 299 clusters span a split. Both numbers will be reported side by side.
Very small test set per class	The session-disjoint split yields 3-4 test clips per class (154 total). Per-class precision/recall/F1 therefore has wide confidence intervals and must not be read to three decimal places.
One class has no test clip in the official split	DRY appears only in the official train file; it is excluded from official-split per-class test metrics rather than silently dropped.
Demographic and geographic bias	INCLUDE was recorded by a small number of experienced signers in India, in a studio-like setting. ISL itself varies regionally. Performance for other signers, dialects, and skin tones is unknown and will very likely be lower.
No signer-disjoint generalisation evidence	Because signer ids are unavailable, no claim of signer-independent performance can be made.
Released keypoints are 2-D + z	x/y are image coordinates; z is MediaPipe's relative depth, which is far noisier. The 225-feature layout is preserved, but the effective spatial information is mostly 2-D.
Axis anisotropy in the release	The official keypoints were produced through a width/height-swapped resize, compressing x by ~3.16x. Measured and corrected (config.X_AXIS_SCALE); residual uncertainty is the spread between the two anatomical priors (3.00 and 3.26). Any result computed without the correction would train a squashed body model that does not transfer to the webcam.
No FPS or duration metadata	Not present in the keypoint release, so temporal statistics are reported in frames.

3. Modelling limitations

• Landmark-based models discard appearance: they cannot use facial expression, finger contact with a surface, or any cue MediaPipe cannot see. Fast or partially occluded signs lose most of their signal.
• The three architectures (LSTM, GRU, GRU+attention) are all small recurrent models trained from scratch on ~650 clips. No pretrained backbone is used, so absolute accuracy is bounded by data size, not by architecture choice alone.
• Augmentation changes the landmark statistics on purpose. Aggressive settings can make the validation distribution differ from the test distribution; the augmentation policy must be identical across the three models or the comparison is void (enforced in backend/preprocessing/sequence.augment, called from one place).
• Robustness experiments perturbe landmarks directly. They measure sensitivity to feature degradation, which is a proxy for - not a substitute for - real occlusion, motion blur and bad lighting.

4. Deployment and human-factors limitations

• Webcam input is distributionally different from the studio recordings (camera height, framing, distance, lighting, background clutter). No cross-domain claim is made.
• Confidence is a softmax score, not a calibrated probability. It is used as a gate, and the threshold must be tuned on validation data (training/tune_threshold.py), never assumed optimal. The default 0.70 is a starting point, not a validated choice.
• Temporal smoothing introduces latency: a prediction needs STABLE_FRAMES_REQUIRED consistent frames before it is emitted.
• Text-to-speech output is English wording for ISL signs; it is a captioning aid, not a faithful translation of ISL grammar.
• Personalisation only adjusts thresholds/vocabulary and optionally fine-tunes on the user's own consented samples. It is deliberately conservative and cannot silently retrain itself from corrections.

5. Claims this project does not make

• It does not claim the attention model beats the GRU/LSTM baselines before the experiment has been run and reported.
• It does not claim the confidence threshold of 0.70 is optimal.
• It does not claim benchmark accuracy equals real-world accuracy.
• It does not claim to understand Indian Sign Language; it classifies isolated signs from a benchmark vocabulary.