docs/failure_analysis.md

# Failure-analysis protocol

Turn detection fails in sequences, not just rows. This protocol turns model
predictions into reviewable product errors without leaking the official test
back into development.

## Scope and non-negotiable rules

1. Analyze train/development and the locked local Hinglish development split
   while iterating.
2. Inspect the official test only once after freeze. Its failure report is for
   disclosure and future data collection, not same-benchmark tuning.
3. Keep raw confusion counts and denominators beside every rate.
4. Group confidence intervals and sampling by speaker/conversation/leakage group,
   never by correlated rows alone.
5. Do not listen to or expose audio without appropriate access and consent.
6. Do not put names, phone numbers, addresses, order IDs, transcripts containing
   PII, or direct speaker identifiers in review exports.

## Error priorities

| Priority | Error | Product effect | Primary measure |
|---:|---|---|---|
| P0 | False `END` on `HOLD` | User is interrupted | FP count, FPR, interrupted-turn rate, interruptions/hour |
| P1 | Repeated/double `END` | Duplicate agent response | events per true turn in controller replay |
| P1 | False `HOLD` on `END` | Response delay or timeout | FN count, FNR, endpoint-delay p50/p90/p95 |
| P1 | Forced timeout on true `HOLD` | Policy interruption despite low model score | timeout reason count/rate |
| P2 | Poor calibration | Threshold fails after shift | Brier, log loss, ECE, reliability bins |
| P2 | Latency/resource regression | Slow or expensive response | end-to-end p50/p95/p99, load time, RSS |

The first promotion gate is a predeclared false-interruption budget. A model with
higher aggregate F1 but materially more P0 errors is not an improvement.

## Failure taxonomy

Assign one primary cause and any secondary tags. Use `unknown` when evidence is
insufficient rather than inventing an explanation.

### Linguistic and conversational

- `filled_pause`: uh/um/haan/matlab/achha, including elongated fillers;
- `discourse_marker`: “so”, “but”, “toh”, “actually” before continuation;
- `syntactic_continuation`: conjunction, subordinate clause, unfinished phrase;
- `slot_continuation`: partial address, PIN, phone, order/tracking ID, date/time;
- `enumeration`: pause between list items;
- `self_repair`: correction, restart, or abandoned phrase;
- `backchannel`: short acknowledgement while retaining/yielding the floor;
- `code_switch_boundary`: Hindi↔English switch near the pause;
- `prosody_rising`, `prosody_level`, `prosody_falling`: perceived boundary tone;
- `complete_but_short`: short legitimate endpoint;
- `deliberate_long_hold`: long thinking pause followed by continuation;
- `annotation_ambiguous`: reasonable listeners disagree.

### Acoustic and collection

- `background_speech`, `traffic`, `music`, `impulse_noise`;
- `far_field`, `reverberation`, `low_snr`;
- `clipping`, `low_gain`, `codec`, `sample_rate_conversion`;
- `vad_truncation`: checkpoint cuts speech/filler incorrectly;
- `trailing_silence`, `leading_silence`, `non_speech_only`;
- `synthetic_prosody`, `source_artifact`, `duplicate_or_template`;
- `duration_crop`: relevant context fell outside the audio suffix.

### System and policy

- `frontend_mismatch`: train/runtime log-mel disagreement;
- `mask_or_padding`: padded frames influence score;
- `onnx_drift`, `quantization_drift`;
- `threshold_shift`: calibration does not transfer to slice/domain;
- `minimum_silence`, `relaxation`, `maximum_timeout`, `debounce`;
- `latency_budget`: correct decision arrived too late;
- `state_reset`: speech after END did not start a clean turn.

## Required review record

Store review metadata separately from raw audio. One JSONL row per checkpoint:

```json
{
  "review_id": "random-id",
  "run_id": "E-010-seed17",
  "split": "validation",
  "group_id": "hashed-group",
  "turn_id": "hashed-turn",
  "checkpoint_index": 2,
  "target_endpoint": false,
  "endpoint_probability": 0.73,
  "threshold": 0.61,
  "model_prediction": true,
  "controller_state": "END",
  "controller_reason": "model_endpoint",
  "silence_ms": 420,
  "inference_ms": 5.8,
  "language": "hinglish",
  "dataset": "private_hinglish_v1",
  "synthetic": false,
  "midfiller": true,
  "endfiller": false,
  "duration_seconds": 3.4,
  "primary_tag": "filled_pause",
  "secondary_tags": ["code_switch_boundary"],
  "reviewer_count": 3,
  "reviewer_agreement": 0.67,
  "notes": "no transcript or PII"
}
```

Use random or hashed IDs. Free-text notes must not quote a participant or
reconstruct customer data.

## Sampling plan

For each candidate and baseline, review:

1. every false interruption when the set is operationally manageable;
2. otherwise, a group-stratified random sample of at least 100 false
   interruptions, with inclusion probabilities retained;
3. at least 100 false holds, oversampling long endpoint delays;
4. 50 highest-confidence correct predictions of each class to catch label or
   shortcut errors;
5. 50 near-threshold examples on each side;
6. a balanced sample across language, source, human/synthetic, filler, duration,
   device/noise, and consented demographic slices;
7. full controller sequences for any repeated END, timeout, or state-reset issue.

When a slice has fewer examples, review all and report the small denominator.
Do not silently drop undefined FPR/recall for one-class slices.

## Analysis workflow

### 1. Validate the evaluation plumbing

- confirm `1 = END` in labels, predictions, and confusion counts;
- recompute `FP/(FP+TN)` and `FN/(FN+TP)` from stored counts;
- verify prediction IDs join one-to-one with the frozen manifest;
- verify the threshold came from development, not test;
- compare frontend metadata and ONNX input/output names with the checkpoint;
- test masked-padding invariance and PyTorch/ONNX parity;
- verify no leakage group/audio hash/metadata key crosses splits.

### 2. Quantify before listening

Generate overall and sliced counts, score histograms, reliability bins, and
duration/silence deciles. Rank potential shortcut signals: source, synthetic
status, audio length, codec, and sample rate. This prevents a few memorable clips
from replacing distributional evidence.

### 3. Conduct blinded review

Reviewers see the checkpoint audio/context and target question, but not model
score, prediction, source model, or other reviewers' votes. Randomize baseline
and candidate failures. Preserve `uncertain` and collect an optional taxonomy
tag. Adjudicate only after independent review.

### 4. Connect clip errors to controller behavior

Replay chronological pause checkpoints. A clip false positive may be suppressed
by minimum silence or debounce; a modest false negative may be harmless if the
next checkpoint ends promptly. Conversely, a low aggregate FPR can still produce
too many interrupted turns if errors cluster across different conversations.

### 5. Choose the smallest justified intervention

Examples:

- filler-specific failures → rebalance real filler minimal pairs or adjust
  auxiliary weighting;
- crop failures → compare 2/4/8-second suffixes;
- calibration-only slice shift → recalibrate or use a validated controller
  policy, not a larger network by default;
- synthetic-source shortcut → reduce synthetic dominance and validate on real
  speakers;
- VAD truncation → fix checkpoint generation before retraining endpointing;
- INT8 score drift → recalibrate INT8 threshold or retain FP32;
- maximum-timeout interruptions → product-policy review, not label relabeling.

Every new intervention gets a new registered experiment and an untouched
development comparison. Do not manually patch individual test failures.

## Current findings

The following aggregate development findings are supportable; causal taxonomy
labels still require authorized blinded listening:

1. **Source-holdout confounding is severe.** In the inspected shard, source-held
   train is 2,614/2,617 synthetic, while validation is 550/648 human-tagged and
   has only English/Spanish. A delta cannot be assigned to source alone.
2. **The packaged TinyTCN misses most endpoints at the strict threshold.** On
   326 IID development clips it produces 10 TP, 3 FP, 163 TN, and 150 FN at
   threshold 0.7410008. All three false interruptions are synthetic-tagged; two
   occur in the 134-row `midfiller` slice. These counts do not establish cause.
3. **Simple acoustics remain stronger.** The full
   partial-shard logistic baseline ranks validation examples at AUROC 0.752, yet
   recovers only 21/160 END examples when limited to 3/166 HOLD false positives.
   The TinyTCN recovers 10/160 at the same FP count; its paired recall delta is
   −0.06875 with 95% CI [−0.13298, −0.01220].
4. **Trailing silence is a model/controller interface risk.** Appending 800 ms
   of valid silence flips 26/326 decisions, raises FPR from 0.0181 to 0.0361,
   and shifts probability by 0.1058 on average in absolute value. The external
   controller also consumes silence, creating potential double-counting.
5. **No Hinglish failure claim exists.** There are no local participant
   recordings. The Hindi-tagged slice has only 16 examples and is not verified
   code-switched speech.
6. **Current grouping cannot establish identity separation.** All 3,265 base IID
   linkage groups are singletons because the shard lacks usable speaker,
   conversation, and voice IDs. Zero observed-key crossings are real, but
   speaker/TTS leakage is unknown and the IID grouped bootstrap is effectively
   row bootstrap. The source stress split instead clusters by dataset source.

Privacy-safe hashed case IDs identify the 3 false interruptions and 150 missed
ends for authorized local review. Public reports contain no audio, transcript,
raw record ID, or source path. No claim is made about specific acoustic or
linguistic causes because no blinded listening analysis has been performed.

## Promotion gates

A release candidate must satisfy all of the following:

- no unresolved data, split-leakage, frontend-parity, or ONNX-parity defect;
- FPR within the predeclared budget on locked development and official test;
- false-interruption and delay targets met in full controller replay;
- no material regression on real-human, Hinglish, filler, long-hold, identifier,
  device/noise, and major source/language slices;
- grouped confidence intervals rule out a practically harmful regression against
  the chosen baseline;
- at least two reviewers approve every sampled P0 taxonomy cluster and document
  residual risks;
- final threshold is calibrated for the exact FP32 or INT8 artifact;
- model/data cards list all known failures, denominators, and excluded records;
- privacy, consent, licensing, and redistribution reviews are complete.

If a gate cannot be measured, mark it unknown. Unknown is not a pass.