| # Tiny Hinglish Turn Detector |
|
|
| An audio-native, low-latency **HOLD vs END** classifier for voice agents, built |
| for the Shiprocket Data Scientist challenge. The model is evaluated only at VAD |
| pause checkpoints: `HOLD` means “keep listening”; `END` means “the agent may |
| respond.” |
|
|
| The packaged artifact is a development preview: a 151,812-parameter causal |
| TinyTCN using the most recent four seconds of audio. It exports to a 625,431-byte |
| FP32 ONNX file and runs waveform-to-probability inference at 1.63 ms p95 on the |
| measured Apple-arm CPU setup. It does **not** beat the acoustic logistic baseline |
| on the current IID development split, and it has not been evaluated on the |
| official test or on verified Hinglish speech. The repository is submission-grade |
| engineering evidence, not a production-readiness claim. |
|
|
| ## Current evidence, at a glance |
|
|
| | Item | Status | What is actually established | |
| |---|---|---| |
| | Upstream train snapshot | Partial locally | 1 of 83 Parquet shards audited: 3,265 rows, 6.857 hours | |
| | Data integrity on that shard | Measured | 3,265/3,265 valid; no exact-audio, ID, or label-conflict duplicates | |
| | Split integrity | Measured, best effort | IID and source-held-out manifests pass observed-key crossing checks; all 3,265 groups are singletons because speaker/conversation/voice IDs are absent | |
| | Interpretable development baseline | Measured, partial-shard only | 10 acoustic statistics: AP 0.7310, AUROC 0.7522, END recall 0.1313 at FPR 0.0181 | |
| | TinyTCN preview | Measured, partial-shard only | 151,812 parameters: AP 0.6881, AUROC 0.7390, END recall 0.0625 at FPR 0.0181 | |
| | Paired comparison | Baseline remains winner | TinyTCN minus baseline: AP −0.0429 (95% bootstrap CI −0.1195 to 0.0352); constrained recall −0.0688 (−0.1330 to −0.0122) | |
| | Domain-shift stress | Measured, confounded | Source-held-out AP 0.5500/0.5368 for TinyTCN/baseline; both near chance in ranking, with wide nine-source bootstrap intervals | |
| | Runtime/export | Measured | 611 KiB FP32 ONNX; exact PyTorch/ONNX parity; ONNX p95 0.956 ms neural-only and 1.629 ms waveform-to-probability | |
| | Controller integration | Synthetic only | Metadata-bound 8-checkpoint/3-turn fixture emits 3 response edges with 0 duplicates; not model or product-quality evidence | |
| | Code quality gate | Passed | Ruff 0.15.16; 136/136 automated tests passed via `unittest` discovery | |
| | Accuracy/generalization | **Not established** | No full-corpus run, collected Hinglish recordings, real conversation replay, or official-test evaluation | |
| | Publication tooling | Guarded | Hash-verified HF publisher plus curated Kaggle/GitHub upload builders; verify each remote runtime separately | |
|
|
| The official test set remains deliberately sealed. The preview is real |
| development evidence, but validation was reused for threshold selection, |
| checkpoint selection, and bounded follow-up experiments. It is therefore |
| adaptive development evidence—not an independent estimate. See |
| [REPORT.md](REPORT.md) for the full reasoning and [MODEL_CARD.md](MODEL_CARD.md) |
| for the release contract. |
|
|
| The acoustic baseline is the first real-data development result: logistic |
| regression on ten waveform statistics, trained on all 2,939 train rows and |
| scored on all 326 validation rows in the single audited shard. At a threshold |
| selected on that same validation split for `FPR <= 0.02`, it has 3 FP, 21 TP, |
| 163 TN, and 139 FN (`FPR 0.0181`, END recall `0.1313`, AUROC `0.7522`, AP |
| `0.7310`). The selected TinyTCN reaches 3 FP, 10 TP, 163 TN, and 150 FN at |
| threshold `0.7410008` on the same split (`FPR 0.0181`, END recall `0.0625`, |
| AUROC `0.7390`, AP `0.6881`, Brier `0.2133`). Its FPR bootstrap interval is |
| `[0, 0.0407]`, so this sample does not establish a population 2% interruption |
| bound. Neither result is independent-test or Hinglish evidence. |
|
|
| Because that baseline is the evidence winner, its 10-weight JSON artifact is |
| packaged beside the neural model under `reference_models/`; the Gradio Space |
| continues to demonstrate the ONNX TinyTCN and labels that choice explicitly. |
|
|
| ## Why this design |
|
|
| Turn detection is asymmetric. A false `END` interrupts the user and can derail |
| an entire transaction; a false `HOLD` adds latency. Accuracy at an arbitrary |
| 0.5 threshold therefore is not the primary target. The selection criterion is |
| **maximum END recall subject to a false-positive (false-interruption) budget**, |
| with FPR reported at 1%, 2%, and 5% operating points. |
|
|
| The production boundary is also explicit: |
|
|
| ```text |
| 16 kHz audio suffix |
| -> 80-bin log-mel frontend |
| -> causal depthwise-separable TinyTCN |
| -> mask-aware attentive mean/std pooling |
| -> END probability + optional filler auxiliary heads |
| -> calibrated SPEAKING / HOLD / END controller |
| ``` |
|
|
| The selected preview has 151,812 parameters, six 128-channel blocks, and a |
| four-second suffix. Its convolutions are causal; final attentive pooling makes |
| this a suffix-level classifier, not a state-cached streaming network. The |
| repository retains a larger 396,164-parameter/eight-second configuration as a |
| proposed full-data experiment, not as the packaged model. The controller—not |
| the neural network—owns minimum silence, threshold relaxation, debounce, and |
| maximum wait. |
|
|
| No ASR transcript is required at inference. That keeps the path small and |
| avoids coupling endpoint latency to transcription, while the auxiliary |
| `midfiller` and `endfiller` objectives encourage the representation to notice |
| exactly the failure cases the challenge emphasizes. A Whisper-tiny teacher |
| configuration is included as an experiment, not assumed to be the best serving |
| model. |
|
|
| ## Repository map |
|
|
| | Path | Purpose | |
| |---|---| |
| | `src/turn_detection/data/` | Lazy Parquet/HF ingestion, schema normalization, audio audit, transitive grouping, split validation | |
| | `src/turn_detection/models/` | Canonical log-mel frontend, TinyTCN student, Whisper teacher, attentive pooling | |
| | `src/turn_detection/training/` | Lazy dataloaders, masked multitask loss, threshold calibration, product metrics, trainer | |
| | `src/turn_detection/runtime/` | NumPy preprocessing, ONNX inference, turn controller, replay evaluation | |
| | `configs/` | Reproducible student, teacher, and smoke configurations | |
| | `scripts/` | Download, audit, split, train, evaluate, export, benchmark, package, publish | |
| | `data/collection/` | Deterministic 900-recording Hinglish assignment plan; no recordings are included | |
| | `reports/` | Measured one-shard audit and split reports | |
| | `tests/` | Unit and integration coverage across data, model, metrics, runtime, and prompts | |
| | `deployment/kaggle/` | Flat, standalone ONNX inference templates for Kaggle Models | |
| | `upload-ready/` | Generated Kaggle and GitHub upload folders; never a source input | |
|
|
| ## Reproduce the pipeline |
|
|
| Python 3.10–3.12 and [`uv`](https://docs.astral.sh/uv/) are recommended. |
| These commands target a full source checkout. Raw audio, downloaded Parquet, |
| processed manifests, and per-example predictions are intentionally not |
| redistributed; recreate them with the pinned download/audit commands below. |
|
|
| ```bash |
| uv sync --extra all |
| uv run pytest |
| ``` |
|
|
| The curated public repository already includes the selected ONNX artifact, so |
| the demo resolves it without an environment variable: |
|
|
| ```bash |
| uv sync --extra demo |
| uv run python app.py |
| ``` |
|
|
| Place the Hugging Face token in the ignored `.env` file; never commit it: |
|
|
| ```bash |
| cp .env.example .env |
| # edit .env and set HF_TOKEN |
| ``` |
|
|
| ### 1. Download the pinned train snapshot |
|
|
| ```bash |
| bash scripts/download_dataset.sh |
| ``` |
|
|
| The script pins revision |
| `e564e2ac567f774d1880aa1db6ce97afb8c519b7`, resumes safely, and refuses to |
| declare success until all 83 Parquet shards are present. |
|
|
| ### 2. Audit and create manifests |
|
|
| ```bash |
| uv run python scripts/audit_dataset.py \ |
| data/raw/smart-turn-data-v3.2-train \ |
| --output data/processed/manifest.jsonl \ |
| --report artifacts/data_audit.json \ |
| --fail-on-error |
| |
| uv run python scripts/prepare_splits.py \ |
| --input data/processed/manifest.jsonl \ |
| --output data/processed/splits.jsonl \ |
| --report artifacts/split_report.json \ |
| --split train=0.9 --split validation=0.1 \ |
| --stratify endpoint,language,synthetic,dataset |
| ``` |
|
|
| The audit hashes encoded audio and creates privacy-preserving linkage keys for |
| conversation, speaker, voice, recording, prompt, record, and repeated normalized |
| text. Union-find turns overlapping keys into transitive components; the split |
| tool assigns whole components and fails validation if any group, audio hash, or |
| metadata key crosses splits. |
|
|
| On the available shard, however, upstream rows expose no usable speaker, |
| conversation, or TTS-voice identity: all 3,265 base linkage groups used by the |
| IID split contain one row. (The separate stress split deliberately regroups rows |
| by 12 source-dataset values.) |
| The measured split is therefore best-effort row-disjoint and duplicate-safe, |
| **not speaker-disjoint**. Its grouped bootstrap is effectively a row bootstrap; |
| speaker/voice leakage remains unknown until richer identity metadata or a new |
| consented speaker-disjoint set exists. |
|
|
| For a harsher domain-shift test, add `--holdout-field dataset`. Interpret that |
| result carefully: in the currently inspected shard, source is strongly |
| confounded with language and synthetic status. Leave-one-source-out helpers are |
| also available in `turn_detection.data.splits`. |
|
|
| ### 3. Prove the training path, then run the real experiment |
|
|
| ```bash |
| # Fast generated-feature integration check; never report as model quality. |
| uv run python scripts/train.py --config configs/smoke.json --smoke-test |
| |
| # Reproduce the selected one-shard preview (development evidence only). |
| uv run python scripts/train.py \ |
| --config configs/partial_shard_warmstart_lr3e4_5ep.json |
| |
| # Proposed primary full-data student experiment, once all shards are audited. |
| uv run python scripts/train.py --config configs/tiny_tcn.yaml |
| ``` |
|
|
| Run the interpretable partial-shard sanity baseline independently: |
|
|
| ```bash |
| uv run python scripts/run_baselines.py audio \ |
| --manifest data/processed/partial-iid-splits.jsonl \ |
| --source-root . \ |
| --output artifacts/partial-baseline \ |
| --epochs 1000 --fpr-budget 0.02 |
| ``` |
|
|
| The trainer writes the resolved configuration, epoch history, calibrated |
| threshold, and best checkpoint. Deterministic mode seeds model initialization, |
| data order, and training. Missing filler labels stay missing and are masked from |
| their auxiliary BCE losses. |
|
|
| ### 4. Evaluate without touching the official test |
|
|
| ```bash |
| uv run python scripts/evaluate.py \ |
| --checkpoint artifacts/partial-shard-warmstart-lr3e4-5ep/best.pt \ |
| --source data/processed/partial-iid-splits.jsonl \ |
| --source-root . \ |
| --split validation \ |
| --output reports/partial_tinytcn_metrics.json |
| ``` |
|
|
| Evaluation reports confusion counts, FPR/FNR, AUROC, average precision, Brier |
| score, ECE/reliability bins, operating points under FPR budgets, and |
| language/source/synthetic slices. Turn-level interruption rates are emitted only |
| when genuine turn/conversation IDs exist; this shard supports only a clearly |
| labelled clip-normalized per-audio-hour proxy. Final comparisons need real |
| speaker- or conversation-cluster confidence intervals. |
|
|
| The selected checkpoint was chosen by average precision after earlier runs had |
| already been inspected on this validation set. Its threshold was also selected |
| on the same 326 examples. Reproduction should yield the stored numbers, but they |
| must not be treated as a fresh holdout result. |
|
|
| ### 5. Freeze, unseal once, and evaluate the official test |
|
|
| Only an experiment trained with exact `run.status: final` can be frozen. The |
| freeze manifest hashes the checkpoint, selected config, source tree, |
| preprocessing, controller policy, split manifest, threshold, and pinned test |
| identity. `configs/final.yaml` and `artifacts/final/*` below are intentionally |
| future-candidate placeholders; create them only after completing the full-data |
| experiment. Only then: |
|
|
| ```bash |
| uv run python scripts/freeze_candidate.py \ |
| --checkpoint artifacts/final/best.pt \ |
| --config configs/final.yaml \ |
| --output artifacts/final/frozen_manifest.json |
| |
| UNSEAL_OFFICIAL_TEST=I_HAVE_FROZEN_MODEL_AND_THRESHOLD \ |
| FROZEN_MANIFEST=artifacts/final/frozen_manifest.json \ |
| bash scripts/download_test_dataset.sh |
| |
| uv run python scripts/evaluate.py \ |
| --checkpoint artifacts/final/best.pt \ |
| --source data/raw/smart-turn-data-v3.2-test \ |
| --dataset-id pipecat-ai/smart-turn-data-v3.2-test \ |
| --revision 0500378e8ed6d38e37b016e24d261e8e6c6a6859 \ |
| --split test \ |
| --allow-sealed-test \ |
| --frozen-manifest artifacts/final/frozen_manifest.json \ |
| --output artifacts/final/test_metrics.json |
| ``` |
|
|
| Do not tune after reading the test result. A new model requires a new untouched |
| test set or a clearly labeled exploratory result. |
|
|
| ### 6. Export and benchmark |
|
|
| ```bash |
| uv run python scripts/export_onnx.py \ |
| --checkpoint artifacts/final/best.pt \ |
| --output artifacts/final/model.onnx |
| |
| uv run python scripts/benchmark.py \ |
| --model artifacts/final/model.onnx \ |
| --metadata artifacts/final/model_metadata.json \ |
| --threads 1 --batch-size 1 \ |
| --output artifacts/final/cpu_benchmark.json |
| ``` |
|
|
| ONNX export checks PyTorch/ONNX numerical parity. Runtime preprocessing, |
| I/O names, activation semantics, threshold, parameter count, and evidence scope |
| live in `model_metadata.json`; hashes, shapes, checkpoint/config/split bindings, |
| parity, and quantization status live in `export_manifest.json`. Static INT8 is |
| supported with representative calibration features; quantized scores require |
| threshold recalibration before release. |
|
|
| The packaged preview is |
| `artifacts/partial-shard-warmstart-lr3e4-5ep/model.onnx`: 625,431 bytes, |
| 151,812 parameters, and FP32 maximum absolute parity error `0.0`. On the |
| recorded Apple-arm/one-thread runs, ONNX neural-only p50/p95/p99 is |
| 0.919/0.956/0.974 ms; four-second waveform-to-probability p50/p95/p99 is |
| 1.572/1.629/1.651 ms. These are local implementation measurements, not a |
| deployment SLA. |
|
|
| ### 7. Run the demo and exercise the controller |
|
|
| ```bash |
| TURN_MODEL_PATH=artifacts/partial-shard-warmstart-lr3e4-5ep/model.onnx \ |
| uv run python app.py |
| |
| uv run python scripts/replay_stream.py \ |
| --input data/collection/controller_replay_fixture.jsonl \ |
| --output reports/controller_replay_integration.jsonl \ |
| --metadata artifacts/partial-shard-warmstart-lr3e4-5ep/model_metadata.json \ |
| --evidence-scope synthetic_integration |
| ``` |
|
|
| The exported metadata binds the endpoint threshold and the complete controller |
| policy. `END` is latched until new speech or an explicit reset; only the first |
| END transition sets `emit_response=true`, preventing duplicate agent replies. |
| The bundled replay fixture is hand-authored integration data: its probabilities, |
| targets, and latency values are synthetic. It validates policy binding, |
| threshold relaxation, timeout, END latching, and edge emission only—it is not |
| model, conversation, latency, or product-quality evidence. Replace its input |
| with private, human-annotated VAD checkpoint logs for a real sequence study. |
|
|
| If weights are absent, the demo displays a conspicuous heuristic fallback. That |
| fallback validates only the UI and must never be cited as a model result. See |
| [docs/demo.md](docs/demo.md) for the pause-checkpoint contract. |
|
|
| ### 8. Build and publish a guarded release |
|
|
| ```bash |
| uv run python scripts/build_release.py \ |
| --model artifacts/final/model.onnx \ |
| --metadata artifacts/final/model_metadata.json \ |
| --metrics artifacts/final/test_metrics.json \ |
| --frozen-manifest artifacts/final/frozen_manifest.json |
| |
| # Dry validation first. |
| uv run python scripts/publish_hf.py --username suvradeepp |
| |
| # Creates/uploads the model repo and Gradio Space only after identity validation. |
| uv run python scripts/publish_hf.py --username suvradeepp --execute |
| ``` |
|
|
| For a personal account without Hugging Face PRO, the publisher requests the |
| free-account-compatible ZeroGPU tier (`--space-hardware zero-a10g`) only when it |
| must create a new Space. Hugging Face currently gates new personal Gradio CPU |
| Spaces behind PRO even when CPU Basic has no hourly charge. ZeroGPU eligibility |
| still requires a verified account in good standing and an available hosted |
| Space slot. Paid accounts may override this with `--space-hardware cpu-basic`. |
|
|
| The release builder refuses smoke checkpoints or a release without measured |
| test metrics unless `--allow-development-artifact` is passed explicitly. That |
| override must remain visibly development-only. For a reviewable partial-data |
| preview, pass that flag to the builder and `--allow-development-release` to both |
| publisher invocations. Development validation metrics are packaged as |
| `development_metrics.json`, never `test_metrics.json`. |
|
|
| The exact development-preview package command is: |
|
|
| ```bash |
| uv run python scripts/build_release.py \ |
| --model artifacts/partial-shard-warmstart-lr3e4-5ep/model.onnx \ |
| --metadata artifacts/partial-shard-warmstart-lr3e4-5ep/model_metadata.json \ |
| --metrics reports/partial_tinytcn_metrics.json \ |
| --output release \ |
| --allow-development-artifact \ |
| --include-synthetic-controller-replay |
| |
| uv run python scripts/publish_hf.py \ |
| --username suvradeepp \ |
| --release-dir release \ |
| --allow-development-release |
| |
| uv run python scripts/build_upload_folders.py \ |
| --release-dir release \ |
| --output upload-ready \ |
| --allow-development-release |
| ``` |
|
|
| `upload-ready/kaggle-model/` is a flat drag-and-drop ONNX bundle. |
| `upload-ready/github-repository/` is a public-repository allowlist containing |
| source, CI, tests, aggregate evidence, ONNX weights, and the two checkpoints |
| needed to reproduce the selected warm start. It excludes credentials, caches, |
| raw/processed data, per-example predictions, stale releases, and publish |
| receipts. Keep the Kaggle variation private until the derived-weight rights |
| review described in `NOTICE` and `MODEL_CARD.md` is resolved. |
|
|
| Add `--execute` only after the dry validation succeeds. The publisher verifies |
| the authenticated owner and every packaged hash before creating or updating the |
| remote model repository and Space. Its receipt binds exact remote HEAD commits, |
| requires a visible build/start transition for a new Space upload, and records |
| the final `RUNNING` stage. Because the installed Hub API does not expose the |
| commit actually serving behind a Space runtime, the receipt explicitly records |
| `runtime_commit_bound=false` instead of overstating that guarantee. |
|
|
| The release hash-binds the 2.7 MB split manifest but intentionally does not |
| redistribute it because it contains upstream text and record identifiers. The |
| public package carries deterministic preparation code, the pinned revision, |
| manifest SHA-256, and aggregate reports; per-example rows stay private. |
|
|
| ## Hinglish evaluation protocol |
|
|
| The upstream corpus is multilingual, but a `hin` or `eng` tag is not evidence |
| of within-utterance code-switching. This repository therefore includes a |
| separate, consent-first Shiprocket-domain protocol: |
|
|
| - 30 speakers × 30 assignments = 900 intended recordings; |
| - 450 `HOLD` and 450 `END` targets; |
| - 21/4/5 speaker-disjoint train/dev/test speakers; |
| - minimal pairs with natural 0.3–2.0 second internal pauses plus roleplay; |
| - three annotators answer whether responding at the checkpoint feels like an |
| interruption; uncertain votes remain disagreement, not forced labels. |
|
|
| Only the assignments and protocol exist today—**no participant audio has been |
| collected**. Do not infer Hinglish performance until that benchmark is recorded |
| and evaluated. See [docs/data_collection_protocol.md](docs/data_collection_protocol.md) |
| and [DATA_CARD.md](DATA_CARD.md). |
|
|
| ## What “good enough” means |
|
|
| A candidate is releaseable only after all of these gates pass: |
|
|
| 1. all 83 train shards audited with zero unresolved hard validation errors; |
| 2. group-safe validation and source/speaker stress tests completed; |
| 3. threshold frozen under a predeclared false-interruption budget; |
| 4. real Hinglish minimal-pair test collected with consent and held out by speaker; |
| 5. official test opened once and reported without subsequent tuning; |
| 6. preprocessing parity, ONNX parity, batch-1 CPU latency, and controller replay measured; |
| 7. failure slices reviewed for language, source, synthetic/human, filler, duration, |
| device/noise, gender/region where consented, and uncertainty; |
| 8. upstream data/model redistribution terms reviewed before publishing weights. |
|
|
| The current repository passes export/runtime engineering gates and supplies |
| honest bounded development evidence. It does not pass the full-data, Hinglish, |
| official-test, sequence-replay, or rights-review gates. |
|
|
| ## Documentation |
|
|
| - [Technical report](REPORT.md) |
| - [Model card](MODEL_CARD.md) |
| - [Data card](DATA_CARD.md) |
| - [Experiment registry](docs/experiment_registry.md) |
| - [Failure-analysis protocol](docs/failure_analysis.md) |
| - [Hinglish collection protocol](docs/data_collection_protocol.md) |
| - [Demo/runtime contract](docs/demo.md) |
|
|
| ## License and responsible use |
|
|
| Repository-authored code is Apache-2.0; see [LICENSE](LICENSE) and |
| [NOTICE](NOTICE). The upstream |
| dataset card did not provide an explicit dataset license at the inspected |
| revision. This repository does not redistribute its audio. Apache-2.0 on the |
| code does **not** grant rights to upstream audio, recordings collected under the |
| local protocol, speaker likeness, customer data, or the development model |
| weights. The preview weights carry no license grant pending an upstream-rights |
| review; review data and derived-weight terms before use or redistribution. |
|
|
| This system decides conversational timing; it should not be used for speaker |
| identification, emotion inference, surveillance, or consequential decisions. |
|
|
