speaker-verification
audio
nemotron
tyger
experimental
ChristophSchuhmann's picture
Publish audited Robby source and completed Nemotron speaker-prediction experiments
b21cfae verified
|
Raw History Blame Contribute Delete
6.76 kB
# TYGER V2: lightweight speaker-embedding pilot
This optional network predicts a normalized 192-dimensional ECAPA speaker cue
from the **frozen Nemotron ASR 3.5 layer-15 representation** already computed by
TYGER: 1024 dimensions per 80 ms, four new frames every 320 ms. It does not replace
or retrain the original TYGER GRU, endpointing, interruption, or backchannel rules.
It is not authentication. Synthetic voices and a small held-out pilot cannot
establish accuracy for real users, microphones, accents, or unseen acoustic rooms.
## Measured outcome
Fifteen actual CPU training runs were completed: eleven initial architecture,
width, context, and loss ablations; one centered-target control; and three
class-balanced runs on a second, broader-emotion corpus. The final experimental
integration checkpoint is `artifacts/best.pt`, with `artifacts/selection.json`.
It is a centered MLP with 470,147 trainable parameters and 640 ms context.
Among five predefined candidates it wins on the broader corpus's **validation**
EER (18.0%), not on test metrics. Held-out test EER is still **25.3%**, and the
conservative validation-calibrated threshold rejects roughly 90% of genuine
short probes. Its measured warmed CPU inference is about 0.32 ms per update.
The long-clean ECAPA teacher control achieves 2.5% EER on that corpus but sees
far more clean audio. It is not an equal-context comparison.
**Keep the student identity gate off by default.** It is useful as an optional
research signal, not reliable multi-user filtering. The 100,000-conversation /
500,000-window expansion was not started because this pilot failed reliability
checks. Read `report.html` for the combined results and `DATASET_CARD.md` for
the data/license/coverage limitations. V2 covers anger, amusement, elation and
sadness, but is German-heavy; this does not establish broad English robustness.
## Reproduce the architecture comparison
From `robo-integration`, using the existing Python environment:
```bash
python -m unittest speaker_v2.test_predictor -v
python -m speaker_v2.train --data speaker_v2/pilot --out speaker_v2/results \
--epochs 30 --threads 6 --device cpu
```
Run `prepare_pilot.py --help` for the separate data preparation workflow. It uses
selective downloads, disjoint clean enrollment/probe source clips, a deterministic
global speaker split, frozen encoders, and compact cached features. Inspect its
manifest and source-license records before distributing any derived dataset.
No microphone recordings or enrolled user embeddings belong in this dataset.
The first concrete pilot contains 32 identities (24 training, 4 validation,
4 testing), 64 procedural 24-second conversations, and 4,800 decision hops.
It has 192 source clips (98 German, 94 English), but bounded prefix extraction
reached only `Fatigue_Exhaustion` and `VoiceNetBKGN_moderately_low` conditions.
This is **not broad emotional coverage**; a later, independently sampled pilot
must test additional emotions before claiming robustness or starting a large
publication-quality corpus. Two long source clips per identity are enrollment
teacher targets, while four different source clips supply conversation probes.
The additional complete-probe ECAPA evaluation quantifies identity/domain noise.
The initial ablation compares MLP, GRU, and causal temporal CNN predictors at
widths 96 and 192 (148k–470k parameters). The validation winner is then compared
at 4, 8, 12, and 25 frames (320, 640, 960, and 2000 ms), and with cosine, MSE, and
mixed regression objectives. Because outputs and targets are normalized,
`192 * MSE = 2 * (1 - cosine)`: this particular loss ablation changes the relative
weight of regression versus the auxiliary activity head, not the embedding
geometry. A constant training-teacher mean and frozen-feature ridge regression
are included as controls. The ridge baseline uses oracle activity labels only
to isolate embedding quality and is not a deployable silence detector.
Architecture selection uses validation identities only. Verification thresholds
are calibrated from validation impostor scores (95th percentile, approximately
5% false accepts), never from test labels. Evaluation reports held-out EER,
teacher cosine, closed-set identification, known acceptance, unknown rejection,
and speech/silence/ambiguous-overlap classification. Trials from adjacent windows
are correlated: their count must not be mistaken for independent people.
Temporal accumulation uses oracle same-speaker boundaries and is explicitly an
optimistic diagnostic rather than a deployed diarization result.
Outputs are `results/results.json`, `results/report.html`, individual reproducible
checkpoints, and `results/best.pt`. The report records actual CPU throughput and
extrapolates only the small predictor's training cost, excluding download,
augmentation, ASR/teacher feature extraction, and evaluation. A pilot should be
assessed before authorizing or beginning 500,000-window training.
The broader corpus uses `--data speaker_v2/pilot_v2`. Controlled alternatives
use `--balanced-activity`, `--centered-target`, and `--only-config` (see help).
`python -m speaker_v2.select` reproduces the validation-only integration
selection once all five predefined candidate reports exist; it never promotes
the student to default-on. `python -m speaker_v2.compile_report` rebuilds the
combined English report. Frozen cross-corpus evaluation uses the immutable
`results/best.pt` and `artifacts_centered/best.pt` checkpoints, **not** the mutable
integration destination `artifacts/best.pt`.
## Incremental integration
```python
from speaker_v2.inference import Predictor
predictor = Predictor("speaker_v2/results/best.pt", device="cpu")
result = predictor.update(new_layer15_frames, energy=microphone_rms)
# result: embedding (normalized numpy[192] or None), kind,
# confidence, context_ms, optional activity_probabilities
predictor.reset() # new session / explicit identity-memory reset
```
Call `update` with **only newly produced frames**, not the full rolling ASR window.
Use actual microphone RMS if available. Silence and ambiguous overlap abstain;
short warm-up context reduces confidence. Accumulation resets on silence or an
obvious embedding change, but this heuristic is not guaranteed diarization.
Confidence is an activity/context indicator, **not** a calibrated probability
that a person matches an enrolled identity. The identity manager must separately
check similarity, enrollment status, margin, activity, timeouts, and uncertainty.
Do not automatically enable a predictor that fails held-out controls. Keep the
teacher-embedding fallback or disable identity filtering instead. Store session
voiceprints only with informed consent; do not publish them.