Download code/speaker_v2/README.md from laion/tyger-speaker-predictor: direct link, hf CLI and curl.
- Browser
- Download file 6.76 kB
-
https://huggingface.co/laion/tyger-speaker-predictor/resolve/main/code/speaker_v2/README.md
- Command line
-
hf download hf://laion/tyger-speaker-predictor/code/speaker_v2/README.md
-
curl -L -o README.md https://huggingface.co/laion/tyger-speaker-predictor/resolve/main/code/speaker_v2/README.md
TYGER V2: lightweight speaker-embedding pilot
This optional network predicts a normalized 192-dimensional ECAPA speaker cue from the frozen Nemotron ASR 3.5 layer-15 representation already computed by TYGER: 1024 dimensions per 80 ms, four new frames every 320 ms. It does not replace or retrain the original TYGER GRU, endpointing, interruption, or backchannel rules. It is not authentication. Synthetic voices and a small held-out pilot cannot establish accuracy for real users, microphones, accents, or unseen acoustic rooms.
Measured outcome
Fifteen actual CPU training runs were completed: eleven initial architecture,
width, context, and loss ablations; one centered-target control; and three
class-balanced runs on a second, broader-emotion corpus. The final experimental
integration checkpoint is artifacts/best.pt, with artifacts/selection.json.
It is a centered MLP with 470,147 trainable parameters and 640 ms context.
Among five predefined candidates it wins on the broader corpus's validation
EER (18.0%), not on test metrics. Held-out test EER is still 25.3%, and the
conservative validation-calibrated threshold rejects roughly 90% of genuine
short probes. Its measured warmed CPU inference is about 0.32 ms per update.
The long-clean ECAPA teacher control achieves 2.5% EER on that corpus but sees
far more clean audio. It is not an equal-context comparison.
Keep the student identity gate off by default. It is useful as an optional
research signal, not reliable multi-user filtering. The 100,000-conversation /
500,000-window expansion was not started because this pilot failed reliability
checks. Read report.html for the combined results and DATASET_CARD.md for
the data/license/coverage limitations. V2 covers anger, amusement, elation and
sadness, but is German-heavy; this does not establish broad English robustness.
Reproduce the architecture comparison
From robo-integration, using the existing Python environment:
python -m unittest speaker_v2.test_predictor -v
python -m speaker_v2.train --data speaker_v2/pilot --out speaker_v2/results \
--epochs 30 --threads 6 --device cpu
Run prepare_pilot.py --help for the separate data preparation workflow. It uses
selective downloads, disjoint clean enrollment/probe source clips, a deterministic
global speaker split, frozen encoders, and compact cached features. Inspect its
manifest and source-license records before distributing any derived dataset.
No microphone recordings or enrolled user embeddings belong in this dataset.
The first concrete pilot contains 32 identities (24 training, 4 validation,
4 testing), 64 procedural 24-second conversations, and 4,800 decision hops.
It has 192 source clips (98 German, 94 English), but bounded prefix extraction
reached only Fatigue_Exhaustion and VoiceNetBKGN_moderately_low conditions.
This is not broad emotional coverage; a later, independently sampled pilot
must test additional emotions before claiming robustness or starting a large
publication-quality corpus. Two long source clips per identity are enrollment
teacher targets, while four different source clips supply conversation probes.
The additional complete-probe ECAPA evaluation quantifies identity/domain noise.
The initial ablation compares MLP, GRU, and causal temporal CNN predictors at
widths 96 and 192 (148k–470k parameters). The validation winner is then compared
at 4, 8, 12, and 25 frames (320, 640, 960, and 2000 ms), and with cosine, MSE, and
mixed regression objectives. Because outputs and targets are normalized,
192 * MSE = 2 * (1 - cosine): this particular loss ablation changes the relative
weight of regression versus the auxiliary activity head, not the embedding
geometry. A constant training-teacher mean and frozen-feature ridge regression
are included as controls. The ridge baseline uses oracle activity labels only
to isolate embedding quality and is not a deployable silence detector.
Architecture selection uses validation identities only. Verification thresholds are calibrated from validation impostor scores (95th percentile, approximately 5% false accepts), never from test labels. Evaluation reports held-out EER, teacher cosine, closed-set identification, known acceptance, unknown rejection, and speech/silence/ambiguous-overlap classification. Trials from adjacent windows are correlated: their count must not be mistaken for independent people. Temporal accumulation uses oracle same-speaker boundaries and is explicitly an optimistic diagnostic rather than a deployed diarization result.
Outputs are results/results.json, results/report.html, individual reproducible
checkpoints, and results/best.pt. The report records actual CPU throughput and
extrapolates only the small predictor's training cost, excluding download,
augmentation, ASR/teacher feature extraction, and evaluation. A pilot should be
assessed before authorizing or beginning 500,000-window training.
The broader corpus uses --data speaker_v2/pilot_v2. Controlled alternatives
use --balanced-activity, --centered-target, and --only-config (see help).
python -m speaker_v2.select reproduces the validation-only integration
selection once all five predefined candidate reports exist; it never promotes
the student to default-on. python -m speaker_v2.compile_report rebuilds the
combined English report. Frozen cross-corpus evaluation uses the immutable
results/best.pt and artifacts_centered/best.pt checkpoints, not the mutable
integration destination artifacts/best.pt.
Incremental integration
from speaker_v2.inference import Predictor
predictor = Predictor("speaker_v2/results/best.pt", device="cpu")
result = predictor.update(new_layer15_frames, energy=microphone_rms)
# result: embedding (normalized numpy[192] or None), kind,
# confidence, context_ms, optional activity_probabilities
predictor.reset() # new session / explicit identity-memory reset
Call update with only newly produced frames, not the full rolling ASR window.
Use actual microphone RMS if available. Silence and ambiguous overlap abstain;
short warm-up context reduces confidence. Accumulation resets on silence or an
obvious embedding change, but this heuristic is not guaranteed diarization.
Confidence is an activity/context indicator, not a calibrated probability
that a person matches an enrolled identity. The identity manager must separately
check similarity, enrollment status, margin, activity, timeouts, and uncertainty.
Do not automatically enable a predictor that fails held-out controls. Keep the teacher-embedding fallback or disable identity filtering instead. Store session voiceprints only with informed consent; do not publish them.