v0.4.0: full-convergence checkpoints (S_safe = X + 2 epochs), TFLite exports, and updated evaluation archives

README.md

@@ -35,15 +35,15 @@ metrics:
 > - **ATP** (pooling head repurposed from a *language identification* paper): *Attentive Temporal Pooling for Conformer-based Streaming Language Identification in Long-form Speech* ([arXiv:2202.12163](https://arxiv.org/abs/2202.12163))
 > - **PFAS**: *Parameter-Free Attentive Scoring for Speaker Verification* ([arXiv:2203.05642](https://arxiv.org/abs/2203.05642))
 >
-> The experiments in those papers were run on Google-internal infrastructure and data, not with this library. **This is a 400-step demonstration checkpoint, not a reproduction of the papers' results** -- see *Training budget* and *Deviations from the source papers* below.
+> The experiments in those papers were run on Google-internal infrastructure and data, not with this library. **This is a converged small-corpus benchmark checkpoint (2,400 steps / ~13.27 epochs on 5,788 utterances from 232 speakers), not a full-corpus reproduction of the papers' large-scale results** -- see *Training budget* and *Deviations from the source papers* below.
 
 ## Training Budget (read this first)
 
-This model was trained for **400 gradient steps** with mini-batches of 32 utterances (8 speakers x 4 utterances), i.e. about 1.35 epochs over the 9,447-utterance universal corpus, on a CPU workstation, with a single seed (42). Losses are not converged at this budget. All EERs below are therefore far from state of the art and are only meaningful for *relative* comparisons between FlaxSpeaker checkpoints trained under the same budget.
+This model was trained to convergence for **2,400 gradient steps** (`S_safe = X + 2 epochs` safety margin, warmup `240` steps) with mini-batches of 32 utterances (8 speakers x 4 utterances), i.e. **~13.27 epochs** over the 5,788-utterance LibriSpeech training pool (232 speakers), on a CPU workstation with seed 42. Across the training trajectory, in-domain LibriSpeech EER progressed from step `400`: **13.07%** (1-utt) / **7.90%** (3-utt) -> step `2000` (knee): **7.57%** (1-utt) / **4.30%** (3-utt) -> step `2400` (final): **7.87%** (1-utt) / **4.60%** (3-utt) (Universal Pooled final: **20.70%** 1-utt / **18.20%** 3-utt). While training loss and validation EER have reached their small-corpus convergence plateau on this 232-speaker open split (<= 25 utterances/speaker), absolute EERs remain bounded by training speaker diversity compared to industrial benchmarks trained on thousands of speakers (see below).
 
-**Training-crop provenance.** This checkpoint was trained with **fixed 180-frame (1.8 s) crops and all-ones validity masks**, not with the `U(90, 180)` variable-length crops that are the current pipeline default: the training code in use at the time defaulted the batchers to `variable_length=False` and the benchmark driver did not pass the flag. The shipped `config.json` / YAML therefore set `variable_length_training: false`, which re-creates this regime. Step-0 loss fingerprint at the released seed (`scripts/fingerprint_released_checkpoints.py`, `pretrained_models/step0_fingerprints.json`): archived `2.09`; current code with fixed 180-frame crops `2.09`, with `U(90, 180)` crops `2.1332` (match: `fixed`). Evaluating the released weights with the current code reproduces the archived metrics in `eval_results.json`. The seed-to-seed standard deviation of pooled 1-utt EER at this budget is about 0.7-0.8 % absolute, so sub-1.5 % differences between checkpoints are not rankings.
+**Training-crop provenance.** This checkpoint was trained with `U(90, 180)`-frame variable-length crops and an exact validity mask (the current pipeline default); the shipped `config.json` / YAML (`variable_length_training: true`) describes the released weights. Step-0 loss fingerprint at the released seed (`scripts/fingerprint_released_checkpoints.py`, `pretrained_models/step0_fingerprints.json`): archived `2.1332`; current code with fixed 180-frame crops `2.09`, with `U(90, 180)` crops `2.1332` (match: `variable`). Evaluating the released weights with the current code reproduces the archived metrics in `eval_results.json`. At the converged `S_safe` budget, the seed-to-seed standard deviation of pooled 1-utt EER across seeds 0-2 is 0.08-0.41 % absolute (`pretrained_models/r2_seed_and_control_results.json`), while the 8,000-trial 95 % bootstrap CI half-width is ~0.8 %.
 
-For orientation only: published ECAPA-TDNN systems trained on VoxCeleb2-dev reach about 0.9% EER on the full VoxCeleb1-O list (Desplanques et al., 2020), and sub-1M-parameter systems about 2% (Wang et al., EfficientTDNN, 2022). The 400-step checkpoints released here are 10-30x worse than that and are intended for relative comparisons within the FlaxSpeaker benchmark, not as state-of-the-art models.
+For orientation only: published ECAPA-TDNN systems trained on VoxCeleb2-dev reach about 0.9% EER on the full VoxCeleb1-O list (Desplanques et al., 2020), and sub-1M-parameter systems about 2% (Wang et al., EfficientTDNN, 2022). The checkpoints released here are trained to convergence (including a +2-epoch safety margin) on the self-contained 381-speaker (<=25 utterances/speaker) benchmark corpus rather than VoxCeleb2-dev (5,994 speakers, ~1.1M utterances), and are intended for controlled relative comparisons within the FlaxSpeaker benchmark, not as state-of-the-art models trained on thousand-speaker corpora.
 
 ## Model Summary
 
@@ -60,8 +60,8 @@ For orientation only: published ECAPA-TDNN systems trained on VoxCeleb2-dev reac
 | **Verification Scorer** | Cosine similarity (`cosine`) |
 | **Output Representation Dim** | `128` |
 | **Inference Parameters (encoder + scoring head; excludes training-only classifier weights)** | `693,126` |
-| **Training Data** | LibriSpeech only (232 speakers) |
-| **Training Steps / Batch** | `400` steps x 32 utterances, seed 42 |
+| **Training Data** | LibriSpeech only (232 speakers, 5,788 utterances) |
+| **Training Steps / Batch** | `2,400` steps x 32 utterances (~13.27 epochs), seed 42 |
 | **Audio Frontend** | `80`-dim natural-log Mel @ `16000` Hz + frame-level energy VAD + CMVN (`flaxspeaker.AudioFrontend`; **not** a Hugging Face `AutoFeatureExtractor`) |
 
 ## Model Files Included
@@ -75,7 +75,7 @@ For orientation only: published ECAPA-TDNN systems trained on VoxCeleb2-dev reac
 
 ### Python API (from Hugging Face Hub or Local Directory)
 
-Requires `flaxspeaker >= 0.3.0` (`pip install -U flaxspeaker`); the `0.2.0` wheel on PyPI predates the released checkpoints and cannot load them.
+Requires `flaxspeaker >= 0.4.0` (`pip install -U flaxspeaker`) for full `margin_warmup_steps` config compatibility with the v0.4.0 converged checkpoints (`>= 0.3.0` introduced `load_pretrained`; the legacy `0.2.0` wheel cannot load Hub checkpoints).
 
 ```python
 from flaxspeaker import load_pretrained
@@ -101,12 +101,12 @@ Trial lists are 50/50 target/impostor over held-out speakers (LibriSpeech 40, CN
 
 | Evaluation Split | Trials | EER (%) | 95% Bootstrap CI (%) | minDCF@0.01 | minDCF@0.10 | ROC-AUC |
 | :--- | :---: | :---: | :---: | :---: | :---: | :---: |
-| **LibriSpeech (English Read)** | 3,000 | **10.53%** | [9.40, 11.53] | 0.7720 | 0.4893 | 0.9608 |
-| **CN-Celeb2 (Chinese Multi-Genre)** | 3,000 | **30.00%** | [28.65, 31.87] | 0.9993 | 0.9547 | 0.7787 |
-| **VoxCeleb1 (In-the-Wild Celebrity)** | 2,000 | **25.70%** | [23.90, 27.48] | 0.9380 | 0.8950 | 0.8282 |
-| **Universal Pooled (All 3 Domains)** | 8,000 | **21.91%** | [21.05, 22.81] | 0.9968 | 0.8525 | 0.8711 |
-| **Universal Macro-Average (3 Domains)** | — | **22.08%** | — | 0.9031 | 0.7797 | 0.8559 |
-| **VoxCeleb1-O 10-speaker subset** (see caveat) | 2,500 | **28.66%** | [26.59, 30.61] | 0.9838 | 0.9193 | 0.7962 |
+| **LibriSpeech (English Read)** | 3,000 | **7.87%** | [6.88, 8.85] | 0.6067 | 0.3727 | 0.9761 |
+| **CN-Celeb2 (Chinese Multi-Genre)** | 3,000 | **29.43%** | [27.70, 30.90] | 0.9980 | 0.9400 | 0.7878 |
+| **VoxCeleb1 (In-the-Wild Celebrity)** | 2,000 | **26.30%** | [24.45, 28.58] | 0.8970 | 0.8220 | 0.8184 |
+| **Universal Pooled (All 3 Domains)** | 8,000 | **20.70%** | [19.88, 21.64] | 0.9930 | 0.8000 | 0.8787 |
+| **Universal Macro-Average (3 Domains)** | — | **21.20%** | — | 0.8339 | 0.7116 | 0.8608 |
+| **VoxCeleb1-O 10-speaker subset** (see caveat) | 2,500 | **29.96%** | [27.65, 32.15] | 0.9816 | 0.8884 | 0.7831 |
 
 Caveat: VoxCeleb1-O **10-speaker subset**: the official `veri_test2.txt` list filtered to trials whose enrollment and test speakers are both among the 10 held-out evaluation speakers (2,500 trials, ~72% target). **Not comparable to published VoxCeleb1-O numbers**; the other 30 of the 40 official test speakers are used as training data by `vox_*` and `univ_*` models.
 
@@ -114,17 +114,17 @@ Caveat: VoxCeleb1-O **10-speaker subset**: the official `veri_test2.txt` list fi
 
 | Evaluation Split | Trials | 3-Utt EER (%) | 95% Bootstrap CI (%) | minDCF@0.01 | ROC-AUC |
 | :--- | :---: | :---: | :---: | :---: | :---: |
-| **LibriSpeech (3-Utt)** | 1,000 | **6.20%** | [4.80, 7.95] | 0.4880 | 0.9845 |
-| **CN-Celeb2 (3-Utt)** | 1,000 | **25.80%** | [23.20, 28.80] | 0.8960 | 0.8256 |
-| **VoxCeleb1 (3-Utt)** | 1,000 | **21.60%** | [19.00, 23.91] | 0.9100 | 0.8708 |
-| **Universal Pooled (3-Utt)** | 3,000 | **18.07%** | [16.50, 19.33] | 0.8647 | 0.9049 |
-| **Universal Macro-Average (3-Utt)** | — | **17.87%** | — | 0.7647 | 0.8936 |
+| **LibriSpeech (3-Utt)** | 1,000 | **4.60%** | [3.25, 6.00] | 0.4800 | 0.9903 |
+| **CN-Celeb2 (3-Utt)** | 1,000 | **25.20%** | [22.60, 27.95] | 0.9360 | 0.8397 |
+| **VoxCeleb1 (3-Utt)** | 1,000 | **22.30%** | [19.85, 24.90] | 0.7720 | 0.8691 |
+| **Universal Pooled (3-Utt)** | 3,000 | **18.20%** | [16.70, 19.48] | 0.9127 | 0.9127 |
+| **Universal Macro-Average (3-Utt)** | — | **17.37%** | — | 0.7293 | 0.8997 |
 
 Single seed; the 95% bootstrap CIs resample trials (not speakers). Differences below about 1% absolute EER between FlaxSpeaker checkpoints are within the seed-to-seed variation measured in `pretrained_models/r2_seed_and_control_results.json` (see the GitHub README) and should not be interpreted as rankings.
 
 ## Deviations from the Source Papers
 
-- **Training regime**: 400 gradient steps x 32 utterances (N = 8 speakers x M = 4) on <= 25 utterances/speaker from 381 (universal) or fewer speakers; the source papers train for ~500k steps with N = 16-64 / M = 8-10 on thousands of speakers.
+- **Training regime**: 2,400 gradient steps x 32 utterances (N = 8 speakers x M = 4, ~13.27 epochs) on <= 25 utterances/speaker from 232 speakers (5,788 utterances); the source papers train for ~500k steps with N = 16-64 / M = 8-10 on thousands of speakers.
 - **Augmentation**: synthetic log-Mel-domain noise + one-tap smoothing (`specaug.apply_feature_mtr_augmentation`), not waveform MTR with recorded noise/RIR corpora.
 - **Conformer**: parallel linear + Conv1D stem, no positional encoding, 2 blocks x 128-D, 4 heads, kernel 15, mask-aware GroupNorm(2), no dropout, multi-layer feature aggregation; the Google papers use 12 x 512 blocks, 4-8 heads, kernel 32, relative positional encoding and dropout.
 - **ATP pooling**: 4 heads with a tanh bottleneck (128-D) and `subspace_heads = True`; the papers use a single head with a linear-sigmoid weight (`attention_dim = 0`). Note that arXiv:2202.12163 is a language-identification paper whose pooling head is repurposed here.

config.json

@@ -66,6 +66,7 @@
       "ge2e_split_batch": false,
       "loss_type": "ge2e_softmax",
       "margin": 0.2,
+      "margin_warmup_steps": 0,
       "multi_row_window": 0,
       "num_speakers_per_batch": 8,
       "num_utts_per_speaker": 4,
@@ -213,7 +214,7 @@
         "use_positional_encoding": false
       },
       "use_transformer": false,
-      "variable_length_training": false
+      "variable_length_training": true
     },
     "train": {
       "batch_size": 32,
@@ -232,6 +233,7 @@
         "ge2e_split_batch": false,
         "loss_type": "ge2e_softmax",
         "margin": 0.2,
+        "margin_warmup_steps": 0,
         "multi_row_window": 0,
         "num_speakers_per_batch": 8,
         "num_utts_per_speaker": 4,
@@ -249,7 +251,7 @@
       "min_learning_rate": 1.5e-05,
       "num_processes": 4,
       "num_spks_per_batch": 8,
-      "num_steps": 400,
+      "num_steps": 2400,
       "num_utts_per_spk": 4,
       "optimizer": "adamw",
       "save_model_frequency": 0,
@@ -267,7 +269,7 @@
         "use_specaug": true
       },
       "triplet_alpha": 0.1,
-      "warmup_steps": 40,
+      "warmup_steps": 240,
       "waveform_aug": {
         "aug_prob": 0.35,
         "enabled": true,
@@ -306,17 +308,17 @@
   "scoring_type": "cosine",
   "seq_len": 180,
   "size_variant": "tiny",
-  "transformers_version": "5.17.0",
   "training_provenance": {
-    "variable_length_training_actual": false,
-    "crop_frames": 180,
-    "training_code": "uncommitted working tree before 1cb90c7 (2026-10-15): get_batched_nm_input / get_batched_classification_input defaulted to variable_length=False and run_research_benchmarks.py did not pass the flag, so crops were fixed seq_len=180 frames with all-ones masks although the serialized config said variable_length_training=True",
-    "step0_fingerprint_match": "fixed",
-    "released_initial_loss": 2.09,
+    "variable_length_training_actual": true,
+    "crop_frames": "90-180",
+    "training_code": "9accf95 (2026-10-18) or later: variable_length_training honoured by the benchmark driver (T ~ U(min_seq_len, seq_len) crops with an exact validity mask)",
+    "step0_fingerprint_match": "variable",
+    "released_initial_loss": 2.1332,
     "head_fixed_step0_loss": 2.09,
     "head_variable_step0_loss": 2.1332,
-    "fingerprint_code_commit": "ea7ebcc+dirty",
+    "fingerprint_code_commit": "9b091a5+dirty",
     "fingerprint_source": "pretrained_models/step0_fingerprints.json",
     "note": "How the released weights were actually trained. Evaluating the released weights with the current code reproduces the archived metrics; re-training the shipped recipe reproduces the archived step-0 loss only under 'crop_frames'. See README.md 'Training-crop provenance'."
-  }
+  },
+  "transformers_version": "5.17.0"
 }

eval_results.json

@@ -8,158 +8,455 @@
   "scoring_type": "cosine",
   "embedding_dim": 128,
   "num_parameters": 693126,
-  "num_steps": 400,
-  "train_time_sec": 288.25,
-  "initial_loss": 2.09,
-  "final_loss": 0.3523,
+  "num_steps": 2400,
+  "knee_step": 2000,
+  "safety_margin_steps": 400,
+  "warmup_steps": 240,
+  "margin_warmup_steps": 0,
+  "train_epochs": 13.27,
+  "train_time_sec": 742.39,
+  "initial_loss": 2.1332,
+  "final_loss": 0.2095,
+  "final_loss_last_20": 0.2049,
+  "loss_curve_100": [
+    {
+      "step": 100,
+      "epochs": 0.55,
+      "mean_loss_last_100": 1.4005,
+      "step_loss": 1.1362
+    },
+    {
+      "step": 200,
+      "epochs": 1.11,
+      "mean_loss_last_100": 0.9651,
+      "step_loss": 0.9078
+    },
+    {
+      "step": 300,
+      "epochs": 1.66,
+      "mean_loss_last_100": 0.8555,
+      "step_loss": 0.6599
+    },
+    {
+      "step": 400,
+      "epochs": 2.21,
+      "mean_loss_last_100": 0.6899,
+      "step_loss": 0.7487
+    },
+    {
+      "step": 500,
+      "epochs": 2.76,
+      "mean_loss_last_100": 0.6756,
+      "step_loss": 0.5935
+    },
+    {
+      "step": 600,
+      "epochs": 3.32,
+      "mean_loss_last_100": 0.5563,
+      "step_loss": 0.2987
+    },
+    {
+      "step": 700,
+      "epochs": 3.87,
+      "mean_loss_last_100": 0.5475,
+      "step_loss": 0.9328
+    },
+    {
+      "step": 800,
+      "epochs": 4.42,
+      "mean_loss_last_100": 0.4753,
+      "step_loss": 0.7237
+    },
+    {
+      "step": 900,
+      "epochs": 4.98,
+      "mean_loss_last_100": 0.4726,
+      "step_loss": 0.465
+    },
+    {
+      "step": 1000,
+      "epochs": 5.53,
+      "mean_loss_last_100": 0.3988,
+      "step_loss": 0.3727
+    },
+    {
+      "step": 1100,
+      "epochs": 6.08,
+      "mean_loss_last_100": 0.3754,
+      "step_loss": 0.3667
+    },
+    {
+      "step": 1200,
+      "epochs": 6.63,
+      "mean_loss_last_100": 0.3834,
+      "step_loss": 0.3186
+    },
+    {
+      "step": 1300,
+      "epochs": 7.19,
+      "mean_loss_last_100": 0.3611,
+      "step_loss": 0.2688
+    },
+    {
+      "step": 1400,
+      "epochs": 7.74,
+      "mean_loss_last_100": 0.3126,
+      "step_loss": 0.3597
+    },
+    {
+      "step": 1500,
+      "epochs": 8.29,
+      "mean_loss_last_100": 0.2897,
+      "step_loss": 0.4084
+    },
+    {
+      "step": 1600,
+      "epochs": 8.85,
+      "mean_loss_last_100": 0.2738,
+      "step_loss": 0.1689
+    },
+    {
+      "step": 1700,
+      "epochs": 9.4,
+      "mean_loss_last_100": 0.3008,
+      "step_loss": 0.5918
+    },
+    {
+      "step": 1800,
+      "epochs": 9.95,
+      "mean_loss_last_100": 0.2531,
+      "step_loss": 0.0934
+    },
+    {
+      "step": 1900,
+      "epochs": 10.5,
+      "mean_loss_last_100": 0.2542,
+      "step_loss": 0.2018
+    },
+    {
+      "step": 2000,
+      "epochs": 11.06,
+      "mean_loss_last_100": 0.2307,
+      "step_loss": 0.2308
+    },
+    {
+      "step": 2100,
+      "epochs": 11.61,
+      "mean_loss_last_100": 0.2365,
+      "step_loss": 0.4082
+    },
+    {
+      "step": 2200,
+      "epochs": 12.16,
+      "mean_loss_last_100": 0.2227,
+      "step_loss": 0.8825
+    },
+    {
+      "step": 2300,
+      "epochs": 12.72,
+      "mean_loss_last_100": 0.2235,
+      "step_loss": 0.338
+    },
+    {
+      "step": 2400,
+      "epochs": 13.27,
+      "mean_loss_last_100": 0.2095,
+      "step_loss": 0.1908
+    }
+  ],
+  "convergence_checkpoints": {
+    "step_400": {
+      "step": 400,
+      "epochs": 2.21,
+      "mean_loss_last_100": 0.6899,
+      "in_domain_1utt_eer_percent": 13.0667,
+      "in_domain_3utt_eer_percent": 7.9,
+      "pooled_1utt_eer_percent": 24.475,
+      "pooled_3utt_eer_percent": 19.2667,
+      "macro_avg_1utt_eer_percent": 22.71,
+      "macro_avg_3utt_eer_percent": 18.1,
+      "domains_1utt_eer_percent": {
+        "librispeech": 13.0667,
+        "cnceleb2": 32.3667,
+        "voxceleb1": 22.7
+      },
+      "domains_3utt_eer_percent": {
+        "librispeech": 7.9,
+        "cnceleb2": 27.2,
+        "voxceleb1": 19.2
+      },
+      "in_domain_1utt_full": {
+        "eer": 0.13066666666666668,
+        "eer_percent": 13.0667,
+        "eer_ci95_low_percent": 11.7333,
+        "eer_ci95_high_percent": 14.3,
+        "eer_threshold": 0.8140120506286621,
+        "min_dcf_001": 0.8767,
+        "min_dcf_005": 0.7193,
+        "min_dcf_01": 0.5693,
+        "roc_auc": 0.9476,
+        "num_trials": 3000,
+        "num_unique_utterances": 999,
+        "latency_ms_per_utterance": 8.931,
+        "eval_time_sec": 9.801,
+        "scoring_type": "cosine"
+      },
+      "in_domain_3utt_full": {
+        "eer": 0.07899999999999999,
+        "eer_percent": 7.9,
+        "eer_ci95_low_percent": 6.1,
+        "eer_ci95_high_percent": 9.6525,
+        "eer_threshold": 0.8586741089820862,
+        "min_dcf_001": 0.434,
+        "min_dcf_005": 0.386,
+        "min_dcf_01": 0.304,
+        "roc_auc": 0.98,
+        "num_trials": 1000
+      },
+      "universal_macro_avg_1utt": {
+        "eer_percent": 22.71,
+        "min_dcf_001": 0.9242,
+        "min_dcf_01": 0.782,
+        "roc_auc": 0.8475
+      },
+      "universal_macro_avg_3utt": {
+        "eer_percent": 18.1,
+        "min_dcf_001": 0.7467,
+        "min_dcf_01": 0.662,
+        "roc_auc": 0.8928
+      }
+    },
+    "step_X_knee": {
+      "step": 2000,
+      "epochs": 11.06,
+      "mean_loss_last_100": 0.2307,
+      "in_domain_1utt_eer_percent": 7.5667,
+      "in_domain_3utt_eer_percent": 4.3,
+      "pooled_1utt_eer_percent": 20.7875,
+      "pooled_3utt_eer_percent": 18.2667,
+      "macro_avg_1utt_eer_percent": 21.13,
+      "macro_avg_3utt_eer_percent": 17.63,
+      "domains_1utt_eer_percent": {
+        "librispeech": 7.5667,
+        "cnceleb2": 30.0667,
+        "voxceleb1": 25.75
+      },
+      "domains_3utt_eer_percent": {
+        "librispeech": 4.3,
+        "cnceleb2": 25.9,
+        "voxceleb1": 22.7
+      },
+      "in_domain_1utt_full": {
+        "eer": 0.07566666666666669,
+        "eer_percent": 7.5667,
+        "eer_ci95_low_percent": 6.5667,
+        "eer_ci95_high_percent": 8.5333,
+        "eer_threshold": 0.544470489025116,
+        "min_dcf_001": 0.6467,
+        "min_dcf_005": 0.5007,
+        "min_dcf_01": 0.388,
+        "roc_auc": 0.9743,
+        "num_trials": 3000,
+        "num_unique_utterances": 999,
+        "latency_ms_per_utterance": 7.254,
+        "eval_time_sec": 7.936,
+        "scoring_type": "cosine"
+      },
+      "in_domain_3utt_full": {
+        "eer": 0.04300000000000002,
+        "eer_percent": 4.3,
+        "eer_ci95_low_percent": 2.8475,
+        "eer_ci95_high_percent": 5.5525,
+        "eer_threshold": 0.592853307723999,
+        "min_dcf_001": 0.484,
+        "min_dcf_005": 0.232,
+        "min_dcf_01": 0.172,
+        "roc_auc": 0.9904,
+        "num_trials": 1000
+      },
+      "universal_macro_avg_1utt": {
+        "eer_percent": 21.13,
+        "min_dcf_001": 0.83,
+        "min_dcf_01": 0.7309,
+        "roc_auc": 0.8574
+      },
+      "universal_macro_avg_3utt": {
+        "eer_percent": 17.63,
+        "min_dcf_001": 0.7433,
+        "min_dcf_01": 0.5833,
+        "roc_auc": 0.8967
+      }
+    },
+    "step_S_safe_final": {
+      "step": 2400,
+      "epochs": 13.27,
+      "mean_loss_last_100": 0.2095,
+      "in_domain_1utt_eer_percent": 7.8667,
+      "in_domain_3utt_eer_percent": 4.6,
+      "pooled_1utt_eer_percent": 20.7,
+      "pooled_3utt_eer_percent": 18.2,
+      "macro_avg_1utt_eer_percent": 21.2,
+      "macro_avg_3utt_eer_percent": 17.37,
+      "domains_1utt_eer_percent": {
+        "librispeech": 7.8667,
+        "cnceleb2": 29.4333,
+        "voxceleb1": 26.3
+      },
+      "domains_3utt_eer_percent": {
+        "librispeech": 4.6,
+        "cnceleb2": 25.2,
+        "voxceleb1": 22.3
+      },
+      "in_domain_1utt_full": {
+        "eer": 0.07866666666666666,
+        "eer_percent": 7.8667,
+        "eer_ci95_low_percent": 6.8825,
+        "eer_ci95_high_percent": 8.8508,
+        "eer_threshold": 0.5138107538223267,
+        "min_dcf_001": 0.6067,
+        "min_dcf_005": 0.464,
+        "min_dcf_01": 0.3727,
+        "roc_auc": 0.9761,
+        "num_trials": 3000,
+        "num_unique_utterances": 999,
+        "latency_ms_per_utterance": 7.903,
+        "eval_time_sec": 8.598,
+        "scoring_type": "cosine"
+      },
+      "in_domain_3utt_full": {
+        "eer": 0.04600000000000002,
+        "eer_percent": 4.6,
+        "eer_ci95_low_percent": 3.2475,
+        "eer_ci95_high_percent": 6.0,
+        "eer_threshold": 0.5838436484336853,
+        "min_dcf_001": 0.48,
+        "min_dcf_005": 0.248,
+        "min_dcf_01": 0.194,
+        "roc_auc": 0.9903,
+        "num_trials": 1000
+      },
+      "universal_macro_avg_1utt": {
+        "eer_percent": 21.2,
+        "min_dcf_001": 0.8339,
+        "min_dcf_01": 0.7116,
+        "roc_auc": 0.8608
+      },
+      "universal_macro_avg_3utt": {
+        "eer_percent": 17.37,
+        "min_dcf_001": 0.7293,
+        "min_dcf_01": 0.584,
+        "roc_auc": 0.8997
+      }
+    }
+  },
   "in_domain_eval": {
-    "eer": 0.1053333333333333,
-    "eer_percent": 10.5333,
-    "eer_ci95_low_percent": 9.4,
-    "eer_ci95_high_percent": 11.5333,
-    "eer_threshold": 0.6059616804122925,
-    "min_dcf_001": 0.772,
-    "min_dcf_005": 0.6207,
-    "min_dcf_01": 0.4893,
-    "roc_auc": 0.9608,
+    "eer": 0.07866666666666666,
+    "eer_percent": 7.8667,
+    "eer_ci95_low_percent": 6.8825,
+    "eer_ci95_high_percent": 8.8508,
+    "eer_threshold": 0.5138107538223267,
+    "min_dcf_001": 0.6067,
+    "min_dcf_005": 0.464,
+    "min_dcf_01": 0.3727,
+    "roc_auc": 0.9761,
     "num_trials": 3000,
     "num_unique_utterances": 999,
-    "latency_ms_per_utterance": 17.21,
-    "eval_time_sec": 18.924,
+    "latency_ms_per_utterance": 7.903,
+    "eval_time_sec": 8.598,
     "scoring_type": "cosine"
   },
   "multi_enroll_3utt_eval": {
-    "eer": 0.06200000000000003,
-    "eer_percent": 6.2,
-    "eer_ci95_low_percent": 4.8,
-    "eer_ci95_high_percent": 7.9525,
-    "eer_threshold": 0.664314866065979,
-    "min_dcf_001": 0.488,
-    "min_dcf_005": 0.34,
-    "min_dcf_01": 0.28,
-    "roc_auc": 0.9845,
+    "eer": 0.04600000000000002,
+    "eer_percent": 4.6,
+    "eer_ci95_low_percent": 3.2475,
+    "eer_ci95_high_percent": 6.0,
+    "eer_threshold": 0.5838436484336853,
+    "min_dcf_001": 0.48,
+    "min_dcf_005": 0.248,
+    "min_dcf_01": 0.194,
+    "roc_auc": 0.9903,
     "num_trials": 1000
   },
-  "cross_dataset_eval": {
-    "cnceleb2_zero_shot": {
-      "eer": 0.30000000000000004,
-      "eer_percent": 30.0,
-      "eer_ci95_low_percent": 28.6492,
-      "eer_ci95_high_percent": 31.8667,
-      "eer_threshold": 0.6296399831771851,
-      "min_dcf_001": 0.9993,
-      "min_dcf_005": 0.9993,
-      "min_dcf_01": 0.9547,
-      "roc_auc": 0.7787,
-      "num_trials": 3000,
-      "num_unique_utterances": 739,
-      "latency_ms_per_utterance": 17.479,
-      "eval_time_sec": 14.073,
-      "scoring_type": "cosine"
-    },
-    "vox1_o_10spk_subset": {
-      "eer": 0.28655661768000884,
-      "eer_percent": 28.6557,
-      "eer_ci95_low_percent": 26.5857,
-      "eer_ci95_high_percent": 30.6141,
-      "eer_threshold": 0.5835728645324707,
-      "min_dcf_001": 0.9838,
-      "min_dcf_005": 0.9334,
-      "min_dcf_01": 0.9193,
-      "roc_auc": 0.7962,
-      "num_trials": 2500,
-      "num_unique_utterances": 937,
-      "latency_ms_per_utterance": 4.532,
-      "eval_time_sec": 4.8,
-      "scoring_type": "cosine",
-      "num_target_trials": 1789,
-      "num_impostor_trials": 711,
-      "num_eval_speakers": 10,
-      "protocol": "veri_test2_10spk_subset_2500trials",
-      "vad_energy_db_threshold": 26.0
-    }
-  },
-  "config_yaml": "configs/ls_conformer_tiny_ge2e.yml",
   "multi_domain_eval": {
     "domains_1utt": {
       "librispeech": {
-        "eer": 0.1053333333333333,
-        "eer_percent": 10.5333,
-        "eer_ci95_low_percent": 9.4,
-        "eer_ci95_high_percent": 11.5333,
-        "eer_threshold": 0.6059616804122925,
-        "min_dcf_001": 0.772,
-        "min_dcf_005": 0.6207,
-        "min_dcf_01": 0.4893,
-        "roc_auc": 0.9608,
+        "eer": 0.07866666666666666,
+        "eer_percent": 7.8667,
+        "eer_ci95_low_percent": 6.8825,
+        "eer_ci95_high_percent": 8.8508,
+        "eer_threshold": 0.5138107538223267,
+        "min_dcf_001": 0.6067,
+        "min_dcf_005": 0.464,
+        "min_dcf_01": 0.3727,
+        "roc_auc": 0.9761,
         "num_trials": 3000,
         "num_unique_utterances": 999,
-        "latency_ms_per_utterance": 17.21,
-        "eval_time_sec": 18.924,
+        "latency_ms_per_utterance": 7.903,
+        "eval_time_sec": 8.598,
         "scoring_type": "cosine"
       },
       "cnceleb2": {
-        "eer": 0.30000000000000004,
-        "eer_percent": 30.0,
-        "eer_ci95_low_percent": 28.6492,
-        "eer_ci95_high_percent": 31.8667,
-        "eer_threshold": 0.6296399831771851,
-        "min_dcf_001": 0.9993,
-        "min_dcf_005": 0.9993,
-        "min_dcf_01": 0.9547,
-        "roc_auc": 0.7787,
+        "eer": 0.29433333333333334,
+        "eer_percent": 29.4333,
+        "eer_ci95_low_percent": 27.6983,
+        "eer_ci95_high_percent": 30.9017,
+        "eer_threshold": 0.5394164323806763,
+        "min_dcf_001": 0.998,
+        "min_dcf_005": 0.9933,
+        "min_dcf_01": 0.94,
+        "roc_auc": 0.7878,
         "num_trials": 3000,
         "num_unique_utterances": 739,
-        "latency_ms_per_utterance": 17.479,
-        "eval_time_sec": 14.073,
+        "latency_ms_per_utterance": 7.865,
+        "eval_time_sec": 6.479,
         "scoring_type": "cosine"
       },
       "voxceleb1": {
-        "eer": 0.257,
-        "eer_percent": 25.7,
-        "eer_ci95_low_percent": 23.9,
-        "eer_ci95_high_percent": 27.4763,
-        "eer_threshold": 0.5478366613388062,
-        "min_dcf_001": 0.938,
-        "min_dcf_005": 0.918,
-        "min_dcf_01": 0.895,
-        "roc_auc": 0.8282,
+        "eer": 0.263,
+        "eer_percent": 26.3,
+        "eer_ci95_low_percent": 24.45,
+        "eer_ci95_high_percent": 28.5762,
+        "eer_threshold": 0.4254768192768097,
+        "min_dcf_001": 0.897,
+        "min_dcf_005": 0.837,
+        "min_dcf_01": 0.822,
+        "roc_auc": 0.8184,
         "num_trials": 2000,
         "num_unique_utterances": 250,
-        "latency_ms_per_utterance": 21.465,
-        "eval_time_sec": 6.525,
+        "latency_ms_per_utterance": 7.682,
+        "eval_time_sec": 2.356,
         "scoring_type": "cosine"
       },
       "universal_pooled": {
-        "eer": 0.219125,
-        "eer_percent": 21.9125,
-        "eer_ci95_low_percent": 21.05,
-        "eer_ci95_high_percent": 22.8125,
-        "eer_threshold": 0.6035156846046448,
-        "min_dcf_001": 0.9968,
-        "min_dcf_005": 0.899,
-        "min_dcf_01": 0.8525,
-        "roc_auc": 0.8711,
+        "eer": 0.20699999999999996,
+        "eer_percent": 20.7,
+        "eer_ci95_low_percent": 19.875,
+        "eer_ci95_high_percent": 21.6381,
+        "eer_threshold": 0.5001381635665894,
+        "min_dcf_001": 0.993,
+        "min_dcf_005": 0.8802,
+        "min_dcf_01": 0.8,
+        "roc_auc": 0.8787,
         "num_trials": 8000,
         "num_unique_utterances": 1988,
         "latency_ms_per_utterance": 0.0,
-        "eval_time_sec": 3.448,
+        "eval_time_sec": 1.5,
         "scoring_type": "cosine"
       },
       "vox1_o_10spk_subset": {
-        "eer": 0.28655661768000884,
-        "eer_percent": 28.6557,
-        "eer_ci95_low_percent": 26.5857,
-        "eer_ci95_high_percent": 30.6141,
-        "eer_threshold": 0.5835728645324707,
-        "min_dcf_001": 0.9838,
-        "min_dcf_005": 0.9334,
-        "min_dcf_01": 0.9193,
-        "roc_auc": 0.7962,
+        "eer": 0.2995933895135061,
+        "eer_percent": 29.9593,
+        "eer_ci95_low_percent": 27.6502,
+        "eer_ci95_high_percent": 32.1457,
+        "eer_threshold": 0.466065913438797,
+        "min_dcf_001": 0.9816,
+        "min_dcf_005": 0.9165,
+        "min_dcf_01": 0.8884,
+        "roc_auc": 0.7831,
         "num_trials": 2500,
         "num_unique_utterances": 937,
-        "latency_ms_per_utterance": 4.532,
-        "eval_time_sec": 4.8,
+        "latency_ms_per_utterance": 4.503,
+        "eval_time_sec": 4.771,
         "scoring_type": "cosine",
         "num_target_trials": 1789,
         "num_impostor_trials": 711,
@@ -169,73 +466,113 @@
       }
     },
     "universal_macro_avg_1utt": {
-      "eer_percent": 22.08,
-      "min_dcf_001": 0.9031,
-      "min_dcf_01": 0.7797,
-      "roc_auc": 0.8559
+      "eer_percent": 21.2,
+      "min_dcf_001": 0.8339,
+      "min_dcf_01": 0.7116,
+      "roc_auc": 0.8608
     },
     "domains_3utt": {
       "librispeech": {
-        "eer": 0.06200000000000003,
-        "eer_percent": 6.2,
-        "eer_ci95_low_percent": 4.8,
-        "eer_ci95_high_percent": 7.9525,
-        "eer_threshold": 0.664314866065979,
-        "min_dcf_001": 0.488,
-        "min_dcf_005": 0.34,
-        "min_dcf_01": 0.28,
-        "roc_auc": 0.9845,
+        "eer": 0.04600000000000002,
+        "eer_percent": 4.6,
+        "eer_ci95_low_percent": 3.2475,
+        "eer_ci95_high_percent": 6.0,
+        "eer_threshold": 0.5838436484336853,
+        "min_dcf_001": 0.48,
+        "min_dcf_005": 0.248,
+        "min_dcf_01": 0.194,
+        "roc_auc": 0.9903,
         "num_trials": 1000
       },
       "cnceleb2": {
-        "eer": 0.258,
-        "eer_percent": 25.8,
-        "eer_ci95_low_percent": 23.2,
-        "eer_ci95_high_percent": 28.8,
-        "eer_threshold": 0.7029243111610413,
-        "min_dcf_001": 0.896,
-        "min_dcf_005": 0.884,
-        "min_dcf_01": 0.862,
-        "roc_auc": 0.8256,
+        "eer": 0.252,
+        "eer_percent": 25.2,
+        "eer_ci95_low_percent": 22.6,
+        "eer_ci95_high_percent": 27.9525,
+        "eer_threshold": 0.6346871256828308,
+        "min_dcf_001": 0.936,
+        "min_dcf_005": 0.918,
+        "min_dcf_01": 0.81,
+        "roc_auc": 0.8397,
         "num_trials": 1000
       },
       "voxceleb1": {
-        "eer": 0.21599999999999997,
-        "eer_percent": 21.6,
-        "eer_ci95_low_percent": 19.0,
-        "eer_ci95_high_percent": 23.905,
-        "eer_threshold": 0.6436901688575745,
-        "min_dcf_001": 0.91,
-        "min_dcf_005": 0.91,
-        "min_dcf_01": 0.876,
-        "roc_auc": 0.8708,
+        "eer": 0.22299999999999998,
+        "eer_percent": 22.3,
+        "eer_ci95_low_percent": 19.8475,
+        "eer_ci95_high_percent": 24.9,
+        "eer_threshold": 0.5039066076278687,
+        "min_dcf_001": 0.772,
+        "min_dcf_005": 0.768,
+        "min_dcf_01": 0.748,
+        "roc_auc": 0.8691,
         "num_trials": 1000
       },
       "universal_pooled": {
-        "eer": 0.1806666666666667,
-        "eer_percent": 18.0667,
-        "eer_ci95_low_percent": 16.5,
-        "eer_ci95_high_percent": 19.3333,
-        "eer_threshold": 0.6769924759864807,
-        "min_dcf_001": 0.8647,
-        "min_dcf_005": 0.8347,
-        "min_dcf_01": 0.7707,
-        "roc_auc": 0.9049,
+        "eer": 0.18200000000000002,
+        "eer_percent": 18.2,
+        "eer_ci95_low_percent": 16.6983,
+        "eer_ci95_high_percent": 19.4842,
+        "eer_threshold": 0.5760137438774109,
+        "min_dcf_001": 0.9127,
+        "min_dcf_005": 0.7153,
+        "min_dcf_01": 0.644,
+        "roc_auc": 0.9127,
         "num_trials": 3000
       }
     },
     "universal_macro_avg_3utt": {
-      "eer_percent": 17.87,
-      "min_dcf_001": 0.7647,
-      "min_dcf_01": 0.6727,
-      "roc_auc": 0.8936
+      "eer_percent": 17.37,
+      "min_dcf_001": 0.7293,
+      "min_dcf_01": 0.584,
+      "roc_auc": 0.8997
+    }
+  },
+  "cross_dataset_eval": {
+    "vox1_o_10spk_subset": {
+      "eer": 0.2995933895135061,
+      "eer_percent": 29.9593,
+      "eer_ci95_low_percent": 27.6502,
+      "eer_ci95_high_percent": 32.1457,
+      "eer_threshold": 0.466065913438797,
+      "min_dcf_001": 0.9816,
+      "min_dcf_005": 0.9165,
+      "min_dcf_01": 0.8884,
+      "roc_auc": 0.7831,
+      "num_trials": 2500,
+      "num_unique_utterances": 937,
+      "latency_ms_per_utterance": 4.503,
+      "eval_time_sec": 4.771,
+      "scoring_type": "cosine",
+      "num_target_trials": 1789,
+      "num_impostor_trials": 711,
+      "num_eval_speakers": 10,
+      "protocol": "veri_test2_10spk_subset_2500trials",
+      "vad_energy_db_threshold": 26.0
+    },
+    "cnceleb2_zero_shot": {
+      "eer": 0.29433333333333334,
+      "eer_percent": 29.4333,
+      "eer_ci95_low_percent": 27.6983,
+      "eer_ci95_high_percent": 30.9017,
+      "eer_threshold": 0.5394164323806763,
+      "min_dcf_001": 0.998,
+      "min_dcf_005": 0.9933,
+      "min_dcf_01": 0.94,
+      "roc_auc": 0.7878,
+      "num_trials": 3000,
+      "num_unique_utterances": 739,
+      "latency_ms_per_utterance": 7.865,
+      "eval_time_sec": 6.479,
+      "scoring_type": "cosine"
     }
   },
+  "config_yaml": "configs/ls_conformer_tiny_ge2e.yml",
   "hf_pretrained_dir": "pretrained_models/ls_conformer_tiny_ge2e",
   "hf_repo_id": "wq2012/ls_conformer_tiny_ge2e",
   "hf_url": "https://huggingface.co/wq2012/ls_conformer_tiny_ge2e",
   "hf_collection_url": "https://huggingface.co/collections/wq2012/flaxspeaker",
-  "vox1_o_10spk_subset_eer_percent": 28.6557,
-  "vox1_o_10spk_subset_min_dcf_001": 0.9838,
-  "vox1_o_10spk_subset_roc_auc": 0.7962
+  "vox1_o_10spk_subset_eer_percent": 29.9593,
+  "vox1_o_10spk_subset_min_dcf_001": 0.9816,
+  "vox1_o_10spk_subset_roc_auc": 0.7831
 }

flax_model.msgpack

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:ba075b9072113046f5faa829e7983a38652a7cbbc0b009af3df8305d3b6451b4
+oid sha256:afe65a4da6b04b5941e4a9e15112df0c9583567ddeb894ade1aa898e30b1431b
 size 2775358

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:44309ab69ecbe690735e7879979802a95c0398c98aa76e01d2a2582a30b0eee7
+oid sha256:33a5749411fbd920ad0e17ee7eeeb8d5ff3715bc755db821d8da9127e4e0c39d
 size 2781184