Release v0.2.2: Updated checkpoint, quantized TFLite model, and N=100 evaluation metrics

README.md

@@ -24,7 +24,7 @@ arxiv: 2008.06006
 Single-source sequence-to-sequence **Vanilla-Seq2seq (`NoSideInputSingleInterfering`)** baseline model trained on 24 kHz **LibriTTS** user speech mixed at 0 dB SNR with reverberant **LJ Speech** (single-speaker TTS interference). Takes only the noisy/reverberant microphone spectrogram (`SpeechEncoderV1`) with zero side input.
 
 - **GitHub Repository**: [https://github.com/wq2012/tec](https://github.com/wq2012/tec)
-- **PyPI Package**: [`textual-echo-cancellation` (v0.2.1)](https://pypi.org/project/textual-echo-cancellation/)
+- **PyPI Package**: [`textual-echo-cancellation` (v0.2.2)](https://pypi.org/project/textual-echo-cancellation/)
 - **Interactive Hugging Face Space Demo**: [https://huggingface.co/spaces/wq2012/tec](https://huggingface.co/spaces/wq2012/tec)
 - **Paper**: [Textual Echo Cancellation (IEEE SLT 2021, arXiv:2008.06006v4)](https://arxiv.org/pdf/2008.06006)
 - **Audio Demo Page**: [https://google.github.io/speaker-id/publications/TEC/](https://google.github.io/speaker-id/publications/TEC/)
@@ -47,31 +47,31 @@ Single-source sequence-to-sequence **Vanilla-Seq2seq (`NoSideInputSingleInterfer
 
 ### 1. Open-Source Reproduction Evaluation (`NoSideInputSingleInterfering`, N = 100 mixtures per split)
 
-Evaluated on 24 kHz **LibriTTS** (`test-clean`: N = 100 mixtures, 1,041 reference words; `test-other`: N = 100 mixtures, 1,039 reference words) mixed at 0 dB SNR under the **Single interfering voice (LibriTTS + LJ Speech)** condition, using `Qwen3-ASR-0.6B-F16` via `audio.cpp` for ASR WER scoring and 13-MFCC Dynamic Time Warping for MCD (with 10,000-resample non-parametric bootstrap 95% confidence intervals):
+Evaluated on 24 kHz **LibriTTS** (`test-clean`: N = 100 mixtures, 965 reference words; `test-other`: N = 100 mixtures, 1,001 reference words) mixed at 0 dB SNR under the **Single interfering voice (LibriTTS + LJ Speech)** condition, using `Qwen3-ASR-0.6B-F16` via `audio.cpp` for ASR WER scoring and 13-MFCC Dynamic Time Warping for MCD (with 10,000-resample non-parametric bootstrap 95% confidence intervals):
 
 | Metric | `test-clean` (N = 100) | `test-other` (N = 100) | Paper Reference (`Vanilla-Seq2seq`) |
 | :--- | :---: | :---: | :---: |
-| **WER (%) ↓** | **73.10%** (761/1041, 95% CI [65.02, 81.44]) | **86.81%** (902/1039, 95% CI [79.79, 93.88]) | 25.4% / 54.0% |
-| **MCD (dB) ↓** | **8.08 dB** (95% CI [7.84, 8.33]) | **8.81 dB** (95% CI [8.45, 9.17]) | 7.85 dB / 8.84 dB |
+| **WER (%) ↓** | **30.88%** (298/965, 95% CI [24.11, 38.88]) | **58.54%** (586/1001, 95% CI [49.60, 67.69]) | 25.4% / 54.0% |
+| **MCD (dB) ↓** | **8.34 dB** (95% CI [8.10, 8.58]) | **9.27 dB** (95% CI [8.96, 9.58]) | 7.85 dB / 8.84 dB |
 | **Side Input Size (KB) ↓** | **0.000 KB** (padded: 0.000 KB) | **0.000 KB** (padded: 0.000 KB) | 0 KB |
-| **Complexity (GFLOPS) ↓** | **3.35 GFLOPS** | **3.16 GFLOPS** | 6.32 GFLOPS (401 frames) |
+| **Complexity (GFLOPS) ↓** | **2.79 GFLOPS** | **2.71 GFLOPS** | 6.32 GFLOPS (401 frames) |
 
 ### 2. Full Comparison Across All Pretrained Models & Baselines (N = 100 per split)
 
 | Condition | Method | Hugging Face Model | WER (%) test-clean ↓ | WER (%) test-other ↓ | MCD (dB) test-clean ↓ | MCD (dB) test-other ↓ | Side Input test-clean (KB) ↓ | Avg Complexity (GFLOPS) ↓ |
 | :--- | :--- | :--- | :---: | :---: | :---: | :---: | :---: | :---: |
-| **Single interfering voice** *(LibriTTS + LJSpeech)* | `GroundTruth` | — | 2.59 [1.54, 3.79] | 6.16 [4.07, 8.47] | 0.00 | 0.00 | 0.000 | 0.00 |
-| | `MicrophoneSignal` | — | 80.79 [72.97, 89.04] | 91.15 [84.42, 98.09] | 8.93 [8.62, 9.24] | 9.56 [9.18, 9.93] | 0.000 | 0.00 |
-| | `NlmsAec` (AEC-NLMS) | — | 70.32 [61.76, 79.26] | 87.49 [79.59, 95.49] | 8.76 [8.46, 9.07] | 9.41 [9.04, 9.79] | 189.494 | 0.05 |
-| | `NoSideInputSingleInterfering` | [`wq2012/vanilla_seq2seq_single_interfering`](https://huggingface.co/wq2012/vanilla_seq2seq_single_interfering) | 73.10 [65.02, 81.44] | 86.81 [79.79, 93.88] | 8.08 [7.84, 8.33] | 8.81 [8.45, 9.17] | 0.000 | 3.35 |
-| | `AecSingleInterfering` | [`wq2012/aec_single_interfering`](https://huggingface.co/wq2012/aec_single_interfering) | 8.17 [6.06, 10.42] | 19.35 [15.23, 23.96] | 5.83 [5.59, 6.07] | 6.20 [5.89, 6.53] | 189.494 | 5.11 |
-| | **`TecSingleInterfering`** | **[`wq2012/tec_single_interfering`](https://huggingface.co/wq2012/tec_single_interfering)** | **64.07 [56.32, 72.33]** | **85.37 [78.48, 92.38]** | **7.99 [7.75, 8.24]** | **8.60 [8.26, 8.96]** | **0.059** | **3.71** |
-| **Multiple interfering voices** *(LibriTTS + VCTK)* | `GroundTruth` | — | 2.59 [1.54, 3.79] | 6.35 [4.22, 8.67] | 0.00 | 0.00 | 0.000 | 0.00 |
-| | `MicrophoneSignal` | — | 83.67 [75.45, 91.89] | 91.72 [85.43, 98.27] | 8.07 [7.86, 8.29] | 8.35 [8.06, 8.65] | 0.000 | 0.00 |
-| | `NlmsAec` (AEC-NLMS) | — | 75.02 [66.40, 83.62] | 86.72 [79.65, 94.04] | 7.85 [7.65, 8.06] | 8.17 [7.88, 8.47] | 195.914 | 0.05 |
-| | `NoSideInputMultiInterfering` | [`wq2012/vanilla_seq2seq_multi_interfering`](https://huggingface.co/wq2012/vanilla_seq2seq_multi_interfering) | 77.33 [67.79, 86.82] | 84.79 [77.53, 91.94] | 7.58 [7.36, 7.80] | 7.97 [7.65, 8.30] | 0.000 | 3.46 |
-| | `AecMultiInterfering` | [`wq2012/aec_multi_interfering`](https://huggingface.co/wq2012/aec_multi_interfering) | 9.32 [6.42, 12.54] | 21.56 [16.63, 26.85] | 5.08 [4.92, 5.25] | 5.32 [5.07, 5.57] | 195.914 | 5.29 |
-| | **`TecMultiInterfering`** | **[`wq2012/tec_multi_interfering`](https://huggingface.co/wq2012/tec_multi_interfering)** | **65.90 [57.38, 74.18]** | **78.25 [71.17, 85.23]** | **7.19 [7.02, 7.36]** | **7.53 [7.26, 7.82]** | **0.062** | **3.84** |
+| **Single interfering voice** *(LibriTTS + LJSpeech)* | `GroundTruth` | — | 3.52 [2.09, 5.12] | 5.19 [3.56, 7.06] | 0.00 | 0.00 | 0.000 | 0.00 |
+| | `MicrophoneSignal` | — | 30.67 [22.83, 39.70] | 56.94 [48.04, 66.46] | 7.19 [6.81, 7.60] | 8.35 [7.89, 8.80] | 0.000 | 0.00 |
+| | `NlmsAec` (AEC-NLMS) | — | 58.86 [51.17, 67.37] | 79.92 [73.98, 86.40] | 7.84 [7.54, 8.16] | 9.12 [8.70, 9.54] | 146.667 | 0.02 |
+| | `NoSideInputSingleInterfering` | [`wq2012/vanilla_seq2seq_single_interfering`](https://huggingface.co/wq2012/vanilla_seq2seq_single_interfering) | 30.88 [24.11, 38.88] | 58.54 [49.60, 67.69] | 8.34 [8.10, 8.58] | 9.27 [8.96, 9.58] | 0.000 | 2.79 |
+| | `AecSingleInterfering` | [`wq2012/aec_single_interfering`](https://huggingface.co/wq2012/aec_single_interfering) | 9.12 [6.59, 11.92] | 21.68 [17.73, 26.00] | 6.62 [6.44, 6.80] | 7.20 [6.96, 7.45] | 146.667 | 4.15 |
+| | **`TecSingleInterfering`** | **[`wq2012/tec_single_interfering`](https://huggingface.co/wq2012/tec_single_interfering)** | **20.31 [15.19, 26.71]** | **38.06 [31.47, 45.35]** | **7.00 [6.72, 7.29]** | **7.76 [7.40, 8.15]** | **0.048** | **3.08** |
+| **Multiple interfering voices** *(LibriTTS + VCTK)* | `GroundTruth` | — | 3.97 [2.58, 5.48] | 6.56 [4.69, 8.65] | 0.00 | 0.00 | 0.000 | 0.00 |
+| | `MicrophoneSignal` | — | 20.30 [14.89, 26.57] | 40.16 [33.27, 47.98] | 5.48 [5.14, 5.85] | 5.88 [5.51, 6.27] | 0.000 | 0.00 |
+| | `NlmsAec` (AEC-NLMS) | — | 34.48 [29.08, 40.35] | 55.70 [49.49, 62.39] | 6.89 [6.61, 7.18] | 7.68 [7.29, 8.08] | 136.043 | 0.02 |
+| | `NoSideInputMultiInterfering` | [`wq2012/vanilla_seq2seq_multi_interfering`](https://huggingface.co/wq2012/vanilla_seq2seq_multi_interfering) | 30.83 [24.72, 37.89] | 50.76 [43.86, 58.31] | 8.57 [8.34, 8.81] | 8.76 [8.51, 9.01] | 0.000 | 2.76 |
+| | `AecMultiInterfering` | [`wq2012/aec_multi_interfering`](https://huggingface.co/wq2012/aec_multi_interfering) | 9.77 [7.40, 12.39] | 24.52 [20.60, 28.74] | 5.93 [5.74, 6.15] | 6.16 [5.92, 6.41] | 136.043 | 4.03 |
+| | **`TecMultiInterfering`** | **[`wq2012/tec_multi_interfering`](https://huggingface.co/wq2012/tec_multi_interfering)** | **20.09 [16.11, 24.53]** | **39.25 [33.79, 45.29]** | **6.52 [6.34, 6.70]** | **6.84 [6.60, 7.09]** | **0.035** | **2.98** |
 
 ### 3. Text-Content Sensitivity Ablation (N = 100 per split)
 
@@ -79,21 +79,23 @@ Verifies that `TtsEncoderV2` actively exploits the semantic and phonetic content
 
 | Split | Side-Input Condition | WER (%) ↓ | MCD (dB) ↓ | ΔWER vs. `matched` | Paired p-value (Bootstrap / Wilcoxon) |
 | :--- | :--- | :---: | :---: | :---: | :---: |
-| `single_test_clean` (`TecSingleInterfering`) | **`matched` (true TTS transcript)** | **64.07%** | **7.99 dB** | — | — |
-| | `shuffled` (mismatched transcript) | 71.66% | 8.06 dB | +7.59% | p < 0.0001 / p = 0.00099 |
-| | `empty` (no text / `Vanilla-Seq2seq`) | 73.10% | 8.08 dB | +9.03% | p < 0.0001 / p = 0.00014 |
-| `multi_test_clean` (`TecMultiInterfering`) | **`matched` (true TTS transcript)** | **65.90%** | **7.19 dB** | — | — |
-| | `shuffled` (mismatched transcript) | 75.31% | 7.46 dB | +9.41% | p < 0.0001 / p = 0.00017 |
-| | `empty` (no text / `Vanilla-Seq2seq`) | 77.33% | 7.58 dB | +11.43% | p < 0.0001 / p = 0.00012 |
+| `single_test_clean` (`TecSingleInterfering`) | **`matched` (true TTS transcript)** | **20.31%** | **7.00 dB** | — | — |
+| | `shuffled` (mismatched transcript) | 30.67% | 8.34 dB | +10.36% | p < 0.0001 / p = 1.94e-05 |
+| | `empty` (no text) | 31.09% | 8.61 dB | +10.78% | p < 0.0001 / p = 1.75e-05 |
+| | `Vanilla-Seq2seq` | 30.88% | 8.34 dB | +10.57% | p < 0.0001 / p = 1.28e-04 |
+| `multi_test_clean` (`TecMultiInterfering`) | **`matched` (true TTS transcript)** | **20.09%** | **6.52 dB** | — | — |
+| | `shuffled` (mismatched transcript) | 28.79% | 8.53 dB | +8.70% | p = 0.0002 / p = 8.30e-04 |
+| | `empty` (no text) | 29.97% | 8.59 dB | +9.88% | p < 0.0001 / p = 6.32e-05 |
+| | `Vanilla-Seq2seq` | 30.83% | 8.57 dB | +10.74% | p < 0.0001 / p = 3.10e-05 |
 
 ### 4. Input SNR Ablation on `single_test_clean` (N = 100 mixtures per SNR)
 
 | Input SNR | `MicrophoneSignal` WER / MCD | `NlmsAec` WER / MCD | `Vanilla-Seq2seq` WER / MCD | **`TecSingleInterfering` WER / MCD** | `AecSingleInterfering` WER / MCD |
 | :---: | :---: | :---: | :---: | :---: | :---: |
-| **+5 dB** | 34.01% / 7.89 dB | 25.55% / 7.68 dB | 24.88% / 7.01 dB | **21.81% / 6.92 dB** | 6.05% / 5.33 dB |
-| **0 dB** | 80.79% / 8.93 dB | 70.32% / 8.76 dB | 73.10% / 8.08 dB | **64.07% / 7.99 dB** | 8.17% / 5.83 dB |
-| **-5 dB** | 104.80% / 9.91 dB | 103.55% / 9.78 dB | 103.55% / 9.27 dB | **102.50% / 9.20 dB** | 33.14% / 6.92 dB |
-| **-10 dB** | 107.88% / 10.75 dB | 107.78% / 10.65 dB | 106.82% / 10.19 dB | **106.05% / 10.11 dB** | 88.86% / 8.14 dB |
+| **+5 dB** | 12.23% / 6.09 dB | 44.04% / 7.29 dB | 14.92% / 7.45 dB | **10.16% / 6.14 dB** | 6.63% / 6.15 dB |
+| **0 dB** | 30.67% / 7.19 dB | 58.86% / 7.84 dB | 30.88% / 8.34 dB | **20.31% / 7.00 dB** | 9.12% / 6.62 dB |
+| **-5 dB** | 79.07% / 8.26 dB | 79.07% / 8.46 dB | 78.24% / 9.31 dB | **52.23% / 7.87 dB** | 18.86% / 7.34 dB |
+| **-10 dB** | 103.21% / 9.20 dB | 94.40% / 9.07 dB | 103.63% / 10.16 dB | **93.89% / 8.63 dB** | 53.06% / 8.11 dB |
 
 ---
 

best.ckpt.data-00000-of-00001

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:bfd5bb9189d91e3c8817295a3f789fffeae3d3e2fcd34d7720bf980b44260dfa
-size 150960736
+oid sha256:0a1d07e2783aa8bb222c82ffbbc67fc8f9e18761bf23c0d0c040aa52c695bb59
+size 150145632

best.ckpt.meta

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:ccfadd4c333682a879cc7940b806e9823dfd85a8a3efbf14b42d98658b0eeae3
-size 6235499
+oid sha256:1b875d2cb8a3eb026e651e259de79a6f3580076525b0b5357d469e7b3126581a
+size 220051

model.tflite

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:ae3edb58fb33bcd348ebc173e45ced48f1b6a54acedf5f63862938dad2487fbd
-size 13152264
+oid sha256:dbc26fc93bb9f4f6327e422b51e710d75381388d1b8a218dd3526fc22f9bd042
+size 12863816