README.md

---
license: apache-2.0
library_name: lingvo
tags:
  - audio
  - speech-enhancement
  - echo-cancellation
  - textual-echo-cancellation
  - tensorflow
  - tflite
datasets:
  - libritts
  - ljspeech
metrics:
  - wer
  - mcd
arxiv: 2008.06006
---

# AEC-Seq2seq Neural Baseline — Single Interfering Voice (`AecSingleInterfering`)

**This is not an officially supported Google product.**

Multi-source attention sequence-to-sequence **Acoustic Echo Cancellation (`AEC-Seq2seq`)** neural baseline model trained on 24 kHz **LibriTTS** user speech mixed at 0 dB SNR with reverberant **LJ Speech** (single-speaker TTS interference). Uses dual audio encoders (`SpeechEncoderV1`) on both the microphone mixture and the full reference TTS playback audio (~180–310 KB side input).

- **GitHub Repository**: [https://github.com/wq2012/tec](https://github.com/wq2012/tec)
- **PyPI Package**: [`textual-echo-cancellation` (v0.2.3)](https://pypi.org/project/textual-echo-cancellation/)
- **Interactive Hugging Face Space Demo**: [https://huggingface.co/spaces/wq2012/tec](https://huggingface.co/spaces/wq2012/tec)
- **Paper**: [Textual Echo Cancellation (IEEE SLT 2021, arXiv:2008.06006v4)](https://arxiv.org/pdf/2008.06006)
- **Audio Demo Page**: [https://google.github.io/speaker-id/publications/TEC/](https://google.github.io/speaker-id/publications/TEC/)

> **Open-Source Reproduction Notice**: This pretrained model was trained using the standalone open-source reproduction library ([`wq2012/tec`](https://github.com/wq2012/tec)) built on [`lingvo`](https://github.com/tensorflow/lingvo) and `tensorflow`. The original IEEE SLT 2021 paper was conducted using Google's internal codebase and 281k-utterance internal LibriTTS + TTS datasets rather than this open-source reproduction.

---

## Training Configuration

- **Condition**: Single interfering voice (LibriTTS + LJ Speech)
- **Dataset Splits**: 400 training utterances (`train`), 100 validation utterances (`dev`), 100 `test-clean` utterances, and 100 `test-other` utterances at 24 kHz mixed at 0 dB SNR with synthetic image-method room impulse responses (RT60 = 0.25 s).
- **Architecture**: Speech encoder (`SpeechEncoderV1`: 2× strided 3×3 Conv2D with stride `(2, 2)`, 1× Bi-CLSTM with stride `(1, 1)`, and 3× Bi-LSTM layers), text encoder (`TtsEncoderV2`: 98-symbol ASCII character vocabulary `VOCAB_SIZE = 98`, 512-dim embedding, 3× Conv1D + 1× Bi-LSTM, for TEC models), and multi-source GMM monotonic attention decoder (`MultiSourceFbeDecoderV1` / `FbeDecoderV1`: 2×256 Pre-Net, 2-layer autoregressive LSTM, frame reduction factor `r = 4`, and 5-layer 1D convolutional `PostEditConvNet`).
- **Trainable Parameters**: `18,496,991` (`18,501,730` total model variables; `221.98 MB` checkpoint shard; `19.18 MB` dynamic-range quantized `model.tflite`)
- **Checkpoint Selection**: Validation-loss early stopping (`best.ckpt` selected on the 100-utterance `dev` split over 500 training steps with batch size 8 and Adam `lr = 1e-3`).

---

## Model Performance

### 1. Open-Source Reproduction Evaluation (`AecSingleInterfering`, N = 100 mixtures per split)

Evaluated on 24 kHz **LibriTTS** (`test-clean`: N = 100 mixtures, 1062 reference words; `test-other`: N = 100 mixtures, 1,012 reference words) mixed at 0 dB SNR under the **Single interfering voice (LibriTTS + LJ Speech)** condition, using `Qwen3-ASR-0.6B-F16` via `audio.cpp` for ASR WER scoring and 13-MFCC Dynamic Time Warping for MCD (with 10,000-resample non-parametric bootstrap 95% confidence intervals):

| Metric | `test-clean` (N = 100) | `test-other` (N = 100) | Paper Reference (`AEC-Seq2seq`) |
| :--- | :---: | :---: | :---: |
| **WER (%) ↓** | **7.91%** (84/1062, 95% CI [5.68, 10.48]) | **17.29%** (175/1012, 95% CI [13.34, 21.76]) | 8.30% / 24.3% |
| **MCD (dB) ↓** | **4.97 dB** (95% CI [4.60, 5.35]) | **4.96 dB** (95% CI [4.61, 5.31]) | 6.38 dB / 7.07 dB |
| **Side Input Size (KB) ↓** | **235.403 KB** (padded: 235.403 KB) | **236.991 KB** (padded: 236.991 KB) | 310 KB |
| **Complexity (GFLOPS) ↓** | **6.36 GFLOPS** | **6.40 GFLOPS** | 9.51 GFLOPS (401 frames) |

### 2. Full Comparison Across All Pretrained Models & Baselines (N = 100 per split)

| Condition | Method | Hugging Face Model | WER (%) test-clean ↓ | WER (%) test-other ↓ | MCD (dB) test-clean ↓ | MCD (dB) test-other ↓ | Side Input test-clean (KB) ↓ | Avg Complexity (GFLOPS) ↓ |
| :--- | :--- | :--- | :---: | :---: | :---: | :---: | :---: | :---: |
| **Single interfering voice** *(LibriTTS + LJSpeech)* | `GroundTruth` | — | 3.86 [2.43, 5.49] | 6.62 [4.66, 8.78] | 0.00 | 0.00 | 0.000 | 0.00 |
| | `MicrophoneSignal` | — | 80.32 [68.49, 93.55] | 102.77 [88.25, 119.32] | 9.92 [9.56, 10.27] | 10.74 [10.32, 11.16] | 0.000 | 0.00 |
| | `NlmsAec` (AEC-NLMS) | — | 58.38 [47.37, 71.05] | 85.77 [71.61, 102.19] | 9.51 [9.16, 9.86] | 10.33 [9.92, 10.75] | 235.403 | 0.24 |
| | `NoSideInputSingleInterfering` | [`wq2012/vanilla_seq2seq_single_interfering`](https://huggingface.co/wq2012/vanilla_seq2seq_single_interfering) | 39.08 [30.26, 48.91] | 54.35 [44.66, 64.93] | 7.07 [6.86, 7.29] | 7.38 [7.13, 7.64] | 0.000 | 4.17 |
| | `AecSingleInterfering` | [`wq2012/aec_single_interfering`](https://huggingface.co/wq2012/aec_single_interfering) | 7.91 [5.68, 10.48] | 17.29 [13.34, 21.76] | 4.97 [4.60, 5.35] | 4.96 [4.61, 5.31] | 235.403 | 6.36 |
| | **`TecSingleInterfering`** | **[`wq2012/tec_single_interfering`](https://huggingface.co/wq2012/tec_single_interfering)** | **29.76 [23.09, 36.97]** | **50.59 [41.96, 59.66]** | **5.52 [5.16, 5.89]** | **5.55 [5.18, 5.93]** | **0.074** | **4.62** |
| **Multiple interfering voices** *(LibriTTS + VCTK)* | `GroundTruth` | — | 3.06 [1.90, 4.35] | 6.85 [4.77, 9.16] | 0.00 | 0.00 | 0.000 | 0.00 |
| | `MicrophoneSignal` | — | 50.77 [40.86, 62.15] | 67.33 [56.40, 79.77] | 7.50 [7.13, 7.87] | 7.82 [7.41, 8.24] | 0.000 | 0.00 |
| | `NlmsAec` (AEC-NLMS) | — | 34.22 [25.72, 44.51] | 48.58 [39.15, 59.91] | 7.01 [6.67, 7.37] | 7.43 [7.04, 7.83] | 203.138 | 0.21 |
| | `NoSideInputMultiInterfering` | [`wq2012/vanilla_seq2seq_multi_interfering`](https://huggingface.co/wq2012/vanilla_seq2seq_multi_interfering) | 63.33 [55.27, 71.69] | 69.86 [61.89, 78.18] | 8.26 [7.76, 8.76] | 9.32 [8.77, 9.89] | 0.000 | 3.59 |
| | `AecMultiInterfering` | [`wq2012/aec_multi_interfering`](https://huggingface.co/wq2012/aec_multi_interfering) | 6.84 [4.98, 8.85] | 14.86 [11.25, 18.79] | 4.65 [4.28, 5.02] | 4.67 [4.28, 5.06] | 203.138 | 5.48 |
| | **`TecMultiInterfering`** | **[`wq2012/tec_multi_interfering`](https://huggingface.co/wq2012/tec_multi_interfering)** | **24.31 [18.94, 30.47]** | **36.25 [30.34, 42.52]** | **4.74 [4.42, 5.07]** | **4.91 [4.54, 5.30]** | **0.050** | **3.90** |

### 3. Text-Content Sensitivity Ablation (N = 100 per split)

Verifies that `TtsEncoderV2` actively exploits the semantic and phonetic content of the interfering TTS transcript rather than acting as a content-independent bias:

| Split | Side-Input Condition | WER (%) ↓ | MCD (dB) ↓ | ΔWER vs. `matched` | Paired p-value (Bootstrap / Wilcoxon) |
| :--- | :--- | :---: | :---: | :---: | :---: |
| `single_test_clean` (`TecSingleInterfering`) | **`matched` (true TTS transcript)** | **29.76%** | **5.52 dB** | — | — |
| | `shuffled` (mismatched transcript) | 70.90% | 9.26 dB | +41.15% | p < 0.0001 / p = 2.61e-12 |
| | `empty` (no text) | 71.94% | 9.32 dB | +42.18% | p < 0.0001 / p = 4.89e-13 |
| | `Vanilla-Seq2seq` | 39.08% | 7.07 dB | +9.32% | p < 0.0001 / p = 6.69e-04 |
| `multi_test_clean` (`TecMultiInterfering`) | **`matched` (true TTS transcript)** | **24.31%** | **4.74 dB** | — | — |
| | `shuffled` (mismatched transcript) | 48.52% | 7.92 dB | +24.21% | p < 0.0001 / p = 9.10e-09 |
| | `empty` (no text) | 47.29% | 7.83 dB | +22.98% | p < 0.0001 / p = 1.41e-07 |
| | `Vanilla-Seq2seq` | 63.33% | 8.26 dB | +39.02% | p < 0.0001 / p = 1.99e-11 |

### 4. Input SNR Ablation on `single_test_clean` (N = 100 mixtures per SNR)

| Input SNR | `MicrophoneSignal` WER / MCD | `NlmsAec` WER / MCD | `Vanilla-Seq2seq` WER / MCD | **`TecSingleInterfering` WER / MCD** | `AecSingleInterfering` WER / MCD |
| :---: | :---: | :---: | :---: | :---: | :---: |
| **+5 dB** | 51.88% / 9.03 dB | 40.58% / 8.72 dB | 9.23% / 6.28 dB | **8.47% / 4.69 dB** | 5.65% / 4.33 dB |
| **0 dB** | 80.32% / 9.92 dB | 58.38% / 9.51 dB | 39.08% / 7.07 dB | **29.76% / 5.52 dB** | 7.91% / 4.97 dB |
| **-5 dB** | 119.02% / 10.73 dB | 90.30% / 10.28 dB | 96.80% / 7.82 dB | **87.76% / 6.34 dB** | 17.23% / 5.73 dB |
| **-10 dB** | 124.86% / 11.40 dB | 121.28% / 10.95 dB | 103.86% / 8.35 dB | **101.04% / 6.91 dB** | 49.25% / 6.33 dB |

---

## Files in This Repository

- `best.ckpt.data-00000-of-00001`, `best.ckpt.index`, `best.ckpt.meta`, `checkpoint`: TensorFlow / Lingvo checkpoint for `AecSingleInterfering` (selected by minimum validation loss on `dev`).
- `model.tflite`: Dynamic-range quantized TensorFlow Lite (`.tflite`) FlatBuffer model for `AecSingleInterfering` (requires TFLite Flex `SELECT_TF_OPS` runtime for dynamic RNN `TensorList` operations).
- `evaluation_metrics.json`: Verified N = 100 evaluation results (`test-clean` and `test-other`), 95% bootstrap confidence intervals, paired significance tests, and ablations for `AecSingleInterfering`.

---

## How to Use

### 1. Install `textual-echo-cancellation`

```bash
pip3 install textual-echo-cancellation huggingface_hub
```

### 2. Download the Model from Hugging Face

```python
from huggingface_hub import snapshot_download

model_dir = snapshot_download(repo_id="wq2012/aec_single_interfering")
print("Downloaded model to:", model_dir)
```

### 3. Run Inference via CLI (`scripts/inference.py`)

```bash
python3 -m scripts.inference \
  --model AecSingleInterfering \
  --checkpoint_path "${MODEL_DIR}/best.ckpt" \
  --mixed_wav /path/to/mixed_input.wav \
  --interfering_wav /path/to/reference_tts.wav \
  --output_wav /tmp/enhanced_clean.wav
```

### 4. Run Inference via Python API

```python
import os
from huggingface_hub import snapshot_download
from tec import inference

model_dir = snapshot_download(repo_id="wq2012/aec_single_interfering")
ckpt_path = os.path.join(model_dir, "best.ckpt")

result = inference.run_inference_on_wav(
    model_name="AecSingleInterfering",
    mixed_wav_path="/path/to/mixed_input.wav",
    interfering_wav_path="/path/to/reference_tts.wav",
    checkpoint_path=ckpt_path,
    output_wav_path="/tmp/enhanced_clean.wav",
)
print("Enhanced log-Mel spectrogram shape:", result["predicted_mel"].shape)
```

### 5. On-Device Inference with Quantized TFLite (`model.tflite`)

Note: Because the multi-source attention decoder uses dynamic `tf.while_loop` / `TensorList` operations, `model.tflite` is exported with `tf.lite.OpsSet.SELECT_TF_OPS` (Flex delegate) enabled.

```python
import os
import numpy as np
from huggingface_hub import snapshot_download
from tec import inference

model_dir = snapshot_download(repo_id="wq2012/aec_single_interfering")
tflite_path = os.path.join(model_dir, "model.tflite")

dummy_mel = np.zeros((100, 128), dtype=np.float32)
tflite_out = inference.run_tflite_inference(
    tflite_path=tflite_path,
    mixed_mel=dummy_mel,
    interfering_mel=dummy_mel,
)
print("TFLite predicted log-Mel shape:", tflite_out["predicted_mel"].shape)
```

Expected console output:

```text
TFLite predicted log-Mel shape: (100, 128)
```

---

## Citation

If you use this model or the `textual-echo-cancellation` library in your research, please cite the original paper:

```bibtex
@inproceedings{ding2021textual,
  title={Textual Echo Cancellation},
  author={Ding, Shaojin and Jia, Ye and Hu, Ke and Wang, Quan},
  booktitle={2021 IEEE Spoken Language Technology Workshop (SLT)},
  pages={666--673},
  year={2021},
  organization={IEEE}
}
```