| --- |
| tags: |
| - audio-visual |
| - speech-enhancement |
| - av-hubert |
| - fairseq |
| - speech-tokens |
| library_name: fairseq |
| pipeline_tag: audio-to-audio |
| --- |
| |
| # AV-STE: Audio-Visual Speech Token Enhancement |
|
|
| **EMNLP 2026** — *Noise Adaptive Streaming Audio-Visual Speech Token Enhancement for Robust Full-Duplex Spoken Dialogue Models* |
|
|
| Code, demo, and full documentation: [github.com/bellagodiva/av-ste](https://github.com/bellagodiva/av-ste) |
|
|
| AV-STE recovers clean Mimi semantic tokens from noisy speech by fusing audio features with lip-ROI video through an entropy-gated cross-attention module built on top of AV-HuBERT. Enhanced tokens can be fed directly into any Mimi-based TTS or codec LM (e.g. Moshi) in place of the noisy ones. |
|
|
| ## Files |
|
|
| | Checkpoint | Use for | |
| |---|---| |
| | `avste.pt` | Clean speech, non-speech background noise, cross-dataset speaker interference | |
| | `avste_lrs3_interference.pt` | Same-corpus-style competing speakers, out-of-domain video | |
|
|
| Together they reproduce every semantic-token-accuracy row in the paper's main table. `large_vox_iter5.pt` (the public AV-HuBERT-Large VoxCeleb2 backbone both were fine-tuned from) is not hosted here — the GitHub repo's download script fetches it directly from its original source. |
|
|
| Full checkpoint-selection guidance: [checkpoints/README.md](https://github.com/bellagodiva/av-ste/blob/main/checkpoints/README.md) in the code repo. |
|
|
| ## Usage |
|
|
| ```python |
| from huggingface_hub import hf_hub_download |
| |
| path = hf_hub_download(repo_id="bgdv99/av-ste", filename="avste.pt") |
| # or filename="avste_lrs3_interference.pt" |
| ``` |
|
|
| Both are fairseq model ensemble files: |
|
|
| ```python |
| from fairseq import checkpoint_utils |
| |
| models, cfg, task = checkpoint_utils.load_model_ensemble_and_task( |
| ["avste.pt"], |
| arg_overrides={"w2v_path": "large_vox_iter5.pt"}, |
| ) |
| ``` |
|
|
| See the [GitHub repo](https://github.com/bellagodiva/av-ste) for the full inference pipeline (`scripts/infer_avste.py`), a runnable demo, and training instructions. |
|
|
| ## Data |
|
|
| These weights were fine-tuned on LRS3 (license-gated) with AudioSet noise/interference augmentation. No training or evaluation data is redistributed here — only the resulting model parameters. |
|
|
| ## Citation |
|
|
| ```bibtex |
| @inproceedings{avste2026, |
| title = {Noise Adaptive Streaming Audio-Visual Speech Token Enhancement for Robust Full-Duplex Spoken Dialogue Models}, |
| author = {TODO}, |
| booktitle = {Proceedings of EMNLP}, |
| year = {2026}, |
| } |
| ``` |
|
|
