---
tags:
  - audio-visual
  - speech-enhancement
  - av-hubert
  - fairseq
  - speech-tokens
library_name: fairseq
pipeline_tag: audio-to-audio
---

# AV-STE: Audio-Visual Speech Token Enhancement

**EMNLP 2026** — *Noise Adaptive Streaming Audio-Visual Speech Token Enhancement for Robust Full-Duplex Spoken Dialogue Models*

Code, demo, and full documentation: [github.com/bellagodiva/av-ste](https://github.com/bellagodiva/av-ste)

AV-STE recovers clean Mimi semantic tokens from noisy speech by fusing audio features with lip-ROI video through an entropy-gated cross-attention module built on top of AV-HuBERT. Enhanced tokens can be fed directly into any Mimi-based TTS or codec LM (e.g. Moshi) in place of the noisy ones.

## Files

| Checkpoint | Use for |
|---|---|
| `avste.pt` | Clean speech, non-speech background noise, cross-dataset speaker interference |
| `avste_lrs3_interference.pt` | Same-corpus-style competing speakers, out-of-domain video |

Together they reproduce every semantic-token-accuracy row in the paper's main table. `large_vox_iter5.pt` (the public AV-HuBERT-Large VoxCeleb2 backbone both were fine-tuned from) is not hosted here — the GitHub repo's download script fetches it directly from its original source.

Full checkpoint-selection guidance: [checkpoints/README.md](https://github.com/bellagodiva/av-ste/blob/main/checkpoints/README.md) in the code repo.

## Usage

```python
from huggingface_hub import hf_hub_download

path = hf_hub_download(repo_id="bgdv99/av-ste", filename="avste.pt")
# or filename="avste_lrs3_interference.pt"
```

Both are fairseq model ensemble files:

```python
from fairseq import checkpoint_utils

models, cfg, task = checkpoint_utils.load_model_ensemble_and_task(
    ["avste.pt"],
    arg_overrides={"w2v_path": "large_vox_iter5.pt"},
)
```

See the [GitHub repo](https://github.com/bellagodiva/av-ste) for the full inference pipeline (`scripts/infer_avste.py`), a runnable demo, and training instructions.

## Data

These weights were fine-tuned on LRS3 (license-gated) with AudioSet noise/interference augmentation. No training or evaluation data is redistributed here — only the resulting model parameters.

## Citation

```bibtex
@inproceedings{avste2026,
  title     = {Noise Adaptive Streaming Audio-Visual Speech Token Enhancement for Robust Full-Duplex Spoken Dialogue Models},
  author    = {TODO},
  booktitle = {Proceedings of EMNLP},
  year      = {2026},
}
```