| --- |
| license: mit |
| tags: |
| - active-speaker-detection |
| - audio-visual |
| - speaker-diarization |
| - streaming |
| - pytorch |
| - pytorch-lightning |
| --- |
| |
| # CAST-ASD |
|
|
| Pretrained weights for **CAST-ASD: Causal Attention over Speakers and Time for Online Active Speaker Detection**. |
|
|
| Pratham Gupta, Haotian Qi, Tushar Sandhan, Gabriel Skantze |
|
|
| [Paper (arXiv:2610.14676)](https://arxiv.org/abs/2610.14676) · [Code](https://github.com/prathamg007/CAST-ASD) · [Project page and interactive demo](https://prathamg007.github.io/CAST-ASD/) |
|
|
| CAST-ASD decides, for every visible face in every video frame, whether that person is speaking, without reading any future frame. It adds one CAST block to a causal per-face backbone: a speaker pass attends across the faces present at the current frame, and a time pass attends causally over each face's own past. The block has 1.57M parameters, independent of the number of faces. On MSDWild it removes 32% of the competing-speaker false alarms of the causal per-face backbone it extends. |
|
|
| ## Files |
|
|
| All four checkpoints are CAST ×1 (one block). Each needs its config, included here and in the code repository. |
|
|
| | File | Config | Seed | Validation mAP | Paper (mean of 3 seeds) | |
| |---|---|:---:|---|---| |
| | `weights/msdwild_cast.ckpt` | `configs/msdwild/cast.yaml` | 42 | few.val 97.01, many.val 90.66 | few.val 96.99, many.val 90.56 | |
| | `weights/msdwild_cast_offline.ckpt` | `configs/msdwild/cast_offline.yaml` | 1 | few.val 97.55, many.val 91.77 | few.val 97.51, many.val 91.70 | |
| | `weights/ava_cast.ckpt` | `configs/ava/cast.yaml` | 2 | official 91.21 | official 91.07 | |
| | `weights/ava_cast_offline.ckpt` | `configs/ava/cast_offline.yaml` | 1 | official 94.31 | official 94.20 | |
|
|
| - `msdwild_*` are trained on MSDWild (few-speaker training split); `ava_*` on AVA-ActiveSpeaker. |
| - `*_offline` checkpoints use the same architecture without the causal constraints and read the whole clip. |
| - Each model was trained with seeds 42, 1 and 2; the released file is the best of the three on the selection metric (few.val mAP on MSDWild, official mAP on AVA). The paper reports the mean over the three seeds. |
| - MSDWild checkpoints are from the final (12th) epoch; AVA checkpoints are the best epoch under early stopping. |
| - The frozen Mimi audio encoder is not stored in these files. It is downloaded from `kyutai/moshiko-pytorch-bf16` and merged in when a checkpoint loads. Optimizer state is stripped, so the files evaluate but cannot resume training. |
|
|
| ## Usage |
|
|
| The checkpoints are PyTorch Lightning files for the code in the [GitHub repository](https://github.com/prathamg007/CAST-ASD). |
|
|
| ```bash |
| git clone https://github.com/prathamg007/CAST-ASD.git |
| cd CAST-ASD |
| pip install -e ".[mimi]" |
| hf download prathamg007/CAST-ASD --include "weights/*" --local-dir . |
| python train.py --config configs/msdwild/cast.yaml --test --checkpoint weights/msdwild_cast.ckpt |
| ``` |
|
|
| Or fetch a single file from Python: |
|
|
| ```python |
| from huggingface_hub import hf_hub_download |
| |
| path = hf_hub_download("prathamg007/CAST-ASD", "weights/msdwild_cast.ckpt") |
| ``` |
|
|
| Data preparation for MSDWild and AVA-ActiveSpeaker is described in the code repository. |
|
|
| ## License |
|
|
| MIT, the same as the code. The models were trained on AVA-ActiveSpeaker and MSDWild, so use them in line with those datasets' terms (MSDWild is distributed under its own license agreement). |
|
|
| ## Citation |
|
|
| ```bibtex |
| @misc{gupta2026castasd, |
| title = {{CAST-ASD}: Causal Attention over Speakers and Time for Online Active Speaker Detection}, |
| author = {Gupta, Pratham and Qi, Haotian and Sandhan, Tushar and Skantze, Gabriel}, |
| year = {2026}, |
| eprint = {2610.14676}, |
| archivePrefix = {arXiv}, |
| primaryClass = {cs.CV}, |
| doi = {10.48550/arXiv.2610.14676}, |
| url = {https://arxiv.org/abs/2610.14676} |
| } |
| ``` |
|
|
