CAST-ASD

Pretrained weights for CAST-ASD: Causal Attention over Speakers and Time for Online Active Speaker Detection.

Pratham Gupta, Haotian Qi, Tushar Sandhan, Gabriel Skantze

Paper (arXiv:2610.14676) · Code · Project page and interactive demo

CAST-ASD decides, for every visible face in every video frame, whether that person is speaking, without reading any future frame. It adds one CAST block to a causal per-face backbone: a speaker pass attends across the faces present at the current frame, and a time pass attends causally over each face's own past. The block has 1.57M parameters, independent of the number of faces. On MSDWild it removes 32% of the competing-speaker false alarms of the causal per-face backbone it extends.

Files

All four checkpoints are CAST ×1 (one block). Each needs its config, included here and in the code repository.

File	Config	Seed	Validation mAP	Paper (mean of 3 seeds)
weights/msdwild_cast.ckpt	configs/msdwild/cast.yaml	42	few.val 97.01, many.val 90.66	few.val 96.99, many.val 90.56
weights/msdwild_cast_offline.ckpt	configs/msdwild/cast_offline.yaml	1	few.val 97.55, many.val 91.77	few.val 97.51, many.val 91.70
weights/ava_cast.ckpt	configs/ava/cast.yaml	2	official 91.21	official 91.07
weights/ava_cast_offline.ckpt	configs/ava/cast_offline.yaml	1	official 94.31	official 94.20

• msdwild_* are trained on MSDWild (few-speaker training split); ava_* on AVA-ActiveSpeaker.
• *_offline checkpoints use the same architecture without the causal constraints and read the whole clip.
• Each model was trained with seeds 42, 1 and 2; the released file is the best of the three on the selection metric (few.val mAP on MSDWild, official mAP on AVA). The paper reports the mean over the three seeds.
• MSDWild checkpoints are from the final (12th) epoch; AVA checkpoints are the best epoch under early stopping.
• The frozen Mimi audio encoder is not stored in these files. It is downloaded from kyutai/moshiko-pytorch-bf16 and merged in when a checkpoint loads. Optimizer state is stripped, so the files evaluate but cannot resume training.

Usage

The checkpoints are PyTorch Lightning files for the code in the GitHub repository.

git clone https://github.com/prathamg007/CAST-ASD.git
cd CAST-ASD
pip install -e ".[mimi]"
hf download prathamg007/CAST-ASD --include "weights/*" --local-dir .
python train.py --config configs/msdwild/cast.yaml --test --checkpoint weights/msdwild_cast.ckpt

Or fetch a single file from Python:

from huggingface_hub import hf_hub_download

path = hf_hub_download("prathamg007/CAST-ASD", "weights/msdwild_cast.ckpt")

Data preparation for MSDWild and AVA-ActiveSpeaker is described in the code repository.

License

MIT, the same as the code. The models were trained on AVA-ActiveSpeaker and MSDWild, so use them in line with those datasets' terms (MSDWild is distributed under its own license agreement).

Citation

@misc{gupta2026castasd,
  title         = {{CAST-ASD}: Causal Attention over Speakers and Time for Online Active Speaker Detection},
  author        = {Gupta, Pratham and Qi, Haotian and Sandhan, Tushar and Skantze, Gabriel},
  year          = {2026},
  eprint        = {2610.14676},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  doi           = {10.48550/arXiv.2610.14676},
  url           = {https://arxiv.org/abs/2610.14676}
}