| --- |
| license: apache-2.0 |
| library_name: pytorch |
| pipeline_tag: audio-to-audio |
| tags: |
| - audio-visual |
| - target-speaker-extraction |
| - speech-separation |
| - streaming |
| datasets: |
| - JusperLee/RealSSA-Real |
| - JusperLee/RealSSA-Sim |
| --- |
| |
| <p align="center"><img src="https://huggingface.co/JusperLee/Falcon/resolve/main/logo.png" alt="Falcon logo" width="120"></p> |
|
|
| # Falcon |
|
|
| Causal audio-visual target speaker extraction model from the paper: a single-pass multi-scale separator trained with an encoder-space speaker contrastive loss (0.5 M trainable parameters). |
|
|
| Code, training and evaluation: <https://github.com/JusperLee/Falcon>. Paper: *Efficient Streaming Audio-Visual Target Speaker Extraction for Real-World Acoustic Scenes* (NeurIPS 2026), Wendi Sang\*, Kai Li\*, Yifan Li, Jianqiang Huang, Xiaolin Hu (\*equal contribution). |
| |
| | File | Config in the code repo | Audio network | Total params | |
| |---|---|---|---| |
| | `falcon.pt` | `configs/Falcon.yml` | `FalconSpeakerContrastive` | 11.72 M | |
| |
| Total parameters include the frozen 11.2 M-parameter FRCNN lip-reading encoder. The file holds only `state_dict` (`audio_model.*`, `video_model.*`), the architecture name and the training epoch; load it with `torch.load(..., weights_only=True)`. `config.yaml` is a copy of `configs/Falcon.yml` from the code repo; the Hub counts downloads by this file, and `FalconRuntime` requests it on every load. |
|
|
| ## Usage |
|
|
| ```bash |
| git clone https://github.com/JusperLee/Falcon.git |
| cd Falcon/studio && pip install -r requirements.txt |
| ``` |
|
|
| ```python |
| import numpy as np |
| from falcon_runtime import FalconRuntime |
| |
| model = FalconRuntime.from_pretrained("JusperLee/Falcon") # downloads falcon.pt on first use |
| audio = np.zeros(32000, dtype=np.float32) # 2 s mono 16 kHz mixture |
| mouth = np.zeros((50, 96, 96), dtype=np.uint8) # 25 fps grayscale mouth crops of the target speaker |
| target = model.infer(audio, mouth) # (32000,) target speech |
| ``` |
|
|
| Whole clips: `python infer_file.py --audio mix.wav --mouth mouth.npy --out target.wav --model JusperLee/Falcon`. Live separation in the browser and in Zoom / Tencent Meeting: see `studio/README.md` in the code repo. |
|
|
| Input: mono 16 kHz audio and synchronized 25 fps grayscale 96×96 mouth crops (center-cropped to 88×88 and normalized inside the runtime). |
|
|
| ## License |
|
|
| Apache License 2.0, the same as the code repository. |
