| language: | |
| - zh | |
| tags: | |
| - audio-visual-speech-separation | |
| - target-speech-extraction | |
| - streaming | |
| - pytorch | |
| # SAV-BandNet | |
| A six-channel, visually conditioned target-speech extractor using 1 s windows, 0.4 s emissions and 0.2 s lookahead. The image frontend is TalkNet; spatial features come from visually conditioned IPDNet2. Candidate streams come from an adapted DAVE separator. Target selection combines audiovisual scores, recurrent history and speaker identity memory. | |
| This release contains the earlier TalkNet extraction checkpoint trained on 4,096 scenes. It is not the later 32,768-scene paper model, and historical CMLR or full-training-set metrics should not be attributed to it. The original head hash is `55505a92fcb523457790ff0d9f1aedb1aab4986eaf57f0d85c6e3b357fa09b59`. Component hashes and lineage are in `PROVENANCE.json`. Optimizer states and historical parent checkpoints are omitted. | |
| ## Use | |
| Download the bundle and `implementation.tar.gz`. Extract the implementation, install its dependencies following `docs/setup.md`, and set `SAV_BUNDLE` to the downloaded bundle directory (or pass `--bundle`). The dataset is [SAVE](https://huggingface.co/datasets/playwithmino/SAVE). | |
| ```bash | |
| python -m sav_bandnet infer --data /path/to/SAVE --bundle /path/to/bundle --output /path/to/results | |
| ``` | |
| Inputs are synchronized 16 kHz six-channel audio and 25 fps 96×96 target-mouth crops with explicit validity. The reference geometry is a six-microphone circular array of radius 3.5 cm. Different arrays, microphone ordering or audiovisual synchronization require validation. The code emits CPU-available waveform chunks; this is not a measured physical capture-to-playback latency guarantee. | |
| `RELEASE_VERIFICATION.json` records parity and bounded training smoke checks. These checks do not establish convergence of the consolidated training command or real-world performance. Candidate alignment uses overlapping waveform correlation; speaker embeddings correct selection scores. | |
| ## Sources and terms | |
| The implementation derives from DAVE/TIGER, TalkNet and IPDNet2. Retained source notices accompany the implementation. Component code licenses do not automatically license upstream pretrained weights or source media. No blanket commercial-use or redistribution-rights assertion is made for all components. Consult the upstream sources and `THIRD_PARTY_NOTICES.md`. | |