| --- |
| license: mit |
| tags: |
| - audio-visual |
| - dog |
| - sound-event-detection |
| - speaker-attribution |
| - pytorch |
| --- |
| |
| # BarkVTrace model weights |
|
|
| Weights for [BarkVTrace](https://github.com/DuyPham72/BarkTrace-AudioVisualAI-PyTorch), the pipeline of the master's thesis *BarkVTrace: |
| Audio-Visual Attribution of Dog Vocalizations in Real-World Videos*. For every dog sound in a video it answers which |
| dog made it, or "outside" when the vocalizing dog is off screen. |
|
|
| The repository's `download_models.sh` fetches these files together with the public models into `models/` and |
| `third_party/beats_sed/`; there is no need to download them by hand. |
|
|
| | Path | What | Used by | |
| |---|---|---| |
| | `synchformer/dog_ft_20261026/seed{0,1,2}.pt` | Dog fine-tuned Synchformer heads ("this crop made the sound"), 3 seeds | Barker attribution | |
| | `lab_beats_ft/final_vocal_noise.pt` | Noise-robust vocalization BEATs head | Dog sound detection | |
| | `lab_beats_ft/final_dog.pt` | All-dog-sound BEATs head | Dog sound detection | |
| | `dog_sound_types/classifier.joblib` | Sound-type classifier (bark, growl, grunt, howl, moan, pant, whine) | Sound types | |
| | `beats_sed/` | The lab's BEATs sound-event-detection model: `beats_work/desed_task` code, `hparams.yaml`, `epoch=109-step=6160.ckpt`, and Microsoft's `BEATS_iter3_plus_AS2M.pt` | Base of the two BEATs heads | |
|
|
| ## Results (from the thesis) |
|
|
| | Stage | Test | Result | |
| |---|---|---| |
| | Detection | 45 videos (110 labeled dog sounds, 31 videos without any) | 86.0% precision, 87.9% recall | |
| | Barker | 297 labeled sounds, held out by YouTube channel | 75.1% right, 2.7% wrong dog | |
| | End to end | 106 sounds in 14 videos | 71.7% right, 0% wrong dog | |
|
|
| ## License and credits |
|
|
| MIT. `beats_sed/` is published with the permission of the lab that trained it. `BEATS_iter3_plus_AS2M.pt` is |
| [BEATs](https://github.com/microsoft/unilm/tree/master/beats) by Microsoft (MIT); the `desed_task` code builds on the |
| [DCASE task 4 baseline](https://github.com/DCASE-REPO/DESED_task) (MIT). Synchformer (MIT) is by Iashin et al.; its |
| base checkpoint is downloaded from the original release. |
|
|
| Note: `classifier.joblib` and the `.pt` / `.ckpt` files are pickles; load them only from this repository. |
|
|
