---
license: mit
tags:
- audio-visual
- dog
- sound-event-detection
- speaker-attribution
- pytorch
---

# BarkVTrace model weights

Weights for [BarkVTrace](https://github.com/DuyPham72/BarkTrace-AudioVisualAI-PyTorch), the pipeline of the master's thesis *BarkVTrace:
Audio-Visual Attribution of Dog Vocalizations in Real-World Videos*. For every dog sound in a video it answers which
dog made it, or "outside" when the vocalizing dog is off screen.

The repository's `download_models.sh` fetches these files together with the public models into `models/` and
`third_party/beats_sed/`; there is no need to download them by hand.

| Path | What | Used by |
|---|---|---|
| `synchformer/dog_ft_20261026/seed{0,1,2}.pt` | Dog fine-tuned Synchformer heads ("this crop made the sound"), 3 seeds | Barker attribution |
| `lab_beats_ft/final_vocal_noise.pt` | Noise-robust vocalization BEATs head | Dog sound detection |
| `lab_beats_ft/final_dog.pt` | All-dog-sound BEATs head | Dog sound detection |
| `dog_sound_types/classifier.joblib` | Sound-type classifier (bark, growl, grunt, howl, moan, pant, whine) | Sound types |
| `beats_sed/` | The lab's BEATs sound-event-detection model: `beats_work/desed_task` code, `hparams.yaml`, `epoch=109-step=6160.ckpt`, and Microsoft's `BEATS_iter3_plus_AS2M.pt` | Base of the two BEATs heads |

## Results (from the thesis)

| Stage | Test | Result |
|---|---|---|
| Detection | 45 videos (110 labeled dog sounds, 31 videos without any) | 86.0% precision, 87.9% recall |
| Barker | 297 labeled sounds, held out by YouTube channel | 75.1% right, 2.7% wrong dog |
| End to end | 106 sounds in 14 videos | 71.7% right, 0% wrong dog |

## License and credits

MIT. `beats_sed/` is published with the permission of the lab that trained it. `BEATS_iter3_plus_AS2M.pt` is
[BEATs](https://github.com/microsoft/unilm/tree/master/beats) by Microsoft (MIT); the `desed_task` code builds on the
[DCASE task 4 baseline](https://github.com/DCASE-REPO/DESED_task) (MIT). Synchformer (MIT) is by Iashin et al.; its
base checkpoint is downloaded from the original release.

Note: `classifier.joblib` and the `.pt` / `.ckpt` files are pickles; load them only from this repository.