canineLexical models

Model weights for Phonetic and Lexical Discovery of Canine Vocalization (Findings of EMNLP 2024). Code: https://github.com/UTA-ACL2/canineLexical · Paper: https://aclanthology.org/2024.findings-emnlp.816/

File Size What it is Used in
dog_hubert.pt 379 MB Dog-HuBERT (HuBERT Base, fairseq), stage 2 after 60k updates Step 4, phone recognition (§3.1)
kmeans_layer11_k140.pkl 0.4 MB scikit-learn MiniBatchKMeans with 140 clusters on layer-11 Dog-HuBERT features. Cluster IDs are the "phones". Step 4
panns_dog_sed_finetuned.pth 323 MB PANNs fine-tuned for frame-level dog-sound detection on 1,483 labelled clips (checkpoint_51_2910) Step 2, sentence extraction (§2.2)
tsn_dog_activity.pth 95 MB TSN-R50 (mmaction2) fine-tuned on 2,534 labelled clips; 16 activity classes Step 6, activity recognition (§3.3)

Dog-HuBERT

  • Initialised from hubert_base_ls960.
  • Stage 1: 80k updates, with k-means targets (100 clusters).
  • Stage 2: 60k updates, with 150 k-means targets on layer 6 of stage 1.
  • lr 1e-4, label rate 50 Hz.
  • Trained on 34,056 dog "sentences" (more than 20 hours) cut from YouTube and AudioSet videos.

This checkpoint is the paper's checkpoint_145_60000.pt with the optimizer state removed. The weights are unchanged: it gives identical features to the original. Data paths in the saved config are replaced with /path/to/... placeholders.

import joblib, torch
from fairseq import checkpoint_utils

(model,), cfg, task = checkpoint_utils.load_model_ensemble_and_task(["dog_hubert.pt"])
model.eval()
km = joblib.load("kmeans_layer11_k140.pkl")

wav = torch.zeros(1, 16000)  # 16 kHz mono waveform, not normalised
with torch.no_grad():
    feats, _ = model.extract_features(source=wav, padding_mask=None, mask=False, output_layer=11)
phones = km.predict(feats[0].numpy())  # one phone ID per 20 ms frame

Of the 140 clusters, 29 are noise labels rather than dog phones: 4, 10, 20, 23, 24, 32, 37, 39, 50, 52, 53, 54, 65, 71, 72, 78, 79, 82, 85, 87, 89, 108, 109, 114, 119, 120, 122, 134, 137.

The k-means model was saved with scikit-learn 1.3. Newer versions load it with a version warning.

PANNs and TSN

  • panns_dog_sed_finetuned.pth loads with 2_sentence_extraction/second_cut/second_cut.py in the code repo. That script needs a clone of audioset_tagging_cnn. The paper uses a detection threshold of 0.28.
  • tsn_dog_activity.pth loads with mmaction2 and the config at 6_semantics/tsn/configs/tsn_shiba_0525.py. The class names are in 6_semantics/tsn/shiba_activity_categories_en.txt.

Citation

@inproceedings{wang-etal-2024-phonetic,
    title = "Phonetic and Lexical Discovery of Canine Vocalization",
    author = "Wang, Theron S. and Li, Xingyuan and Zhang, Chunhao and Wu, Mengyue and Zhu, Kenny Q.",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2024",
    year = "2024",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2024.findings-emnlp.816",
    pages = "13972--13983",
}
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support