Instructions to use ArlingtonCL2/canine-lexical-models with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Fairseq
How to use ArlingtonCL2/canine-lexical-models with Fairseq:
from fairseq.checkpoint_utils import load_model_ensemble_and_task_from_hf_hub models, cfg, task = load_model_ensemble_and_task_from_hf_hub( "ArlingtonCL2/canine-lexical-models" ) - Notebooks
- Google Colab
- Kaggle
canineLexical models
Model weights for Phonetic and Lexical Discovery of Canine Vocalization (Findings of EMNLP 2024). Code: https://github.com/UTA-ACL2/canineLexical · Paper: https://aclanthology.org/2024.findings-emnlp.816/
| File | Size | What it is | Used in |
|---|---|---|---|
dog_hubert.pt |
379 MB | Dog-HuBERT (HuBERT Base, fairseq), stage 2 after 60k updates | Step 4, phone recognition (§3.1) |
kmeans_layer11_k140.pkl |
0.4 MB | scikit-learn MiniBatchKMeans with 140 clusters on layer-11 Dog-HuBERT features. Cluster IDs are the "phones". |
Step 4 |
panns_dog_sed_finetuned.pth |
323 MB | PANNs fine-tuned for frame-level dog-sound detection on 1,483 labelled clips (checkpoint_51_2910) |
Step 2, sentence extraction (§2.2) |
tsn_dog_activity.pth |
95 MB | TSN-R50 (mmaction2) fine-tuned on 2,534 labelled clips; 16 activity classes | Step 6, activity recognition (§3.3) |
Dog-HuBERT
- Initialised from
hubert_base_ls960. - Stage 1: 80k updates, with k-means targets (100 clusters).
- Stage 2: 60k updates, with 150 k-means targets on layer 6 of stage 1.
- lr 1e-4, label rate 50 Hz.
- Trained on 34,056 dog "sentences" (more than 20 hours) cut from YouTube and AudioSet videos.
This checkpoint is the paper's checkpoint_145_60000.pt with the optimizer state removed. The weights are unchanged: it gives identical features to the original. Data paths in the saved config are replaced with /path/to/... placeholders.
import joblib, torch
from fairseq import checkpoint_utils
(model,), cfg, task = checkpoint_utils.load_model_ensemble_and_task(["dog_hubert.pt"])
model.eval()
km = joblib.load("kmeans_layer11_k140.pkl")
wav = torch.zeros(1, 16000) # 16 kHz mono waveform, not normalised
with torch.no_grad():
feats, _ = model.extract_features(source=wav, padding_mask=None, mask=False, output_layer=11)
phones = km.predict(feats[0].numpy()) # one phone ID per 20 ms frame
Of the 140 clusters, 29 are noise labels rather than dog phones: 4, 10, 20, 23, 24, 32, 37, 39, 50, 52, 53, 54, 65, 71, 72, 78, 79, 82, 85, 87, 89, 108, 109, 114, 119, 120, 122, 134, 137.
The k-means model was saved with scikit-learn 1.3. Newer versions load it with a version warning.
PANNs and TSN
panns_dog_sed_finetuned.pthloads with2_sentence_extraction/second_cut/second_cut.pyin the code repo. That script needs a clone of audioset_tagging_cnn. The paper uses a detection threshold of 0.28.tsn_dog_activity.pthloads with mmaction2 and the config at6_semantics/tsn/configs/tsn_shiba_0525.py. The class names are in6_semantics/tsn/shiba_activity_categories_en.txt.
Citation
@inproceedings{wang-etal-2024-phonetic,
title = "Phonetic and Lexical Discovery of Canine Vocalization",
author = "Wang, Theron S. and Li, Xingyuan and Zhang, Chunhao and Wu, Mengyue and Zhu, Kenny Q.",
booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2024",
year = "2024",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2024.findings-emnlp.816",
pages = "13972--13983",
}