--- license: cc-by-4.0 library_name: coreml pipeline_tag: audio-classification tags: - audio-classification - sound-event-detection - audioset - panns - coreml --- # PANNs CNN14 (16 kHz), Core ML conversion A Core ML conversion of the **PANNs CNN14 16 kHz** AudioSet tagging model, used by VERITA as an optional second sound-event detector next to Apple's built-in classifier. The app downloads this folder file by file and checks every file's SHA-256 before using it. ## Source and attribution - **Model and weights:** Qiuqiang Kong, Yin Cao, Turab Iqbal, Yuxuan Wang, Wenwu Wang, Mark D. Plumbley, "PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition", IEEE/ACM TASLP 2020 ([arXiv:1912.10211](https://arxiv.org/abs/1912.10211)). - **Weights file used:** `Cnn14_16k_mAP=0.438.pth` from Zenodo record [3987831](https://zenodo.org/records/3987831), licensed [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/); SHA-256 `e2ee543a27919542c2ea03eabaa70b24dcd4e6c8e05621de6b67a94e4c5058e6` (358,668,570 bytes). - **Model code:** [qiuqiangkong/audioset_tagging_cnn](https://github.com/qiuqiangkong/audioset_tagging_cnn) (MIT). - **Class names (`labels.txt`):** the 527 AudioSet classes in the model's output order, from `class_labels_indices.csv` of that repository (AudioSet, Google, CC BY 4.0). This conversion is licensed **CC BY 4.0**, like the original weights. If you use it, credit the PANNs authors as above. ## Changes made to the original The weights are the original ones; only their format and precision changed: 1. The PyTorch model (`Cnn14`, sample rate 16000, window 512, hop 160, 64 mel bins, 50–8000 Hz, 527 classes) was put in inference mode and wrapped to return only `clipwise_output`. 2. It was traced and converted with coremltools to an ML Program (`Cnn14_16k.mlpackage`, minimum macOS 13), with **float16** weights and compute precision. Batch normalisation was folded into the convolutions by the converter. 3. Check after conversion: on the same input windows, Core ML scores differ from PyTorch scores by at most 0.0006. Nothing was retrained or fine-tuned. ## Interface - Input `waveform`: float32 `[batch, 48000]` (batch 1 to 64), mono audio at 16 kHz, 3 s per row, in [-1, 1]. - Output `scores`: `[batch, 527]`, one sigmoid score per AudioSet class (order of `labels.txt`). VERITA runs it on 3 s windows every 1.5 s and maps a reviewed subset of AudioSet classes to its own sound classes; a score of 0.5 or more makes a detection, which a person then decides on. ## Files | File | Bytes | SHA-256 | |---|---|---| | `Cnn14_16k.mlpackage/Manifest.json` | 617 | `f1b114c52d5ed2c12cbf9b2bc554dd04dcf14c54b431bb728d6613e6eeb8bb6e` | | `Cnn14_16k.mlpackage/Data/com.apple.CoreML/model.mlmodel` | 33,882 | `8bd1a28ec9c0a12b4da928b99bacd83699e1e3cc0951e18258bfcd3d2e7007a1` | | `Cnn14_16k.mlpackage/Data/com.apple.CoreML/weights/weight.bin` | 162,052,894 | `38cdbed0c4696e576eb01aeae276f7918ccdbb6ab9755703867fe46fb43a931d` | | `labels.txt` | 6,337 | `83e2e4fd16cbe86f94f3fe23954a4d302c013e20e7981b5ce7466e50126a1e3d` | Folder hash used by VERITA's model registry (SHA-256 of the sorted lines `\t\n` over the four files above): `1243a933f9dac6931e24256daa474d444d26a1f319f4385bbc29c46a8b945bc0`. ## Limitations Measured in VERITA's own benchmark (44 CC0 sound effects mixed into generated speech), at a 0.5 score: 23 % of the effects caught on its own, and no effect caught that Apple's classifier missed; shouting was not recognised. AudioSet tags describe sounds, not facts: every detection is a suggestion for a person to check.