| --- |
| license: cc-by-4.0 |
| library_name: coreml |
| pipeline_tag: audio-classification |
| tags: |
| - audio-classification |
| - sound-event-detection |
| - audioset |
| - panns |
| - coreml |
| --- |
| |
| # PANNs CNN14 (16 kHz), Core ML conversion |
|
|
| A Core ML conversion of the **PANNs CNN14 16 kHz** AudioSet tagging model, used by VERITA as an |
| optional second sound-event detector next to Apple's built-in classifier. The app downloads this |
| folder file by file and checks every file's SHA-256 before using it. |
|
|
| ## Source and attribution |
|
|
| - **Model and weights:** Qiuqiang Kong, Yin Cao, Turab Iqbal, Yuxuan Wang, Wenwu Wang, Mark D. |
| Plumbley, "PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition", |
| IEEE/ACM TASLP 2020 ([arXiv:1912.10211](https://arxiv.org/abs/1912.10211)). |
| - **Weights file used:** `Cnn14_16k_mAP=0.438.pth` from Zenodo record |
| [3987831](https://zenodo.org/records/3987831), licensed |
| [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/); SHA-256 |
| `e2ee543a27919542c2ea03eabaa70b24dcd4e6c8e05621de6b67a94e4c5058e6` (358,668,570 bytes). |
| - **Model code:** [qiuqiangkong/audioset_tagging_cnn](https://github.com/qiuqiangkong/audioset_tagging_cnn) (MIT). |
| - **Class names (`labels.txt`):** the 527 AudioSet classes in the model's output order, from |
| `class_labels_indices.csv` of that repository (AudioSet, Google, CC BY 4.0). |
|
|
| This conversion is licensed **CC BY 4.0**, like the original weights. If you use it, credit the |
| PANNs authors as above. |
|
|
| ## Changes made to the original |
|
|
| The weights are the original ones; only their format and precision changed: |
|
|
| 1. The PyTorch model (`Cnn14`, sample rate 16000, window 512, hop 160, 64 mel bins, 50–8000 Hz, |
| 527 classes) was put in inference mode and wrapped to return only `clipwise_output`. |
| 2. It was traced and converted with coremltools to an ML Program (`Cnn14_16k.mlpackage`, minimum |
| macOS 13), with **float16** weights and compute precision. Batch normalisation was folded into |
| the convolutions by the converter. |
| 3. Check after conversion: on the same input windows, Core ML scores differ from PyTorch scores by |
| at most 0.0006. |
|
|
| Nothing was retrained or fine-tuned. |
|
|
| ## Interface |
|
|
| - Input `waveform`: float32 `[batch, 48000]` (batch 1 to 64), mono audio at 16 kHz, 3 s per row, in |
| [-1, 1]. |
| - Output `scores`: `[batch, 527]`, one sigmoid score per AudioSet class (order of `labels.txt`). |
|
|
| VERITA runs it on 3 s windows every 1.5 s and maps a reviewed subset of AudioSet classes to its |
| own sound classes; a score of 0.5 or more makes a detection, which a person then decides on. |
|
|
| ## Files |
|
|
| | File | Bytes | SHA-256 | |
| |---|---|---| |
| | `Cnn14_16k.mlpackage/Manifest.json` | 617 | `f1b114c52d5ed2c12cbf9b2bc554dd04dcf14c54b431bb728d6613e6eeb8bb6e` | |
| | `Cnn14_16k.mlpackage/Data/com.apple.CoreML/model.mlmodel` | 33,882 | `8bd1a28ec9c0a12b4da928b99bacd83699e1e3cc0951e18258bfcd3d2e7007a1` | |
| | `Cnn14_16k.mlpackage/Data/com.apple.CoreML/weights/weight.bin` | 162,052,894 | `38cdbed0c4696e576eb01aeae276f7918ccdbb6ab9755703867fe46fb43a931d` | |
| | `labels.txt` | 6,337 | `83e2e4fd16cbe86f94f3fe23954a4d302c013e20e7981b5ce7466e50126a1e3d` | |
|
|
| Folder hash used by VERITA's model registry (SHA-256 of the sorted lines |
| `<path>\t<file SHA-256>\n` over the four files above): |
| `1243a933f9dac6931e24256daa474d444d26a1f319f4385bbc29c46a8b945bc0`. |
|
|
| ## Limitations |
|
|
| Measured in VERITA's own benchmark (44 CC0 sound effects mixed into generated speech), at a |
| 0.5 score: 23 % of the effects caught on its own, and no effect caught that Apple's classifier |
| missed; shouting was not recognised. AudioSet tags describe sounds, not facts: every detection is |
| a suggestion for a person to check. |
|
|
