|
Download README.md from laion/audio-taxonomy-classifier: direct link, hf CLI and curl.
- Browser
- Download file 7.63 kB
-
https://huggingface.co/laion/audio-taxonomy-classifier/resolve/main/README.md
- Command line
-
hf download hf://laion/audio-taxonomy-classifier/README.md
-
curl -L -o README.md https://huggingface.co/laion/audio-taxonomy-classifier/resolve/main/README.md
7.63 kB
| license: apache-2.0 | |
| library_name: pytorch | |
| pipeline_tag: audio-classification | |
| base_model: mispeech/dasheng-0.6B | |
| tags: | |
| - audio | |
| - audio-classification | |
| - audio-tagging | |
| - sound-taxonomy | |
| - data-curation | |
| - dasheng | |
| # Audio taxonomy classifier | |
| Tags an audio clip with a class from a hierarchical sound taxonomy of 4,416 nodes: 11 top-level groups, depth up to 5, | |
| 4,085 leaves. The model predicts 3,718 classes, the taxonomy nodes at depth 2 and 3 (deeper nodes are merged into their | |
| depth-3 ancestor), and separately one of the 11 top-level groups: | |
| Speech · Music · Human non-speech · Animal / bioacoustics · Nature / environmental · Domestic / household · | |
| Urban / transport · Industrial / mechanical · Interface / electronic · Sound effects / designed sounds · | |
| Sports / recreation | |
| The AudioSet ontology has 632 classes. This taxonomy is about seven times larger and much finer: for example, | |
| `Speech/Noisy in-the-wild speech/Speech interrupted by electronic buzzer`, or 100 classes under | |
| `Music/Rock / metal-based music`. | |
| **Intended use:** cheap labelling of large audio collections for filtering, curation, balancing and search. Examples: | |
| measure how a collection splits over the taxonomy, cap over-represented groups, or find rare sound types. The | |
| top-level prediction is the reliable signal. The fine classes are best used as a ranked list (top-5). | |
| ## Model | |
| - **Encoder:** [Dasheng-0.6B](https://github.com/XiaoMi/dasheng), frozen. The first 10 s of a clip are resampled to | |
| 16 kHz mono and zero-padded to 10 s. The frame features are mean-pooled to one 1280-d vector. | |
| - **Head** (6.0M parameters): Linear 1280→1024, BatchNorm, GELU, dropout 0.2, then three linear outputs: 3,718 | |
| classes, 822 second-level nodes and 11 top-level groups. | |
| - **Readout:** a class scores log P(top-level) + log P(second level) + log P(class), using its own ancestors, followed | |
| by a softmax over the 3,718 classes. The top-level group comes from the 11-way output. | |
| ## Files | |
| | file | contents | | |
| |---|---| | |
| | `head.safetensors` | head weights; the metadata holds the config and the ordered class list | | |
| | `taxonomy.yaml` | the full taxonomy (4,416 nodes, each with a description) | | |
| | `inference.py` | loads Dasheng-0.6B and the head and classifies files | | |
| ## Usage | |
| ```bash | |
| pip install torch dasheng soundfile scipy safetensors | |
| python inference.py clip.flac | |
| ``` | |
| ```python | |
| import torch | |
| from inference import TaxonomyClassifier, load_audio | |
| model = TaxonomyClassifier("head.safetensors") # Dasheng weights download on first use | |
| top, cls = model(torch.from_numpy(load_audio("clip.flac"))[None]) | |
| print(model.top_names[top.argmax()], model.classes[cls.argmax()]) | |
| ``` | |
| Example output for a 2-s German sentence followed by an electronic buzzer: | |
| ``` | |
| top-level: Speech (1.000) | |
| 0.665 Speech/Noisy in-the-wild speech/Speech interrupted by electronic buzzer | |
| 0.304 Speech/Instructional - tutorial speech | |
| 0.012 Speech/Noisy in-the-wild speech/Speech preceding electronic test tone | |
| ``` | |
| Class names are taxonomy paths joined by `/`. A `/` inside a node name is written as ` - `. | |
| ## Training | |
| The training clips come from public audio datasets and from web audio. | |
| 1. **Caption-distilled labels.** A large language model read the text caption of each of 232,728 clips and assigned | |
| the clip to a taxonomy node. The head was trained on the frozen Dasheng features with class-balanced | |
| cross-entropy (weights 1/√frequency) plus auxiliary top-level and second-level losses. The train, validation and | |
| test splits were made by source archive. | |
| 2. **Audio-based relabelling (this release).** Caption labels are noisy for non-speech sounds, so 80,159 more clips | |
| were sampled, weighted towards the eight groups other than Speech, Music and Animal, and towards clips where the | |
| first head was unsure. Three audio models labelled each clip independently: two contrastive audio-text models, each ranking the | |
| class names as text prompts, and Qwen2.5-Omni-7B, instructed to pick a top-level group. For those eight groups a | |
| clip was kept only if all three models agreed on its top-level group (14,981 clips). Its class label is then the | |
| top class of the more confident contrastive model. In Speech, Music and Animal, the first head's own label was | |
| kept when its top-level probability was ≥ 0.6 (17,639 clips). The head was then fine-tuned from stage 1 on | |
| these 32,620 clips for 60 epochs, using the same loss, AdamW at lr 1e-3 and a cosine schedule. | |
| The benchmark datasets used for evaluation below were excluded from the relabelling sample, by source dataset. They | |
| were also excluded from the stage-1 data, except VocalSound and TAU Urban Acoustic Scenes 2022. Those two were not | |
| filtered there, but any overlap is small: stage 1 sampled 232,728 clips from a much larger pool. Copies of benchmark | |
| recordings that reached the pool through other sources (re-uploads) were not screened out. | |
| ## Evaluation | |
| Top-level accuracy on public benchmark test sets. Each benchmark class has a fixed set of acceptable top-level | |
| groups, chosen before scoring (1–3 groups; ESC-50 "crackling fire", for example, accepts Nature, Domestic or Sound | |
| effects). A prediction counts as correct if its group is in that set. For FSD50K (multi-label), the set is the union | |
| over the clip's labels. "Leaf-derived" reads the top-level group off the best class instead of the 11-way output. | |
| | benchmark | clips | top-level acc. (95 % CI) | leaf-derived | | |
| |---|--:|--:|--:| | |
| | CommonVoice (test) | 2,500 | 99.9 (99.6–100.0) | 99.9 | | |
| | CREMA-D | 1,500 | 100.0 (99.7–100.0) | 100.0 | | |
| | GTZAN | 999 | 99.9 (99.4–100.0) | 99.8 | | |
| | BirdCLEF 2025 | 1,500 | 96.0 (94.9–96.9) | 96.2 | | |
| | NSynth (test) | 2,000 | 76.3 (74.4–78.2) | 78.4 | | |
| | ESC-50 | 2,000 | 76.2 (74.3–78.0) | 78.7 | | |
| | FSD50K (eval) | 2,000 | 75.0 (73.1–76.8) | 76.5 | | |
| | VocalSound | 2,500 | 73.6 (71.8–75.3) | 75.3 | | |
| | UrbanSound8K | 835 | 73.2 (70.1–76.1) | 72.7 | | |
| | **macro, 9 benchmarks** | 15,834 | **85.6** | **86.4** | | |
| | TAU Urban Acoustic Scenes 2022 (scenes, not events) | 1,006 | 63.8 | 56.4 | | |
| - Calibration: the expected calibration error of the top-level probability is 0.066 over the 9 benchmarks. | |
| - For comparison, Qwen2.5-Omni-7B, instructed to pick one of the 11 groups, reaches 87.6 % macro on the same clips. It | |
| costs far more per clip. | |
| - Class level: the predicted class matches the benchmark label (by a name pattern per benchmark class) for ESC-50 | |
| 47.0 % (top-5: 66.8 %), GTZAN 69.4 % (86.1 %), VocalSound 52.0 % (75.7 %), UrbanSound8K 29.5 % (41.9 %) and NSynth | |
| 24.9 % (53.2 %). | |
| - On a random 3 % of its own relabelled training set, held out, the head agrees with the consensus labels on 96.0 % of | |
| top-level groups and 48.6 % of classes. | |
| ## Limitations | |
| - Fine classes are noisy. No large ground-truth set exists at that level, so use them as a ranked list. | |
| - Only the first 10 s of a clip are used, and a clip gets a single path. Mixed content (speech over music, film audio) | |
| gets one label. | |
| - Top-level accuracy is lowest on everyday sound events, non-speech vocal sounds and isolated instrument notes (ESC-50, | |
| FSD50K, UrbanSound8K, VocalSound, NSynth: 73–76 %). On acoustic scenes it is lower still (TAU 63.8 %). | |
| - The taxonomy was grown from captions of web audio, so the depth of its branches follows what is common on the web, | |
| not a designed balance. | |
| ## Licence and attribution | |
| Apache-2.0. The encoder is Dasheng-0.6B (Apache-2.0). The `Human non-speech` vocal-burst classes in `taxonomy.yaml` | |
| were merged from [LAION-AI/voice-taxonomies](https://github.com/LAION-AI/voice-taxonomies) (CC-BY-4.0). | |