Model card: labels, per-type accuracy, usage, method, limitations
Browse files
README.md
CHANGED
|
@@ -1,65 +1,175 @@
|
|
| 1 |
---
|
| 2 |
license: apache-2.0
|
|
|
|
| 3 |
pipeline_tag: audio-classification
|
| 4 |
tags:
|
| 5 |
- audio-classification
|
| 6 |
- music
|
| 7 |
- samples
|
|
|
|
| 8 |
- clap
|
| 9 |
- core-ai
|
| 10 |
- apple-silicon
|
|
|
|
| 11 |
---
|
| 12 |
|
| 13 |
-
# muse sample types
|
| 14 |
|
| 15 |
-
One Core AI
|
|
|
|
| 16 |
|
| 17 |
-
|
| 18 |
-
|
| 19 |
-
|
| 20 |
-
|
| 21 |
-
- synth: lead, pluck, pad, stab, keys
|
| 22 |
-
- vocals: vocal
|
| 23 |
-
- fx: sweep, impact, atmo, fx
|
| 24 |
-
- **Does a key signature mean anything here?** `atonal` or `tonal` (`gate.txt`) β so a
|
| 25 |
-
kick drum is not labelled "C minor".
|
| 26 |
|
| 27 |
-
|
|
|
|
|
|
|
| 28 |
|
| 29 |
-
|
| 30 |
-
files whose label came from their folder name:
|
| 31 |
|
| 32 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 33 |
|---|---|---|
|
| 34 |
-
|
|
| 35 |
-
|
|
| 36 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 37 |
|
| 38 |
-
|
| 39 |
-
|
| 40 |
-
Most common confusions: lead β pluck, bass β stab, sweep / atmo β fx.
|
| 41 |
|
| 42 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 43 |
|
| 44 |
-
|
| 45 |
|
| 46 |
-
|
| 47 |
-
|
| 48 |
-
|
| 49 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 50 |
|
| 51 |
## How it was made
|
| 52 |
|
| 53 |
-
1. **Encoder
|
| 54 |
-
(LAION, Apache-2.0), weights unchanged. It
|
| 55 |
-
|
| 56 |
-
|
| 57 |
-
|
| 58 |
-
|
| 59 |
-
|
| 60 |
-
|
| 61 |
-
|
| 62 |
-
|
| 63 |
-
|
| 64 |
-
|
| 65 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
license: apache-2.0
|
| 3 |
+
base_model: laion/clap-htsat-unfused
|
| 4 |
pipeline_tag: audio-classification
|
| 5 |
tags:
|
| 6 |
- audio-classification
|
| 7 |
- music
|
| 8 |
- samples
|
| 9 |
+
- sample-library
|
| 10 |
- clap
|
| 11 |
- core-ai
|
| 12 |
- apple-silicon
|
| 13 |
+
- on-device
|
| 14 |
---
|
| 15 |
|
| 16 |
+
# muse sample types
|
| 17 |
|
| 18 |
+
**Name any audio sample on a Mac, on-device.** One Core AI model that listens to a sample
|
| 19 |
+
once and answers two questions:
|
| 20 |
|
| 21 |
+
1. **What is it?** One of 22 sample types, such as kick, snare, bass, pluck, pad, vocal or
|
| 22 |
+
sweep, with the runner-up types and their probabilities.
|
| 23 |
+
2. **Does a musical key mean anything here?** Tonal or atonal, so a kick drum is never
|
| 24 |
+
labelled "C minor".
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 25 |
|
| 26 |
+
It also returns the sound's 512-number CLAP embedding, for "find sounds like this" search.
|
| 27 |
+
It is the model behind the `class` and `tonal` columns of the
|
| 28 |
+
[`muse`](https://github.com/arraypress) command-line tool, which downloads it on first use.
|
| 29 |
|
| 30 |
+
## Labels
|
|
|
|
| 31 |
|
| 32 |
+
22 types in five families (`types.txt`). A label says what the sound *is*; whether it is a
|
| 33 |
+
one-shot or a loop is a separate question, so a sub-bass hit and a rolling bass line are
|
| 34 |
+
both `bass`.
|
| 35 |
+
|
| 36 |
+
| Family | Types |
|
| 37 |
+
|---|---|
|
| 38 |
+
| drums | `kick`, `snare`, `clap`, `hat_closed`, `hat_open`, `cymbal` (crash, ride), `tom`, `percussion` (incl. shakers), `drum_loop` (a full loop, with a kick), `perc_loop` (hats, shakers, percussion, no kick) |
|
| 39 |
+
| bass | `bass` (hits, loops, sub, reese), `acid` (303-style lines) |
|
| 40 |
+
| synth | `lead`, `pluck`, `pad`, `stab` (incl. chords and synth hits), `keys` (incl. piano) |
|
| 41 |
+
| vocals | `vocal` (hits and loops) |
|
| 42 |
+
| fx | `sweep` (risers and downlifters), `impact`, `atmo` (incl. noise), `fx` |
|
| 43 |
+
|
| 44 |
+
Gate labels (`gate.txt`): `atonal`, `tonal`.
|
| 45 |
+
|
| 46 |
+
## Accuracy
|
| 47 |
+
|
| 48 |
+
Measured with 5-fold cross-validation **split by sample pack**: every number is for packs
|
| 49 |
+
the model never saw in training, counting only files whose label came from their folder
|
| 50 |
+
name.
|
| 51 |
+
|
| 52 |
+
| | v1 (Aug 2026) | **v2** |
|
| 53 |
|---|---|---|
|
| 54 |
+
| Sample type, right first time | 68.3% (39 types) | **85.4%** (22 types) |
|
| 55 |
+
| Sample type, right in the top three | β | **97.5%** |
|
| 56 |
+
| Tonal gate | 93.3% (Create ML) | **98.6%** |
|
| 57 |
+
|
| 58 |
+
Per type, v2, right first time / in the top three:
|
| 59 |
+
|
| 60 |
+
| Type | Top-1 | Top-3 | | Type | Top-1 | Top-3 |
|
| 61 |
+
|---|---|---|---|---|---|---|
|
| 62 |
+
| kick | 97.5% | 99.7% | | lead | 76.8% | 97.6% |
|
| 63 |
+
| drum_loop | 96.7% | 99.3% | | stab | 74.6% | 92.6% |
|
| 64 |
+
| vocal | 93.1% | 98.7% | | pluck | 74.3% | 97.6% |
|
| 65 |
+
| perc_loop | 92.4% | 98.2% | | percussion | 72.2% | 96.1% |
|
| 66 |
+
| cymbal | 92.3% | 98.7% | | acid | 71.7% | 93.0% |
|
| 67 |
+
| hat_closed | 90.0% | 99.5% | | keys | 69.3% | 93.7% |
|
| 68 |
+
| impact | 89.7% | 96.4% | | fx | 62.0% | 93.5% |
|
| 69 |
+
| pad | 88.7% | 96.7% | | atmo | 61.7% | 91.5% |
|
| 70 |
+
| bass | 88.2% | 98.0% | | sweep | 57.6% | 94.3% |
|
| 71 |
+
| clap | 87.7% | 98.3% | | | | |
|
| 72 |
+
| hat_open | 85.4% | 98.3% | | | | |
|
| 73 |
+
| snare | 81.9% | 97.3% | | | | |
|
| 74 |
+
| tom | 78.3% | 97.7% | | | | |
|
| 75 |
+
|
| 76 |
+
The most common confusions are between neighbours a listener would also hesitate over:
|
| 77 |
+
lead β pluck, bass β stab, sweep and atmo β fx, snare β clap. Show the top three
|
| 78 |
+
rather than one answer where it matters.
|
| 79 |
+
|
| 80 |
+
## Use
|
| 81 |
+
|
| 82 |
+
### With muse
|
| 83 |
+
|
| 84 |
+
```sh
|
| 85 |
+
brew install arraypress/tap/muse # fetches this model on first use
|
| 86 |
+
muse analyze kick.wav
|
| 87 |
+
# class kick (0.97) Β· drums β or tom 0.02, percussion 0.01
|
| 88 |
+
```
|
| 89 |
+
|
| 90 |
+
### From Swift
|
| 91 |
|
| 92 |
+
With [swift-music-analysis](https://github.com/arraypress/swift-music-analysis), pointing at
|
| 93 |
+
a folder holding this repository's files:
|
|
|
|
| 94 |
|
| 95 |
+
```swift
|
| 96 |
+
let classifier = try await SampleClassifier(directory: modelsFolder)
|
| 97 |
+
let verdict = try await classifier.classify(url: sampleURL)
|
| 98 |
+
verdict.label // "kick"
|
| 99 |
+
verdict.ranking.prefix(3) // the three likeliest, best first
|
| 100 |
+
verdict.tonal // probability that a key means something
|
| 101 |
+
```
|
| 102 |
|
| 103 |
+
### From Python (Core AI runtime)
|
| 104 |
|
| 105 |
+
```python
|
| 106 |
+
import asyncio, numpy as np, coreai.runtime as rt
|
| 107 |
+
|
| 108 |
+
options = rt.SpecializationOptions.from_preferred_compute_unit_kind(rt.ComputeUnitKind.gpu())
|
| 109 |
+
model = asyncio.run(rt.AIModel.load("muse-sample-types.aimodel", options))
|
| 110 |
+
classify = model.load_function("classify")
|
| 111 |
+
out = asyncio.run(classify(inputs={"input_features": rt.NDArray(mel)})) # mel: [1, 1, 1001, 64] float32
|
| 112 |
+
types = out["types"].numpy()[0]; gate = out["gate"].numpy()[0]
|
| 113 |
+
```
|
| 114 |
+
|
| 115 |
+
`mel` is CLAP's feature extractor output (`ClapFeatureExtractor` from
|
| 116 |
+
`laion/clap-htsat-unfused`, `padding="repeatpad"`) on the first ten seconds of the sound at
|
| 117 |
+
48 kHz mono. `mel-filters.f32` is its 513Γ64 slaney filterbank for implementations that
|
| 118 |
+
compute the spectrogram themselves.
|
| 119 |
+
|
| 120 |
+
## Files
|
| 121 |
+
|
| 122 |
+
| File | What it is |
|
| 123 |
+
|---|---|
|
| 124 |
+
| `muse-sample-types.aimodel/` | The model: entry point `classify`; in `input_features` [1, 1, 1001, 64]; out `types` [1, 22], `gate` [1, 2] (softmax probabilities) and `embedding` [1, 512] (L2-normalised) |
|
| 125 |
+
| `types.txt`, `gate.txt` | The labels of the two outputs, in order |
|
| 126 |
+
| `mel-filters.f32` | 513Γ64 slaney mel filterbank, little-endian float32, row-major |
|
| 127 |
+
|
| 128 |
+
Batch size is fixed at 1. Requires macOS 27 and Apple silicon (Core AI).
|
| 129 |
|
| 130 |
## How it was made
|
| 131 |
|
| 132 |
+
1. **Encoder.** [laion/clap-htsat-unfused](https://huggingface.co/laion/clap-htsat-unfused)
|
| 133 |
+
(LAION, Apache-2.0), weights unchanged. It won a contest on the same 10,322 samples,
|
| 134 |
+
packs held out: CLAP 70.7% top-1 with a linear probe, against larger_clap_general 68.2%,
|
| 135 |
+
AST-AudioSet 64.6% and MERT-v1-95M 58.1%.
|
| 136 |
+
2. **Data.** A licensed library of electronic-music sample packs: 80,102 one-shots, loops
|
| 137 |
+
and synth preset previews from 159 packs (construction-kit stems, demos and duplicates
|
| 138 |
+
left out). Every file was fingerprinted on its first ten seconds, the same window the
|
| 139 |
+
model hears in use.
|
| 140 |
+
3. **Labels.** From folder and file names, mapped onto the 22 types. Then checked: a head
|
| 141 |
+
trained without each file's pack predicted it, and the 4,163 files (5.5%) where it was
|
| 142 |
+
confident in another type and gave the folder's type almost nothing were reviewed. A
|
| 143 |
+
listening check found the folder name wrong in those cases (a sub-bass drop filed as
|
| 144 |
+
FX, vocal chops filed as stabs), so they took the model's label. The accuracy above
|
| 145 |
+
counts only files whose label came from their name.
|
| 146 |
+
4. **Heads.** Two small GELU networks on the normalised embedding, 512 β 512 β 22 and
|
| 147 |
+
512 β 128 β 2, trained with label smoothing and a cap of 3,000 files per type per epoch
|
| 148 |
+
(there are 15,848 kicks). The tonal gate's classes follow v1: pitched instruments are
|
| 149 |
+
tonal, drums and percussion atonal, FX left out of its training as ambiguous.
|
| 150 |
+
5. **Packaging.** Encoder and both heads exported together to Core AI; on the GPU the
|
| 151 |
+
outputs match PyTorch to within 5Γ10β»β·.
|
| 152 |
+
|
| 153 |
+
## Limitations
|
| 154 |
+
|
| 155 |
+
- Trained on electronic music sample packs (house, techno, trance, psytrance and
|
| 156 |
+
neighbours). Acoustic instruments, orchestral sounds and field recordings sit outside it;
|
| 157 |
+
guitar, brass, strings and choir were too rare to learn and have no label of their own.
|
| 158 |
+
- It hears the first ten seconds only.
|
| 159 |
+
- The FX family is the weakest because its types overlap in use; the top three is the
|
| 160 |
+
better answer there.
|
| 161 |
+
- Rising versus falling sweeps, and one-shot versus loop, are not this model's job:
|
| 162 |
+
CLAP barely hears direction, and both are measured separately in muse.
|
| 163 |
+
|
| 164 |
+
## History
|
| 165 |
+
|
| 166 |
+
- **v2 (October 2026):** one asset with both heads; 22 types (from 39), cleaned labels,
|
| 167 |
+
synth preset previews added, and the tonal gate moved onto CLAP. Supersedes the separate
|
| 168 |
+
[muse-tonal-gate](https://huggingface.co/arraypress/muse-tonal-gate).
|
| 169 |
+
- **v1 (October 2026):** CLAP encoder plus a separate 39-type head; tonal gate as a Create
|
| 170 |
+
ML model.
|
| 171 |
+
|
| 172 |
+
## Licence
|
| 173 |
+
|
| 174 |
+
Apache-2.0, following CLAP. The training audio is not distributed and cannot be recovered
|
| 175 |
+
from these files.
|