arraypress commited on
Commit
ea53cbb
Β·
verified Β·
1 Parent(s): 471461c

Model card: labels, per-type accuracy, usage, method, limitations

Browse files
Files changed (1) hide show
  1. README.md +150 -40
README.md CHANGED
@@ -1,65 +1,175 @@
1
  ---
2
  license: apache-2.0
 
3
  pipeline_tag: audio-classification
4
  tags:
5
  - audio-classification
6
  - music
7
  - samples
 
8
  - clap
9
  - core-ai
10
  - apple-silicon
 
11
  ---
12
 
13
- # muse sample types (v2)
14
 
15
- One Core AI asset that listens to an audio sample once and answers two questions:
 
16
 
17
- - **What is it?** One of **22 types** in five families (`types.txt`):
18
- - drums: kick, snare, clap, hat_closed, hat_open, cymbal, tom, percussion,
19
- drum_loop (has a kick), perc_loop (no kick)
20
- - bass: bass, acid
21
- - synth: lead, pluck, pad, stab, keys
22
- - vocals: vocal
23
- - fx: sweep, impact, atmo, fx
24
- - **Does a key signature mean anything here?** `atonal` or `tonal` (`gate.txt`) β€” so a
25
- kick drum is not labelled "C minor".
26
 
27
- ## Measured
 
 
28
 
29
- All on whole sample packs the model never saw (5-fold, split by pack), counting only
30
- files whose label came from their folder name:
31
 
32
- | | v1 | **v2** |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
33
  |---|---|---|
34
- | sample type, right first time | 68.3% (39 types) | **85.4%** (22 types) |
35
- | sample type, right in the top three | β€” | **97.5%** |
36
- | tonal gate | 93.3% (Create ML) | **98.6%** |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
37
 
38
- Strongest: kick 97.5%, drum_loop 96.7%, vocal 93.1%, perc_loop 92.4%, cymbal 92.3%.
39
- Weakest are inside the FX family (sweep 58%, fx 62%, atmo 62%), whose types overlap.
40
- Most common confusions: lead ↔ pluck, bass ↔ stab, sweep / atmo ↔ fx.
41
 
42
- ## Inputs and outputs
 
 
 
 
 
 
43
 
44
- `muse-sample-types.aimodel`, entry point `classify`:
45
 
46
- - in: `input_features` [1, 1, 1001, 64] β€” CLAP's mel spectrogram (48 kHz mono, the first
47
- ten seconds, repeat-padded; `mel-filters.f32` is the 513Γ—64 slaney filterbank)
48
- - out: `types` [1, 22] and `gate` [1, 2] softmax probabilities; `embedding` [1, 512], the
49
- L2-normalised CLAP embedding, for similarity search
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
50
 
51
  ## How it was made
52
 
53
- 1. **Encoder:** [laion/clap-htsat-unfused](https://huggingface.co/laion/clap-htsat-unfused)
54
- (LAION, Apache-2.0), weights unchanged. It beat larger_clap_general, MERT-v1-95M and
55
- AST-AudioSet in a held-out-pack contest on the same 10,322 samples.
56
- 2. **Labels:** from folder and file names over a licensed library (80,102 samples and synth
57
- preset previews, 159 packs), then checked by the model itself: 4,163 files where a head
58
- trained without that file's pack confidently disagreed were relabelled after a listening
59
- check found the folder names wrong in those cases.
60
- 3. **Heads:** two small GELU MLPs on the normalised embedding (512 β†’ 512 β†’ 22, and
61
- 512 β†’ 128 β†’ 2), trained with label smoothing and per-type caps, then compiled into the
62
- asset with the encoder. Core AI output matches PyTorch to within 5Γ—10⁻⁷.
63
-
64
- The training audio is not distributed and cannot be recovered from these files.
65
- Apache-2.0, following CLAP.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  license: apache-2.0
3
+ base_model: laion/clap-htsat-unfused
4
  pipeline_tag: audio-classification
5
  tags:
6
  - audio-classification
7
  - music
8
  - samples
9
+ - sample-library
10
  - clap
11
  - core-ai
12
  - apple-silicon
13
+ - on-device
14
  ---
15
 
16
+ # muse sample types
17
 
18
+ **Name any audio sample on a Mac, on-device.** One Core AI model that listens to a sample
19
+ once and answers two questions:
20
 
21
+ 1. **What is it?** One of 22 sample types, such as kick, snare, bass, pluck, pad, vocal or
22
+ sweep, with the runner-up types and their probabilities.
23
+ 2. **Does a musical key mean anything here?** Tonal or atonal, so a kick drum is never
24
+ labelled "C minor".
 
 
 
 
 
25
 
26
+ It also returns the sound's 512-number CLAP embedding, for "find sounds like this" search.
27
+ It is the model behind the `class` and `tonal` columns of the
28
+ [`muse`](https://github.com/arraypress) command-line tool, which downloads it on first use.
29
 
30
+ ## Labels
 
31
 
32
+ 22 types in five families (`types.txt`). A label says what the sound *is*; whether it is a
33
+ one-shot or a loop is a separate question, so a sub-bass hit and a rolling bass line are
34
+ both `bass`.
35
+
36
+ | Family | Types |
37
+ |---|---|
38
+ | drums | `kick`, `snare`, `clap`, `hat_closed`, `hat_open`, `cymbal` (crash, ride), `tom`, `percussion` (incl. shakers), `drum_loop` (a full loop, with a kick), `perc_loop` (hats, shakers, percussion, no kick) |
39
+ | bass | `bass` (hits, loops, sub, reese), `acid` (303-style lines) |
40
+ | synth | `lead`, `pluck`, `pad`, `stab` (incl. chords and synth hits), `keys` (incl. piano) |
41
+ | vocals | `vocal` (hits and loops) |
42
+ | fx | `sweep` (risers and downlifters), `impact`, `atmo` (incl. noise), `fx` |
43
+
44
+ Gate labels (`gate.txt`): `atonal`, `tonal`.
45
+
46
+ ## Accuracy
47
+
48
+ Measured with 5-fold cross-validation **split by sample pack**: every number is for packs
49
+ the model never saw in training, counting only files whose label came from their folder
50
+ name.
51
+
52
+ | | v1 (Aug 2026) | **v2** |
53
  |---|---|---|
54
+ | Sample type, right first time | 68.3% (39 types) | **85.4%** (22 types) |
55
+ | Sample type, right in the top three | β€” | **97.5%** |
56
+ | Tonal gate | 93.3% (Create ML) | **98.6%** |
57
+
58
+ Per type, v2, right first time / in the top three:
59
+
60
+ | Type | Top-1 | Top-3 | | Type | Top-1 | Top-3 |
61
+ |---|---|---|---|---|---|---|
62
+ | kick | 97.5% | 99.7% | | lead | 76.8% | 97.6% |
63
+ | drum_loop | 96.7% | 99.3% | | stab | 74.6% | 92.6% |
64
+ | vocal | 93.1% | 98.7% | | pluck | 74.3% | 97.6% |
65
+ | perc_loop | 92.4% | 98.2% | | percussion | 72.2% | 96.1% |
66
+ | cymbal | 92.3% | 98.7% | | acid | 71.7% | 93.0% |
67
+ | hat_closed | 90.0% | 99.5% | | keys | 69.3% | 93.7% |
68
+ | impact | 89.7% | 96.4% | | fx | 62.0% | 93.5% |
69
+ | pad | 88.7% | 96.7% | | atmo | 61.7% | 91.5% |
70
+ | bass | 88.2% | 98.0% | | sweep | 57.6% | 94.3% |
71
+ | clap | 87.7% | 98.3% | | | | |
72
+ | hat_open | 85.4% | 98.3% | | | | |
73
+ | snare | 81.9% | 97.3% | | | | |
74
+ | tom | 78.3% | 97.7% | | | | |
75
+
76
+ The most common confusions are between neighbours a listener would also hesitate over:
77
+ lead ↔ pluck, bass ↔ stab, sweep and atmo ↔ fx, snare ↔ clap. Show the top three
78
+ rather than one answer where it matters.
79
+
80
+ ## Use
81
+
82
+ ### With muse
83
+
84
+ ```sh
85
+ brew install arraypress/tap/muse # fetches this model on first use
86
+ muse analyze kick.wav
87
+ # class kick (0.97) Β· drums β€” or tom 0.02, percussion 0.01
88
+ ```
89
+
90
+ ### From Swift
91
 
92
+ With [swift-music-analysis](https://github.com/arraypress/swift-music-analysis), pointing at
93
+ a folder holding this repository's files:
 
94
 
95
+ ```swift
96
+ let classifier = try await SampleClassifier(directory: modelsFolder)
97
+ let verdict = try await classifier.classify(url: sampleURL)
98
+ verdict.label // "kick"
99
+ verdict.ranking.prefix(3) // the three likeliest, best first
100
+ verdict.tonal // probability that a key means something
101
+ ```
102
 
103
+ ### From Python (Core AI runtime)
104
 
105
+ ```python
106
+ import asyncio, numpy as np, coreai.runtime as rt
107
+
108
+ options = rt.SpecializationOptions.from_preferred_compute_unit_kind(rt.ComputeUnitKind.gpu())
109
+ model = asyncio.run(rt.AIModel.load("muse-sample-types.aimodel", options))
110
+ classify = model.load_function("classify")
111
+ out = asyncio.run(classify(inputs={"input_features": rt.NDArray(mel)})) # mel: [1, 1, 1001, 64] float32
112
+ types = out["types"].numpy()[0]; gate = out["gate"].numpy()[0]
113
+ ```
114
+
115
+ `mel` is CLAP's feature extractor output (`ClapFeatureExtractor` from
116
+ `laion/clap-htsat-unfused`, `padding="repeatpad"`) on the first ten seconds of the sound at
117
+ 48 kHz mono. `mel-filters.f32` is its 513Γ—64 slaney filterbank for implementations that
118
+ compute the spectrogram themselves.
119
+
120
+ ## Files
121
+
122
+ | File | What it is |
123
+ |---|---|
124
+ | `muse-sample-types.aimodel/` | The model: entry point `classify`; in `input_features` [1, 1, 1001, 64]; out `types` [1, 22], `gate` [1, 2] (softmax probabilities) and `embedding` [1, 512] (L2-normalised) |
125
+ | `types.txt`, `gate.txt` | The labels of the two outputs, in order |
126
+ | `mel-filters.f32` | 513Γ—64 slaney mel filterbank, little-endian float32, row-major |
127
+
128
+ Batch size is fixed at 1. Requires macOS 27 and Apple silicon (Core AI).
129
 
130
  ## How it was made
131
 
132
+ 1. **Encoder.** [laion/clap-htsat-unfused](https://huggingface.co/laion/clap-htsat-unfused)
133
+ (LAION, Apache-2.0), weights unchanged. It won a contest on the same 10,322 samples,
134
+ packs held out: CLAP 70.7% top-1 with a linear probe, against larger_clap_general 68.2%,
135
+ AST-AudioSet 64.6% and MERT-v1-95M 58.1%.
136
+ 2. **Data.** A licensed library of electronic-music sample packs: 80,102 one-shots, loops
137
+ and synth preset previews from 159 packs (construction-kit stems, demos and duplicates
138
+ left out). Every file was fingerprinted on its first ten seconds, the same window the
139
+ model hears in use.
140
+ 3. **Labels.** From folder and file names, mapped onto the 22 types. Then checked: a head
141
+ trained without each file's pack predicted it, and the 4,163 files (5.5%) where it was
142
+ confident in another type and gave the folder's type almost nothing were reviewed. A
143
+ listening check found the folder name wrong in those cases (a sub-bass drop filed as
144
+ FX, vocal chops filed as stabs), so they took the model's label. The accuracy above
145
+ counts only files whose label came from their name.
146
+ 4. **Heads.** Two small GELU networks on the normalised embedding, 512 β†’ 512 β†’ 22 and
147
+ 512 β†’ 128 β†’ 2, trained with label smoothing and a cap of 3,000 files per type per epoch
148
+ (there are 15,848 kicks). The tonal gate's classes follow v1: pitched instruments are
149
+ tonal, drums and percussion atonal, FX left out of its training as ambiguous.
150
+ 5. **Packaging.** Encoder and both heads exported together to Core AI; on the GPU the
151
+ outputs match PyTorch to within 5Γ—10⁻⁷.
152
+
153
+ ## Limitations
154
+
155
+ - Trained on electronic music sample packs (house, techno, trance, psytrance and
156
+ neighbours). Acoustic instruments, orchestral sounds and field recordings sit outside it;
157
+ guitar, brass, strings and choir were too rare to learn and have no label of their own.
158
+ - It hears the first ten seconds only.
159
+ - The FX family is the weakest because its types overlap in use; the top three is the
160
+ better answer there.
161
+ - Rising versus falling sweeps, and one-shot versus loop, are not this model's job:
162
+ CLAP barely hears direction, and both are measured separately in muse.
163
+
164
+ ## History
165
+
166
+ - **v2 (October 2026):** one asset with both heads; 22 types (from 39), cleaned labels,
167
+ synth preset previews added, and the tonal gate moved onto CLAP. Supersedes the separate
168
+ [muse-tonal-gate](https://huggingface.co/arraypress/muse-tonal-gate).
169
+ - **v1 (October 2026):** CLAP encoder plus a separate 39-type head; tonal gate as a Create
170
+ ML model.
171
+
172
+ ## Licence
173
+
174
+ Apache-2.0, following CLAP. The training audio is not distributed and cannot be recovered
175
+ from these files.