File size: 7,706 Bytes
64a505d
 
ea53cbb
64a505d
 
 
 
 
ea53cbb
64a505d
 
 
ea53cbb
64a505d
 
ea53cbb
471461c
ea53cbb
 
471461c
ea53cbb
 
 
 
471461c
ea53cbb
 
 
471461c
ea53cbb
471461c
ea53cbb
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
471461c
ea53cbb
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
3448141
 
ea53cbb
3448141
 
 
ea53cbb
 
 
471461c
ea53cbb
 
471461c
ea53cbb
 
 
 
 
 
 
471461c
ea53cbb
471461c
ea53cbb
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
471461c
 
 
ea53cbb
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
---
license: apache-2.0
base_model: laion/clap-htsat-unfused
pipeline_tag: audio-classification
tags:
  - audio-classification
  - music
  - samples
  - sample-library
  - clap
  - core-ai
  - apple-silicon
  - on-device
---

# muse sample types

**Name any audio sample on a Mac, on-device.** One Core AI model that listens to a sample
once and answers two questions:

1. **What is it?** One of 22 sample types, such as kick, snare, bass, pluck, pad, vocal or
   sweep, with the runner-up types and their probabilities.
2. **Does a musical key mean anything here?** Tonal or atonal, so a kick drum is never
   labelled "C minor".

It also returns the sound's 512-number CLAP embedding, for "find sounds like this" search.
It is the model behind the `class` and `tonal` columns of the
[`muse`](https://github.com/arraypress) command-line tool, which downloads it on first use.

## Labels

22 types in five families (`types.txt`). A label says what the sound *is*; whether it is a
one-shot or a loop is a separate question, so a sub-bass hit and a rolling bass line are
both `bass`.

| Family | Types |
|---|---|
| drums | `kick`, `snare`, `clap`, `hat_closed`, `hat_open`, `cymbal` (crash, ride), `tom`, `percussion` (incl. shakers), `drum_loop` (a full loop, with a kick), `perc_loop` (hats, shakers, percussion, no kick) |
| bass | `bass` (hits, loops, sub, reese), `acid` (303-style lines) |
| synth | `lead`, `pluck`, `pad`, `stab` (incl. chords and synth hits), `keys` (incl. piano) |
| vocals | `vocal` (hits and loops) |
| fx | `sweep` (risers and downlifters), `impact`, `atmo` (incl. noise), `fx` |

Gate labels (`gate.txt`): `atonal`, `tonal`.

## Accuracy

Measured with 5-fold cross-validation **split by sample pack**: every number is for packs
the model never saw in training, counting only files whose label came from their folder
name.

| | v1 (Aug 2026) | **v2** |
|---|---|---|
| Sample type, right first time | 68.3% (39 types) | **85.4%** (22 types) |
| Sample type, right in the top three | β€” | **97.5%** |
| Tonal gate | 93.3% (Create ML) | **98.6%** |

Per type, v2, right first time / in the top three:

| Type | Top-1 | Top-3 | | Type | Top-1 | Top-3 |
|---|---|---|---|---|---|---|
| kick | 97.5% | 99.7% | | lead | 76.8% | 97.6% |
| drum_loop | 96.7% | 99.3% | | stab | 74.6% | 92.6% |
| vocal | 93.1% | 98.7% | | pluck | 74.3% | 97.6% |
| perc_loop | 92.4% | 98.2% | | percussion | 72.2% | 96.1% |
| cymbal | 92.3% | 98.7% | | acid | 71.7% | 93.0% |
| hat_closed | 90.0% | 99.5% | | keys | 69.3% | 93.7% |
| impact | 89.7% | 96.4% | | fx | 62.0% | 93.5% |
| pad | 88.7% | 96.7% | | atmo | 61.7% | 91.5% |
| bass | 88.2% | 98.0% | | sweep | 57.6% | 94.3% |
| clap | 87.7% | 98.3% | | | | |
| hat_open | 85.4% | 98.3% | | | | |
| snare | 81.9% | 97.3% | | | | |
| tom | 78.3% | 97.7% | | | | |

The most common confusions are between neighbours a listener would also hesitate over:
lead ↔ pluck, bass ↔ stab, sweep and atmo ↔ fx, snare ↔ clap. Show the top three
rather than one answer where it matters.

## Use

### With muse

`muse` downloads this model the first time it needs it. A real run on a snare one-shot:

```sh
muse analyze snare.wav
#   key     β€”  (atonal β€” suppressed by gate)
#   class   snare  (0.49)  Β· drums  β€” or percussion 0.38, tom 0.03
```

### From Swift

With [swift-music-analysis](https://github.com/arraypress/swift-music-analysis), pointing at
a folder holding this repository's files:

```swift
let classifier = try await SampleClassifier(directory: modelsFolder)
let verdict = try await classifier.classify(url: sampleURL)
verdict.label            // "kick"
verdict.ranking.prefix(3) // the three likeliest, best first
verdict.tonal            // probability that a key means something
```

### From Python (Core AI runtime)

```python
import asyncio, numpy as np, coreai.runtime as rt

options = rt.SpecializationOptions.from_preferred_compute_unit_kind(rt.ComputeUnitKind.gpu())
model = asyncio.run(rt.AIModel.load("muse-sample-types.aimodel", options))
classify = model.load_function("classify")
out = asyncio.run(classify(inputs={"input_features": rt.NDArray(mel)}))   # mel: [1, 1, 1001, 64] float32
types = out["types"].numpy()[0]; gate = out["gate"].numpy()[0]
```

`mel` is CLAP's feature extractor output (`ClapFeatureExtractor` from
`laion/clap-htsat-unfused`, `padding="repeatpad"`) on the first ten seconds of the sound at
48 kHz mono. `mel-filters.f32` is its 513Γ—64 slaney filterbank for implementations that
compute the spectrogram themselves.

## Files

| File | What it is |
|---|---|
| `muse-sample-types.aimodel/` | The model: entry point `classify`; in `input_features` [1, 1, 1001, 64]; out `types` [1, 22], `gate` [1, 2] (softmax probabilities) and `embedding` [1, 512] (L2-normalised) |
| `types.txt`, `gate.txt` | The labels of the two outputs, in order |
| `mel-filters.f32` | 513Γ—64 slaney mel filterbank, little-endian float32, row-major |

Batch size is fixed at 1. Requires macOS 27 and Apple silicon (Core AI).

## How it was made

1. **Encoder.** [laion/clap-htsat-unfused](https://huggingface.co/laion/clap-htsat-unfused)
   (LAION, Apache-2.0), weights unchanged. It won a contest on the same 10,322 samples,
   packs held out: CLAP 70.7% top-1 with a linear probe, against larger_clap_general 68.2%,
   AST-AudioSet 64.6% and MERT-v1-95M 58.1%.
2. **Data.** A licensed library of electronic-music sample packs: 80,102 one-shots, loops
   and synth preset previews from 159 packs (construction-kit stems, demos and duplicates
   left out). Every file was fingerprinted on its first ten seconds, the same window the
   model hears in use.
3. **Labels.** From folder and file names, mapped onto the 22 types. Then checked: a head
   trained without each file's pack predicted it, and the 4,163 files (5.5%) where it was
   confident in another type and gave the folder's type almost nothing were reviewed. A
   listening check found the folder name wrong in those cases (a sub-bass drop filed as
   FX, vocal chops filed as stabs), so they took the model's label. The accuracy above
   counts only files whose label came from their name.
4. **Heads.** Two small GELU networks on the normalised embedding, 512 β†’ 512 β†’ 22 and
   512 β†’ 128 β†’ 2, trained with label smoothing and a cap of 3,000 files per type per epoch
   (there are 15,848 kicks). The tonal gate's classes follow v1: pitched instruments are
   tonal, drums and percussion atonal, FX left out of its training as ambiguous.
5. **Packaging.** Encoder and both heads exported together to Core AI; on the GPU the
   outputs match PyTorch to within 5Γ—10⁻⁷.

## Limitations

- Trained on electronic music sample packs (house, techno, trance, psytrance and
  neighbours). Acoustic instruments, orchestral sounds and field recordings sit outside it;
  guitar, brass, strings and choir were too rare to learn and have no label of their own.
- It hears the first ten seconds only.
- The FX family is the weakest because its types overlap in use; the top three is the
  better answer there.
- Rising versus falling sweeps, and one-shot versus loop, are not this model's job:
  CLAP barely hears direction, and both are measured separately in muse.

## History

- **v2 (October 2026):** one asset with both heads; 22 types (from 39), cleaned labels,
  synth preset previews added, and the tonal gate moved onto CLAP. Supersedes the separate
  [muse-tonal-gate](https://huggingface.co/arraypress/muse-tonal-gate).
- **v1 (October 2026):** CLAP encoder plus a separate 39-type head; tonal gate as a Create
  ML model.

## Licence

Apache-2.0, following CLAP. The training audio is not distributed and cannot be recovered
from these files.