File size: 7,706 Bytes
64a505d ea53cbb 64a505d ea53cbb 64a505d ea53cbb 64a505d ea53cbb 471461c ea53cbb 471461c ea53cbb 471461c ea53cbb 471461c ea53cbb 471461c ea53cbb 471461c ea53cbb 3448141 ea53cbb 3448141 ea53cbb 471461c ea53cbb 471461c ea53cbb 471461c ea53cbb 471461c ea53cbb 471461c ea53cbb | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 | ---
license: apache-2.0
base_model: laion/clap-htsat-unfused
pipeline_tag: audio-classification
tags:
- audio-classification
- music
- samples
- sample-library
- clap
- core-ai
- apple-silicon
- on-device
---
# muse sample types
**Name any audio sample on a Mac, on-device.** One Core AI model that listens to a sample
once and answers two questions:
1. **What is it?** One of 22 sample types, such as kick, snare, bass, pluck, pad, vocal or
sweep, with the runner-up types and their probabilities.
2. **Does a musical key mean anything here?** Tonal or atonal, so a kick drum is never
labelled "C minor".
It also returns the sound's 512-number CLAP embedding, for "find sounds like this" search.
It is the model behind the `class` and `tonal` columns of the
[`muse`](https://github.com/arraypress) command-line tool, which downloads it on first use.
## Labels
22 types in five families (`types.txt`). A label says what the sound *is*; whether it is a
one-shot or a loop is a separate question, so a sub-bass hit and a rolling bass line are
both `bass`.
| Family | Types |
|---|---|
| drums | `kick`, `snare`, `clap`, `hat_closed`, `hat_open`, `cymbal` (crash, ride), `tom`, `percussion` (incl. shakers), `drum_loop` (a full loop, with a kick), `perc_loop` (hats, shakers, percussion, no kick) |
| bass | `bass` (hits, loops, sub, reese), `acid` (303-style lines) |
| synth | `lead`, `pluck`, `pad`, `stab` (incl. chords and synth hits), `keys` (incl. piano) |
| vocals | `vocal` (hits and loops) |
| fx | `sweep` (risers and downlifters), `impact`, `atmo` (incl. noise), `fx` |
Gate labels (`gate.txt`): `atonal`, `tonal`.
## Accuracy
Measured with 5-fold cross-validation **split by sample pack**: every number is for packs
the model never saw in training, counting only files whose label came from their folder
name.
| | v1 (Aug 2026) | **v2** |
|---|---|---|
| Sample type, right first time | 68.3% (39 types) | **85.4%** (22 types) |
| Sample type, right in the top three | β | **97.5%** |
| Tonal gate | 93.3% (Create ML) | **98.6%** |
Per type, v2, right first time / in the top three:
| Type | Top-1 | Top-3 | | Type | Top-1 | Top-3 |
|---|---|---|---|---|---|---|
| kick | 97.5% | 99.7% | | lead | 76.8% | 97.6% |
| drum_loop | 96.7% | 99.3% | | stab | 74.6% | 92.6% |
| vocal | 93.1% | 98.7% | | pluck | 74.3% | 97.6% |
| perc_loop | 92.4% | 98.2% | | percussion | 72.2% | 96.1% |
| cymbal | 92.3% | 98.7% | | acid | 71.7% | 93.0% |
| hat_closed | 90.0% | 99.5% | | keys | 69.3% | 93.7% |
| impact | 89.7% | 96.4% | | fx | 62.0% | 93.5% |
| pad | 88.7% | 96.7% | | atmo | 61.7% | 91.5% |
| bass | 88.2% | 98.0% | | sweep | 57.6% | 94.3% |
| clap | 87.7% | 98.3% | | | | |
| hat_open | 85.4% | 98.3% | | | | |
| snare | 81.9% | 97.3% | | | | |
| tom | 78.3% | 97.7% | | | | |
The most common confusions are between neighbours a listener would also hesitate over:
lead β pluck, bass β stab, sweep and atmo β fx, snare β clap. Show the top three
rather than one answer where it matters.
## Use
### With muse
`muse` downloads this model the first time it needs it. A real run on a snare one-shot:
```sh
muse analyze snare.wav
# key β (atonal β suppressed by gate)
# class snare (0.49) Β· drums β or percussion 0.38, tom 0.03
```
### From Swift
With [swift-music-analysis](https://github.com/arraypress/swift-music-analysis), pointing at
a folder holding this repository's files:
```swift
let classifier = try await SampleClassifier(directory: modelsFolder)
let verdict = try await classifier.classify(url: sampleURL)
verdict.label // "kick"
verdict.ranking.prefix(3) // the three likeliest, best first
verdict.tonal // probability that a key means something
```
### From Python (Core AI runtime)
```python
import asyncio, numpy as np, coreai.runtime as rt
options = rt.SpecializationOptions.from_preferred_compute_unit_kind(rt.ComputeUnitKind.gpu())
model = asyncio.run(rt.AIModel.load("muse-sample-types.aimodel", options))
classify = model.load_function("classify")
out = asyncio.run(classify(inputs={"input_features": rt.NDArray(mel)})) # mel: [1, 1, 1001, 64] float32
types = out["types"].numpy()[0]; gate = out["gate"].numpy()[0]
```
`mel` is CLAP's feature extractor output (`ClapFeatureExtractor` from
`laion/clap-htsat-unfused`, `padding="repeatpad"`) on the first ten seconds of the sound at
48 kHz mono. `mel-filters.f32` is its 513Γ64 slaney filterbank for implementations that
compute the spectrogram themselves.
## Files
| File | What it is |
|---|---|
| `muse-sample-types.aimodel/` | The model: entry point `classify`; in `input_features` [1, 1, 1001, 64]; out `types` [1, 22], `gate` [1, 2] (softmax probabilities) and `embedding` [1, 512] (L2-normalised) |
| `types.txt`, `gate.txt` | The labels of the two outputs, in order |
| `mel-filters.f32` | 513Γ64 slaney mel filterbank, little-endian float32, row-major |
Batch size is fixed at 1. Requires macOS 27 and Apple silicon (Core AI).
## How it was made
1. **Encoder.** [laion/clap-htsat-unfused](https://huggingface.co/laion/clap-htsat-unfused)
(LAION, Apache-2.0), weights unchanged. It won a contest on the same 10,322 samples,
packs held out: CLAP 70.7% top-1 with a linear probe, against larger_clap_general 68.2%,
AST-AudioSet 64.6% and MERT-v1-95M 58.1%.
2. **Data.** A licensed library of electronic-music sample packs: 80,102 one-shots, loops
and synth preset previews from 159 packs (construction-kit stems, demos and duplicates
left out). Every file was fingerprinted on its first ten seconds, the same window the
model hears in use.
3. **Labels.** From folder and file names, mapped onto the 22 types. Then checked: a head
trained without each file's pack predicted it, and the 4,163 files (5.5%) where it was
confident in another type and gave the folder's type almost nothing were reviewed. A
listening check found the folder name wrong in those cases (a sub-bass drop filed as
FX, vocal chops filed as stabs), so they took the model's label. The accuracy above
counts only files whose label came from their name.
4. **Heads.** Two small GELU networks on the normalised embedding, 512 β 512 β 22 and
512 β 128 β 2, trained with label smoothing and a cap of 3,000 files per type per epoch
(there are 15,848 kicks). The tonal gate's classes follow v1: pitched instruments are
tonal, drums and percussion atonal, FX left out of its training as ambiguous.
5. **Packaging.** Encoder and both heads exported together to Core AI; on the GPU the
outputs match PyTorch to within 5Γ10β»β·.
## Limitations
- Trained on electronic music sample packs (house, techno, trance, psytrance and
neighbours). Acoustic instruments, orchestral sounds and field recordings sit outside it;
guitar, brass, strings and choir were too rare to learn and have no label of their own.
- It hears the first ten seconds only.
- The FX family is the weakest because its types overlap in use; the top three is the
better answer there.
- Rising versus falling sweeps, and one-shot versus loop, are not this model's job:
CLAP barely hears direction, and both are measured separately in muse.
## History
- **v2 (October 2026):** one asset with both heads; 22 types (from 39), cleaned labels,
synth preset previews added, and the tonal gate moved onto CLAP. Supersedes the separate
[muse-tonal-gate](https://huggingface.co/arraypress/muse-tonal-gate).
- **v1 (October 2026):** CLAP encoder plus a separate 39-type head; tonal gate as a Create
ML model.
## Licence
Apache-2.0, following CLAP. The training audio is not distributed and cannot be recovered
from these files.
|