arraypress commited on
Commit
471461c
·
verified ·
1 Parent(s): 9445269

v2: one Core AI asset with 22 types and the tonal gate (85.4% / 97.5% top-3, gate 98.6%)

Browse files
.gitattributes CHANGED
@@ -36,3 +36,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
36
  clap-audio.aimodel/main.mlirb filter=lfs diff=lfs merge=lfs -text
37
  head-w1.f32 filter=lfs diff=lfs merge=lfs -text
38
  mel-filters.f32 filter=lfs diff=lfs merge=lfs -text
 
 
36
  clap-audio.aimodel/main.mlirb filter=lfs diff=lfs merge=lfs -text
37
  head-w1.f32 filter=lfs diff=lfs merge=lfs -text
38
  mel-filters.f32 filter=lfs diff=lfs merge=lfs -text
39
+ muse-sample-types.aimodel/main.mlirb filter=lfs diff=lfs merge=lfs -text
README.md CHANGED
@@ -10,27 +10,56 @@ tags:
10
  - apple-silicon
11
  ---
12
 
13
- # muse sample types
14
-
15
- Names what kind of sound a sample is, from **39 types** — kick, snare, clap, hat_closed,
16
- bass_loop, sub_bass, pluck, pad, riser, vocal_hit and more (`head-labels.txt`).
17
- [`muse`](https://github.com/arraypress) uses it for its `class` column.
18
-
19
- **68.3% right on held-out sample packs** (packs not seen in training; chance is
20
- about 3%). Most misses are between types a person would also confuse, such as
21
- sub_bass against bass_hit.
22
-
23
- ## How it works
24
-
25
- 1. `mel-filters.f32` — a 64-band mel spectrogram (48 kHz, 1,001 frames).
26
- 2. `clap-audio.aimodel` — the CLAP audio encoder,
27
- [laion/clap-htsat-unfused](https://huggingface.co/laion/clap-htsat-unfused)
28
- (LAION, Apache-2.0), converted to Core AI with its weights unchanged; it turns
29
- the spectrogram into a 512-number embedding.
30
- 3. `head-*.f32` — a small trained head on the L2-normalised embedding:
31
- 512 → 512 (GELU) → 39, softmax. Little-endian float32, row-major in PyTorch's
32
- Linear layout (outputs × inputs): `w1` [512×512], `b1` [512], `w2` [39×512],
33
- `b2` [39].
34
-
35
- Only the head was trained, on a licensed sample library; the audio is not
36
- distributed and cannot be recovered from these files. Apache-2.0, following CLAP.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
10
  - apple-silicon
11
  ---
12
 
13
+ # muse sample types (v2)
14
+
15
+ One Core AI asset that listens to an audio sample once and answers two questions:
16
+
17
+ - **What is it?** One of **22 types** in five families (`types.txt`):
18
+ - drums: kick, snare, clap, hat_closed, hat_open, cymbal, tom, percussion,
19
+ drum_loop (has a kick), perc_loop (no kick)
20
+ - bass: bass, acid
21
+ - synth: lead, pluck, pad, stab, keys
22
+ - vocals: vocal
23
+ - fx: sweep, impact, atmo, fx
24
+ - **Does a key signature mean anything here?** `atonal` or `tonal` (`gate.txt`) — so a
25
+ kick drum is not labelled "C minor".
26
+
27
+ ## Measured
28
+
29
+ All on whole sample packs the model never saw (5-fold, split by pack), counting only
30
+ files whose label came from their folder name:
31
+
32
+ | | v1 | **v2** |
33
+ |---|---|---|
34
+ | sample type, right first time | 68.3% (39 types) | **85.4%** (22 types) |
35
+ | sample type, right in the top three | — | **97.5%** |
36
+ | tonal gate | 93.3% (Create ML) | **98.6%** |
37
+
38
+ Strongest: kick 97.5%, drum_loop 96.7%, vocal 93.1%, perc_loop 92.4%, cymbal 92.3%.
39
+ Weakest are inside the FX family (sweep 58%, fx 62%, atmo 62%), whose types overlap.
40
+ Most common confusions: lead ↔ pluck, bass ↔ stab, sweep / atmo ↔ fx.
41
+
42
+ ## Inputs and outputs
43
+
44
+ `muse-sample-types.aimodel`, entry point `classify`:
45
+
46
+ - in: `input_features` [1, 1, 1001, 64] — CLAP's mel spectrogram (48 kHz mono, the first
47
+ ten seconds, repeat-padded; `mel-filters.f32` is the 513×64 slaney filterbank)
48
+ - out: `types` [1, 22] and `gate` [1, 2] softmax probabilities; `embedding` [1, 512], the
49
+ L2-normalised CLAP embedding, for similarity search
50
+
51
+ ## How it was made
52
+
53
+ 1. **Encoder:** [laion/clap-htsat-unfused](https://huggingface.co/laion/clap-htsat-unfused)
54
+ (LAION, Apache-2.0), weights unchanged. It beat larger_clap_general, MERT-v1-95M and
55
+ AST-AudioSet in a held-out-pack contest on the same 10,322 samples.
56
+ 2. **Labels:** from folder and file names over a licensed library (80,102 samples and synth
57
+ preset previews, 159 packs), then checked by the model itself: 4,163 files where a head
58
+ trained without that file's pack confidently disagreed were relabelled after a listening
59
+ check found the folder names wrong in those cases.
60
+ 3. **Heads:** two small GELU MLPs on the normalised embedding (512 → 512 → 22, and
61
+ 512 → 128 → 2), trained with label smoothing and per-type caps, then compiled into the
62
+ asset with the encoder. Core AI output matches PyTorch to within 5×10⁻⁷.
63
+
64
+ The training audio is not distributed and cannot be recovered from these files.
65
+ Apache-2.0, following CLAP.
clap-audio.aimodel/main.hash DELETED
@@ -1 +0,0 @@
1
- ��}��w���UBgdhAO4�jJ���
 
 
clap-audio.aimodel/metadata.json DELETED
@@ -1,8 +0,0 @@
1
- {
2
- "producer" : "coreai-core 1.0.0b2",
3
- "creationDate" : "20260827T182520Z",
4
- "license" : "Apache-2.0",
5
- "description" : "CLAP audio encoder (laion\/clap-htsat-unfused) converted to Core AI. Mel spectrogram [B,1,1001,64] in, 512-d projected joint-space audio embedding out. Weights unchanged; format conversion only.",
6
- "assetVersion" : "2.0",
7
- "author" : "LAION"
8
- }
 
 
 
 
 
 
 
 
 
gate.txt ADDED
@@ -0,0 +1,2 @@
 
 
 
1
+ atonal
2
+ tonal
head-b1.f32 DELETED
Binary file (2.05 kB)
 
head-b2.f32 DELETED
Binary file (156 Bytes)
 
head-w1.f32 DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:8619987d18be7f5b3f1ff565c5f68eb02bb814c3220b93c7571a17c1400003a9
3
- size 1048576
 
 
 
 
head-w2.f32 DELETED
Binary file (79.9 kB)
 
muse-sample-types.aimodel/main.hash ADDED
@@ -0,0 +1,2 @@
 
 
 
1
+ }�_���Z�>dHpq�Їԏ��J�o��
2
+
{clap-audio.aimodel → muse-sample-types.aimodel}/main.mlirb RENAMED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:bcb27d80f07790ba835542676468414f34da6a4afbbb950df80ecf5c67392546
3
- size 132854940
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7d8c015fe113dfc85aca3e64487071128a01d087d48fb1f2a24ad46feff50a0d
3
+ size 134219854
muse-sample-types.aimodel/metadata.json ADDED
@@ -0,0 +1,8 @@
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "license" : "Apache-2.0",
3
+ "assetVersion" : "2.0",
4
+ "description" : "CLAP (laion\/clap-htsat-unfused) + two heads. Mel [1,1,1001,64] in; types [1,22] (types.txt), gate [1,2] (gate.txt), embedding [1,512] L2-normalised out.",
5
+ "producer" : "coreai-core 1.0.0b2",
6
+ "creationDate" : "20261010T165935Z",
7
+ "author" : "LAION (CLAP encoder); sample-type and tonal-gate heads trained for muse"
8
+ }
head-labels.txt → types.txt RENAMED
@@ -1,15 +1,8 @@
1
  acid
2
- arp
3
  atmo
4
  bass
5
- bass_hit
6
- bass_loop
7
- chord
8
  clap
9
- crash
10
  cymbal
11
- downer
12
- drum_fill
13
  drum_loop
14
  fx
15
  hat_closed
@@ -18,22 +11,12 @@ impact
18
  keys
19
  kick
20
  lead
21
- noise
22
  pad
 
23
  percussion
24
- piano
25
  pluck
26
- reese_bass
27
- ride
28
- riser
29
- shaker
30
  snare
31
  stab
32
- strings
33
- sub_bass
34
  sweep
35
- synth_hit
36
- synth_loop
37
  tom
38
- vocal_hit
39
- vocal_loop
 
1
  acid
 
2
  atmo
3
  bass
 
 
 
4
  clap
 
5
  cymbal
 
 
6
  drum_loop
7
  fx
8
  hat_closed
 
11
  keys
12
  kick
13
  lead
 
14
  pad
15
+ perc_loop
16
  percussion
 
17
  pluck
 
 
 
 
18
  snare
19
  stab
 
 
20
  sweep
 
 
21
  tom
22
+ vocal