v2: one Core AI asset with 22 types and the tonal gate (85.4% / 97.5% top-3, gate 98.6%)
Browse files- .gitattributes +1 -0
- README.md +53 -24
- clap-audio.aimodel/main.hash +0 -1
- clap-audio.aimodel/metadata.json +0 -8
- gate.txt +2 -0
- head-b1.f32 +0 -0
- head-b2.f32 +0 -0
- head-w1.f32 +0 -3
- head-w2.f32 +0 -0
- muse-sample-types.aimodel/main.hash +2 -0
- {clap-audio.aimodel → muse-sample-types.aimodel}/main.mlirb +2 -2
- muse-sample-types.aimodel/metadata.json +8 -0
- head-labels.txt → types.txt +2 -19
.gitattributes
CHANGED
|
@@ -36,3 +36,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
|
| 36 |
clap-audio.aimodel/main.mlirb filter=lfs diff=lfs merge=lfs -text
|
| 37 |
head-w1.f32 filter=lfs diff=lfs merge=lfs -text
|
| 38 |
mel-filters.f32 filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
| 36 |
clap-audio.aimodel/main.mlirb filter=lfs diff=lfs merge=lfs -text
|
| 37 |
head-w1.f32 filter=lfs diff=lfs merge=lfs -text
|
| 38 |
mel-filters.f32 filter=lfs diff=lfs merge=lfs -text
|
| 39 |
+
muse-sample-types.aimodel/main.mlirb filter=lfs diff=lfs merge=lfs -text
|
README.md
CHANGED
|
@@ -10,27 +10,56 @@ tags:
|
|
| 10 |
- apple-silicon
|
| 11 |
---
|
| 12 |
|
| 13 |
-
# muse sample types
|
| 14 |
-
|
| 15 |
-
|
| 16 |
-
|
| 17 |
-
|
| 18 |
-
|
| 19 |
-
|
| 20 |
-
|
| 21 |
-
|
| 22 |
-
|
| 23 |
-
|
| 24 |
-
|
| 25 |
-
|
| 26 |
-
|
| 27 |
-
|
| 28 |
-
|
| 29 |
-
|
| 30 |
-
|
| 31 |
-
|
| 32 |
-
|
| 33 |
-
|
| 34 |
-
|
| 35 |
-
|
| 36 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 10 |
- apple-silicon
|
| 11 |
---
|
| 12 |
|
| 13 |
+
# muse sample types (v2)
|
| 14 |
+
|
| 15 |
+
One Core AI asset that listens to an audio sample once and answers two questions:
|
| 16 |
+
|
| 17 |
+
- **What is it?** One of **22 types** in five families (`types.txt`):
|
| 18 |
+
- drums: kick, snare, clap, hat_closed, hat_open, cymbal, tom, percussion,
|
| 19 |
+
drum_loop (has a kick), perc_loop (no kick)
|
| 20 |
+
- bass: bass, acid
|
| 21 |
+
- synth: lead, pluck, pad, stab, keys
|
| 22 |
+
- vocals: vocal
|
| 23 |
+
- fx: sweep, impact, atmo, fx
|
| 24 |
+
- **Does a key signature mean anything here?** `atonal` or `tonal` (`gate.txt`) — so a
|
| 25 |
+
kick drum is not labelled "C minor".
|
| 26 |
+
|
| 27 |
+
## Measured
|
| 28 |
+
|
| 29 |
+
All on whole sample packs the model never saw (5-fold, split by pack), counting only
|
| 30 |
+
files whose label came from their folder name:
|
| 31 |
+
|
| 32 |
+
| | v1 | **v2** |
|
| 33 |
+
|---|---|---|
|
| 34 |
+
| sample type, right first time | 68.3% (39 types) | **85.4%** (22 types) |
|
| 35 |
+
| sample type, right in the top three | — | **97.5%** |
|
| 36 |
+
| tonal gate | 93.3% (Create ML) | **98.6%** |
|
| 37 |
+
|
| 38 |
+
Strongest: kick 97.5%, drum_loop 96.7%, vocal 93.1%, perc_loop 92.4%, cymbal 92.3%.
|
| 39 |
+
Weakest are inside the FX family (sweep 58%, fx 62%, atmo 62%), whose types overlap.
|
| 40 |
+
Most common confusions: lead ↔ pluck, bass ↔ stab, sweep / atmo ↔ fx.
|
| 41 |
+
|
| 42 |
+
## Inputs and outputs
|
| 43 |
+
|
| 44 |
+
`muse-sample-types.aimodel`, entry point `classify`:
|
| 45 |
+
|
| 46 |
+
- in: `input_features` [1, 1, 1001, 64] — CLAP's mel spectrogram (48 kHz mono, the first
|
| 47 |
+
ten seconds, repeat-padded; `mel-filters.f32` is the 513×64 slaney filterbank)
|
| 48 |
+
- out: `types` [1, 22] and `gate` [1, 2] softmax probabilities; `embedding` [1, 512], the
|
| 49 |
+
L2-normalised CLAP embedding, for similarity search
|
| 50 |
+
|
| 51 |
+
## How it was made
|
| 52 |
+
|
| 53 |
+
1. **Encoder:** [laion/clap-htsat-unfused](https://huggingface.co/laion/clap-htsat-unfused)
|
| 54 |
+
(LAION, Apache-2.0), weights unchanged. It beat larger_clap_general, MERT-v1-95M and
|
| 55 |
+
AST-AudioSet in a held-out-pack contest on the same 10,322 samples.
|
| 56 |
+
2. **Labels:** from folder and file names over a licensed library (80,102 samples and synth
|
| 57 |
+
preset previews, 159 packs), then checked by the model itself: 4,163 files where a head
|
| 58 |
+
trained without that file's pack confidently disagreed were relabelled after a listening
|
| 59 |
+
check found the folder names wrong in those cases.
|
| 60 |
+
3. **Heads:** two small GELU MLPs on the normalised embedding (512 → 512 → 22, and
|
| 61 |
+
512 → 128 → 2), trained with label smoothing and per-type caps, then compiled into the
|
| 62 |
+
asset with the encoder. Core AI output matches PyTorch to within 5×10⁻⁷.
|
| 63 |
+
|
| 64 |
+
The training audio is not distributed and cannot be recovered from these files.
|
| 65 |
+
Apache-2.0, following CLAP.
|
clap-audio.aimodel/main.hash
DELETED
|
@@ -1 +0,0 @@
|
|
| 1 |
-
��}��w���UBgdhAO4�jJ���
|
|
|
|
|
|
clap-audio.aimodel/metadata.json
DELETED
|
@@ -1,8 +0,0 @@
|
|
| 1 |
-
{
|
| 2 |
-
"producer" : "coreai-core 1.0.0b2",
|
| 3 |
-
"creationDate" : "20260827T182520Z",
|
| 4 |
-
"license" : "Apache-2.0",
|
| 5 |
-
"description" : "CLAP audio encoder (laion\/clap-htsat-unfused) converted to Core AI. Mel spectrogram [B,1,1001,64] in, 512-d projected joint-space audio embedding out. Weights unchanged; format conversion only.",
|
| 6 |
-
"assetVersion" : "2.0",
|
| 7 |
-
"author" : "LAION"
|
| 8 |
-
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
gate.txt
ADDED
|
@@ -0,0 +1,2 @@
|
|
|
|
|
|
|
|
|
|
| 1 |
+
atonal
|
| 2 |
+
tonal
|
head-b1.f32
DELETED
|
Binary file (2.05 kB)
|
|
|
head-b2.f32
DELETED
|
Binary file (156 Bytes)
|
|
|
head-w1.f32
DELETED
|
@@ -1,3 +0,0 @@
|
|
| 1 |
-
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:8619987d18be7f5b3f1ff565c5f68eb02bb814c3220b93c7571a17c1400003a9
|
| 3 |
-
size 1048576
|
|
|
|
|
|
|
|
|
|
|
|
head-w2.f32
DELETED
|
Binary file (79.9 kB)
|
|
|
muse-sample-types.aimodel/main.hash
ADDED
|
@@ -0,0 +1,2 @@
|
|
|
|
|
|
|
|
|
|
| 1 |
+
}�_���Z�>dHpq�Їԏ��J�o��
|
| 2 |
+
|
{clap-audio.aimodel → muse-sample-types.aimodel}/main.mlirb
RENAMED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:7d8c015fe113dfc85aca3e64487071128a01d087d48fb1f2a24ad46feff50a0d
|
| 3 |
+
size 134219854
|
muse-sample-types.aimodel/metadata.json
ADDED
|
@@ -0,0 +1,8 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"license" : "Apache-2.0",
|
| 3 |
+
"assetVersion" : "2.0",
|
| 4 |
+
"description" : "CLAP (laion\/clap-htsat-unfused) + two heads. Mel [1,1,1001,64] in; types [1,22] (types.txt), gate [1,2] (gate.txt), embedding [1,512] L2-normalised out.",
|
| 5 |
+
"producer" : "coreai-core 1.0.0b2",
|
| 6 |
+
"creationDate" : "20261010T165935Z",
|
| 7 |
+
"author" : "LAION (CLAP encoder); sample-type and tonal-gate heads trained for muse"
|
| 8 |
+
}
|
head-labels.txt → types.txt
RENAMED
|
@@ -1,15 +1,8 @@
|
|
| 1 |
acid
|
| 2 |
-
arp
|
| 3 |
atmo
|
| 4 |
bass
|
| 5 |
-
bass_hit
|
| 6 |
-
bass_loop
|
| 7 |
-
chord
|
| 8 |
clap
|
| 9 |
-
crash
|
| 10 |
cymbal
|
| 11 |
-
downer
|
| 12 |
-
drum_fill
|
| 13 |
drum_loop
|
| 14 |
fx
|
| 15 |
hat_closed
|
|
@@ -18,22 +11,12 @@ impact
|
|
| 18 |
keys
|
| 19 |
kick
|
| 20 |
lead
|
| 21 |
-
noise
|
| 22 |
pad
|
|
|
|
| 23 |
percussion
|
| 24 |
-
piano
|
| 25 |
pluck
|
| 26 |
-
reese_bass
|
| 27 |
-
ride
|
| 28 |
-
riser
|
| 29 |
-
shaker
|
| 30 |
snare
|
| 31 |
stab
|
| 32 |
-
strings
|
| 33 |
-
sub_bass
|
| 34 |
sweep
|
| 35 |
-
synth_hit
|
| 36 |
-
synth_loop
|
| 37 |
tom
|
| 38 |
-
|
| 39 |
-
vocal_loop
|
|
|
|
| 1 |
acid
|
|
|
|
| 2 |
atmo
|
| 3 |
bass
|
|
|
|
|
|
|
|
|
|
| 4 |
clap
|
|
|
|
| 5 |
cymbal
|
|
|
|
|
|
|
| 6 |
drum_loop
|
| 7 |
fx
|
| 8 |
hat_closed
|
|
|
|
| 11 |
keys
|
| 12 |
kick
|
| 13 |
lead
|
|
|
|
| 14 |
pad
|
| 15 |
+
perc_loop
|
| 16 |
percussion
|
|
|
|
| 17 |
pluck
|
|
|
|
|
|
|
|
|
|
|
|
|
| 18 |
snare
|
| 19 |
stab
|
|
|
|
|
|
|
| 20 |
sweep
|
|
|
|
|
|
|
| 21 |
tom
|
| 22 |
+
vocal
|
|
|