Release drone-v2 and rewrite the model card
Browse files- README.md +192 -66
- drone-v2.json +0 -0
- drone-v2.onnx +3 -0
- drone-v2.pt +3 -0
- drone-v2.safetensors +3 -0
README.md
CHANGED
|
@@ -1,99 +1,225 @@
|
|
| 1 |
---
|
| 2 |
license: mit
|
| 3 |
-
library_name:
|
|
|
|
| 4 |
tags:
|
| 5 |
- audio
|
| 6 |
- music
|
| 7 |
- key-detection
|
|
|
|
|
|
|
| 8 |
- constant-q-transform
|
| 9 |
- cnn
|
|
|
|
|
|
|
| 10 |
model-index:
|
| 11 |
-
- name: drone-
|
| 12 |
results:
|
| 13 |
- task:
|
| 14 |
name: Musical key detection
|
| 15 |
-
type: classification
|
| 16 |
dataset:
|
| 17 |
-
name: GiantSteps
|
| 18 |
-
type:
|
| 19 |
metrics:
|
| 20 |
-
- name:
|
| 21 |
-
type:
|
| 22 |
-
value:
|
| 23 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 24 |
type: other
|
| 25 |
-
value:
|
|
|
|
| 26 |
---
|
| 27 |
|
| 28 |
-
# drone-
|
| 29 |
-
|
| 30 |
-
|
| 31 |
-
|
| 32 |
-
|
| 33 |
-
|
| 34 |
-
|
| 35 |
-
|
| 36 |
-
|
| 37 |
-
|
| 38 |
-
|
| 39 |
-
|
|
| 40 |
-
|
|
| 41 |
-
|
|
| 42 |
-
|
|
| 43 |
-
|
|
| 44 |
-
|
| 45 |
-
|
| 46 |
-
|
| 47 |
-
|
| 48 |
-
|
| 49 |
-
|
| 50 |
-
|
| 51 |
-
|
| 52 |
-
-
|
| 53 |
-
|
| 54 |
-
|
| 55 |
-
|
| 56 |
-
-
|
| 57 |
-
|
| 58 |
-
|
| 59 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 60 |
|
| 61 |
-
|
| 62 |
|
| 63 |
-
|
|
|
|
|
|
|
| 64 |
|
| 65 |
```python
|
| 66 |
from drone import Predictor
|
| 67 |
|
| 68 |
-
predictor = Predictor() #
|
| 69 |
-
|
|
|
|
| 70 |
```
|
| 71 |
|
| 72 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 73 |
|
| 74 |
```python
|
|
|
|
| 75 |
import numpy as np
|
| 76 |
import onnxruntime as ort
|
| 77 |
-
import librosa
|
| 78 |
|
| 79 |
-
|
| 80 |
-
cqt = np.log1p(np.abs(librosa.cqt(
|
| 81 |
-
y, sr=22050, hop_length=4096, fmin=25.95661061576629,
|
| 82 |
-
n_bins=208, bins_per_octave=24)))
|
| 83 |
-
x = cqt[8:200].astype(np.float32)[None]
|
| 84 |
-
logits = ort.InferenceSession("drone-v1.onnx").run(["logits"], {"features": x})[0]
|
| 85 |
-
```
|
| 86 |
|
| 87 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 88 |
|
| 89 |
-
|
| 90 |
-
|
| 91 |
-
|
| 92 |
-
|
| 93 |
-
|
| 94 |
-
in the source repository.
|
| 95 |
-
|
| 96 |
-
## Caveats
|
| 97 |
|
| 98 |
-
|
| 99 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
license: mit
|
| 3 |
+
library_name: pytorch
|
| 4 |
+
pipeline_tag: audio-classification
|
| 5 |
tags:
|
| 6 |
- audio
|
| 7 |
- music
|
| 8 |
- key-detection
|
| 9 |
+
- music-information-retrieval
|
| 10 |
+
- dj
|
| 11 |
- constant-q-transform
|
| 12 |
- cnn
|
| 13 |
+
- onnx
|
| 14 |
+
- safetensors
|
| 15 |
model-index:
|
| 16 |
+
- name: drone-v2
|
| 17 |
results:
|
| 18 |
- task:
|
| 19 |
name: Musical key detection
|
| 20 |
+
type: audio-classification
|
| 21 |
dataset:
|
| 22 |
+
name: GiantSteps Key with GiantSteps+ expert labels, single-key tracks (457)
|
| 23 |
+
type: giantsteps-plus
|
| 24 |
metrics:
|
| 25 |
+
- name: Exact accuracy
|
| 26 |
+
type: accuracy
|
| 27 |
+
value: 87.96
|
| 28 |
+
verified: false
|
| 29 |
+
- task:
|
| 30 |
+
name: Musical key detection
|
| 31 |
+
type: audio-classification
|
| 32 |
+
dataset:
|
| 33 |
+
name: GiantSteps Key, original labels (604)
|
| 34 |
+
type: giantsteps-key
|
| 35 |
+
metrics:
|
| 36 |
+
- name: Exact accuracy
|
| 37 |
+
type: accuracy
|
| 38 |
+
value: 69.87
|
| 39 |
+
verified: false
|
| 40 |
+
- name: MIREX weighted score
|
| 41 |
type: other
|
| 42 |
+
value: 75.45
|
| 43 |
+
verified: false
|
| 44 |
---
|
| 45 |
|
| 46 |
+
# drone-v2: musical key detection on a CPU
|
| 47 |
+
|
| 48 |
+
drone-v2 tells you the musical key of a song: one of 24 keys, the 12 tonics
|
| 49 |
+
(C, C#, D ... B) each in major or minor. It is a 109,128-parameter
|
| 50 |
+
convolutional network that reads a constant-Q spectrogram of the whole track
|
| 51 |
+
and runs in about 60 ms on one CPU core for a two-minute track. It was built for DJ software
|
| 52 |
+
([Euphoria](https://github.com/lvndry/euphoria)) and works anywhere you can
|
| 53 |
+
run PyTorch, ONNX Runtime, or a few hundred lines of native code.
|
| 54 |
+
|
| 55 |
+
| | |
|
| 56 |
+
|---|---|
|
| 57 |
+
| Output | One of 24 keys, with a softmax score for each |
|
| 58 |
+
| Size | 109,128 parameters; 0.44 MB of float32 weights |
|
| 59 |
+
| Speed | About 0.3 s per two-minute track on one core of an Apple M5, of which 57 ms is the network |
|
| 60 |
+
| Source | [github.com/lvndry/drone](https://github.com/lvndry/drone) (MIT) |
|
| 61 |
+
| Previous version | `drone-v1`, still in this repository |
|
| 62 |
+
|
| 63 |
+
## How accurate is it?
|
| 64 |
+
|
| 65 |
+
All detectors below were run on the same 604 GiantSteps previews (two-minute
|
| 66 |
+
excerpts of electronic dance music) and scored against the same labels.
|
| 67 |
+
**Exact** means the predicted key equals the reference key.
|
| 68 |
+
|
| 69 |
+
| Detector | Original GiantSteps labels (604) | Expert labels (571) | Expert labels, single-key tracks (457) |
|
| 70 |
+
|---|---:|---:|---:|
|
| 71 |
+
| libkeyfinder 2.2.8 | 60.4% | 69.9% | 76.2% |
|
| 72 |
+
| madmom KEY_CNN | 67.9% | 77.4% | 83.4% |
|
| 73 |
+
| drone-v1 | 68.5% | 77.4% | 83.4% |
|
| 74 |
+
| **drone-v2** | **69.9%** | **80.7%** | **88.0%** |
|
| 75 |
+
|
| 76 |
+
Why three columns? The original GiantSteps labels agree with the
|
| 77 |
+
[GiantSteps+ expert re-annotation](https://zenodo.org/records/4153506) on only
|
| 78 |
+
82% of tracks, so part of every detector's "error" on the original labels is
|
| 79 |
+
label error. The single-key column keeps only tracks the experts marked with
|
| 80 |
+
one key throughout, the cleanest reference available.
|
| 81 |
+
|
| 82 |
+
On the expert single-key tracks, drone-v2 is ahead of libkeyfinder by 11.8
|
| 83 |
+
points (paired 95% interval 8.1 to 15.5) and of drone-v1 by 4.6 points
|
| 84 |
+
(2.0 to 7.4). On the half of those tracks never used for any development
|
| 85 |
+
decision, the gaps are 12.4 and 6.2 points.
|
| 86 |
+
|
| 87 |
+
Outside dance music, on 500 tracks of mixed genres from the Free Music Archive
|
| 88 |
+
([FMAKv2](https://zenodo.org/records/12759100)), drone-v2 scores 75.0% exact
|
| 89 |
+
against drone-v1's 65.4%: 72.4% and 77.6% on the two 250-track panels, a gain
|
| 90 |
+
of 9 to 10 points on each.
|
| 91 |
+
|
| 92 |
+
### What these numbers do not say
|
| 93 |
+
|
| 94 |
+
- **Not compared with Mixed In Key.** Published Mixed In Key results (85 to
|
| 95 |
+
89% exact) come from other tracks, labels and scoring, so they are not
|
| 96 |
+
comparable with this table.
|
| 97 |
+
- **GiantSteps is not a blind test for this model family.** It was used while
|
| 98 |
+
developing Drone, though never to train it. The 604 previews share no track
|
| 99 |
+
with the training set.
|
| 100 |
+
- **On the original GiantSteps labels**, a larger published model (KeyMyna,
|
| 101 |
+
22M parameters) reports 72.0% exact, above drone-v2's 69.9%.
|
| 102 |
+
- **Previews, not full mixes.** GiantSteps tracks are two-minute excerpts and
|
| 103 |
+
most training audio is 30-second clips or previews. Full-length tracks work
|
| 104 |
+
(the network averages over time), but accuracy on 6 to 10 minute club tracks
|
| 105 |
+
has not been measured separately.
|
| 106 |
+
- **One key per track.** A track that changes key gets the key the network
|
| 107 |
+
hears most.
|
| 108 |
+
- **The scores are not probabilities.** The softmax output ranks keys; a score
|
| 109 |
+
of 0.8 does not mean the key is right 80% of the time.
|
| 110 |
+
|
| 111 |
+
## Use it
|
| 112 |
+
|
| 113 |
+
### Python
|
| 114 |
+
|
| 115 |
+
In a project managed by [uv](https://docs.astral.sh/uv/):
|
| 116 |
+
|
| 117 |
+
```bash
|
| 118 |
+
uv add "drone-key @ git+https://github.com/lvndry/drone"
|
| 119 |
+
```
|
| 120 |
|
| 121 |
+
With pip:
|
| 122 |
|
| 123 |
+
```bash
|
| 124 |
+
pip install "drone-key @ git+https://github.com/lvndry/drone"
|
| 125 |
+
```
|
| 126 |
|
| 127 |
```python
|
| 128 |
from drone import Predictor
|
| 129 |
|
| 130 |
+
predictor = Predictor() # loads drone-v2; reuse it for many files
|
| 131 |
+
tonic, mode, score = predictor.predict("track.mp3")
|
| 132 |
+
print(tonic, mode) # e.g. "A minor"
|
| 133 |
```
|
| 134 |
|
| 135 |
+
`ffmpeg` must be installed: the package decodes audio with it. From a shell:
|
| 136 |
+
`python -m drone track.mp3`.
|
| 137 |
+
|
| 138 |
+
### Any language, with ONNX
|
| 139 |
+
|
| 140 |
+
`drone-v2.onnx` takes the spectrogram and returns 24 logits. You compute the
|
| 141 |
+
spectrogram yourself; this is the exact recipe, checked against the Python
|
| 142 |
+
package on 20 of 20 test tracks:
|
| 143 |
|
| 144 |
```python
|
| 145 |
+
import librosa
|
| 146 |
import numpy as np
|
| 147 |
import onnxruntime as ort
|
|
|
|
| 148 |
|
| 149 |
+
TONICS = ["C", "C#", "D", "D#", "E", "F", "F#", "G", "G#", "A", "A#", "B"]
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 150 |
|
| 151 |
+
samples, _ = librosa.load("track.mp3", sr=22050, mono=True)
|
| 152 |
+
lowest = librosa.note_to_hz("C1") * 2 ** (-4 / 12) # four semitones below C1
|
| 153 |
+
transform = librosa.cqt(
|
| 154 |
+
samples, sr=22050, hop_length=4096, fmin=lowest, n_bins=208, bins_per_octave=24
|
| 155 |
+
)
|
| 156 |
+
spectrogram = np.log1p(np.abs(transform))
|
| 157 |
+
features = spectrogram[8:200].astype(np.float32)[None] # (1, 192, frames)
|
| 158 |
|
| 159 |
+
session = ort.InferenceSession("drone-v2.onnx")
|
| 160 |
+
logits = session.run(["logits"], {"features": features})[0][0]
|
| 161 |
+
key_class = int(logits.argmax())
|
| 162 |
+
print(TONICS[key_class // 2], "minor" if key_class % 2 else "major")
|
| 163 |
+
```
|
|
|
|
|
|
|
|
|
|
| 164 |
|
| 165 |
+
### The contract, for a native port
|
| 166 |
+
|
| 167 |
+
| Step | Specification |
|
| 168 |
+
|---|---|
|
| 169 |
+
| Audio | Mono (average the channels), 22,050 Hz |
|
| 170 |
+
| Spectrogram | Constant-Q transform as in `librosa.cqt` 1.0: 208 bins, 24 per octave, lowest bin four semitones below C1 (25.9565 Hz), hop 4,096 samples, default Hann window and filter scale |
|
| 171 |
+
| Values | `log1p` of the magnitude, float32 |
|
| 172 |
+
| Network input | Rows 8 to 199 of the spectrogram, shape `(batch, 192, frames)`, at least 8 frames (about 1.5 s) |
|
| 173 |
+
| Output | 24 logits. Class `k` is tonic `TONICS[k // 2]`, minor when `k` is odd: 0 is C major, 1 C minor, 2 C# major ... 23 B minor |
|
| 174 |
+
|
| 175 |
+
`drone-v2.safetensors` holds every weight and BatchNorm statistic in float32
|
| 176 |
+
under its PyTorch name, for hosts without PyTorch or ONNX Runtime. The
|
| 177 |
+
architecture is three blocks of two 3x3 convolutions (16, 32, 64 channels), each
|
| 178 |
+
convolution followed by BatchNorm and ELU, each block ending in 2x2 max pooling;
|
| 179 |
+
then an average over time only, which keeps the 24 frequency rows, and one
|
| 180 |
+
linear layer from 64 x 24 features to 24 keys. See
|
| 181 |
+
[`drone/model.py`](https://github.com/lvndry/drone/blob/main/drone/model.py).
|
| 182 |
+
Euphoria's [Rust implementation](https://github.com/lvndry/euphoria) is a
|
| 183 |
+
complete example.
|
| 184 |
+
|
| 185 |
+
### Files
|
| 186 |
+
|
| 187 |
+
| File | For |
|
| 188 |
+
|---|---|
|
| 189 |
+
| `drone-v2.pt` | The Python package and PyTorch. Includes training provenance in its metadata |
|
| 190 |
+
| `drone-v2.onnx` | ONNX Runtime in any language. Opset 18, dynamic batch and length; matches PyTorch within 2e-5 |
|
| 191 |
+
| `drone-v2.json` | The ONNX file's contracts (features, labels, input layout) |
|
| 192 |
+
| `drone-v2.safetensors` | Native implementations |
|
| 193 |
+
| `drone-v1.*` | The previous release, unchanged |
|
| 194 |
+
|
| 195 |
+
## How it was trained
|
| 196 |
+
|
| 197 |
+
- **Data:** 6,263 labelled tracks: 4,988 from FMAKv2 (Free Music Archive tracks
|
| 198 |
+
of many genres with key labels) and 1,275 Beatport dance-music previews from
|
| 199 |
+
GiantSteps-MTG-Key. The training set shares no track ID with the 604
|
| 200 |
+
GiantSteps Key previews.
|
| 201 |
+
- **Distillation:** three networks first trained on the labels alone were
|
| 202 |
+
averaged into a "teacher". drone-v2 learned half from the human labels and
|
| 203 |
+
half from the teacher's softer predictions, and also from the teacher's
|
| 204 |
+
predictions on 24,827 unlabelled 30-second clips (FMA medium).
|
| 205 |
+
- **Pitch shifting:** every training excerpt was shifted by up to 12
|
| 206 |
+
semitones either way, with its key label moved to match, so the network
|
| 207 |
+
learns every key from every song.
|
| 208 |
+
- **Recipe:** AdamW (learning rate 1e-3, weight decay 1e-4), cosine schedule,
|
| 209 |
+
120 epochs, batch 32, 60-second crops. Seed 42; the checkpoint from epoch 88
|
| 210 |
+
was chosen on a held-back 10% of the training data.
|
| 211 |
+
|
| 212 |
+
The full study, including 13 experiments and everything that did not work
|
| 213 |
+
(larger networks, more unlabelled audio, other spectrogram views, label
|
| 214 |
+
cleaning), is in
|
| 215 |
+
[`experiments/studies/frontier-distillation`](https://github.com/lvndry/drone/blob/main/experiments/studies/frontier-distillation/REPORT.md).
|
| 216 |
+
The release evaluation is in
|
| 217 |
+
[`experiments/studies/drone-v2-release`](https://github.com/lvndry/drone/blob/main/experiments/studies/drone-v2-release/REPORT.md).
|
| 218 |
+
|
| 219 |
+
## License
|
| 220 |
+
|
| 221 |
+
The weights and code are MIT-licensed. The training audio and labels keep their
|
| 222 |
+
own terms: Free Music Archive audio under each track's Creative Commons
|
| 223 |
+
licence, FMAKv2 labels under CC BY 4.0, GiantSteps+ annotations under
|
| 224 |
+
CC BY-SA 4.0, and GiantSteps-MTG-Key, whose publisher states no licence. The
|
| 225 |
+
K10 architecture follows the MIT-licensed MIREX 2026 submission.
|
drone-v2.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
drone-v2.onnx
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:d0f0f1dff52150b8c3fe1b7d70958a6fe17cdb862ba921421fecefeb4d302d0f
|
| 3 |
+
size 462056
|
drone-v2.pt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:f069fc37004062e5f7ac931dcf5f7414b2da805eadaa2c1b28b82f32eef6acb5
|
| 3 |
+
size 578449
|
drone-v2.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:f3badc870fd1f8c79fc0cabb4e1819d55e38a3b724ea0158a63d5b781765af97
|
| 3 |
+
size 441456
|