patchy-planet commited on
Commit
1cc1b05
Β·
verified Β·
1 Parent(s): 191d479

Upload feature_extractor_cpu.py with huggingface_hub

Browse files
Files changed (1) hide show
  1. feature_extractor_cpu.py +978 -0
feature_extractor_cpu.py ADDED
@@ -0,0 +1,978 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # feature_extractor_cpu.py β€” HF Spaces CPU Basic variant.
2
+ # Identical to feature_extractor.py. shifts=5 is preserved as required.
3
+ # Expected processing time on CPU Basic (2 vCPU): ~33–136 min per track
4
+ # depending on track length. This is expected β€” do not reduce shifts.
5
+ # No GPU code. No spaces.GPU decorator. Do not import the 'spaces' package.
6
+
7
+ import numpy as np
8
+ import librosa, torch, openl3, warnings
9
+ from demucs.pretrained import get_model
10
+ from demucs.apply import apply_model
11
+ from scipy.stats import entropy
12
+ from scipy.signal import butter, sosfilt, resample_poly
13
+ import pyloudnorm as pyln
14
+
15
+ warnings.filterwarnings('ignore'); EPS = 1e-10
16
+
17
+ # ─────────────────────────────────────────────────────────────────────────────
18
+ # ENG_DIMS β€” ordered names for every element of the eng_vec returned by
19
+ # extract_mir_features(). Used by lane_model.py and analyze_submissions.py
20
+ # so that dimension names and indices are never hardcoded in two places.
21
+ #
22
+ # DESIGN RATIONALE β€” SPLITTING COMPOSITE DIMENSIONS
23
+ # ──────────────────────────────────────────────────
24
+ # Every dimension here is intended to be atomic: one measurable quantity,
25
+ # one conceptual meaning. Dimensions that previously bundled distinct concepts
26
+ # have been decomposed. The affected cases are documented below.
27
+ #
28
+ # 1. vocal_prominence (old) β†’ vocal_level + vocal_presence + vocal_brightness
29
+ # The old single ratio (vocal RMS / total RMS) conflated three independent
30
+ # questions: how loud are the vocals when they sing? how much of the song
31
+ # has singing at all? how upfront/forward do the vocals sit spectrally?
32
+ # A track can have loud but infrequent vocals (high level, low presence),
33
+ # or frequent but spectrally buried vocals (high presence, low brightness).
34
+ # These are now three separate dimensions.
35
+ #
36
+ # 2. drum_punch (kept, modified in v4) + drum_prominence (new in v3)
37
+ # drum_punch is the crest factor of the drum stem gated by drum_confidence.
38
+ # drum_prominence is the drum stem's RMS share of the total mix.
39
+ #
40
+ # Before v4, drum_punch was a raw crest factor: peak(drum_stem) /
41
+ # rms(drum_stem). This produced false positives for songs with no drums:
42
+ # htdemucs leaks guitar strum transients into the drum stem, giving the
43
+ # stem a tiny RMS denominator but real peaks β†’ inflated crest factor.
44
+ # drum_prominence did not protect against this because it measures the
45
+ # stem's share of the total mix, not the stem's internal peak-to-RMS ratio.
46
+ #
47
+ # Fix (v4): drum_punch is now multiplied by drum_confidence [0,1], a
48
+ # frequency-domain drum detection score. Supporting sub-dimensions:
49
+ #
50
+ # drum_kick_presence β€” (v5, unchanged in v6) Drum stem's dominance in the
51
+ # 40–120 Hz kick band, measured as:
52
+ # RMS(drum_stem, 40–120 Hz) / RMS(full_mix, 40–120 Hz)
53
+ # When a real kick drum is present, the drum stem
54
+ # provides most of the mix's sub-bass energy (ratio
55
+ # 0.50–0.90). Guitar bleed: kick_dom β‰ˆ 0.05–0.30.
56
+ #
57
+ # drum_cymbal_presence β€” (v6 REDESIGN) HPSS-based sustained energy in the
58
+ # 4–10 kHz band of the drum stem. Measures crash and
59
+ # ride cymbal ring-out as opposed to transient hi-hat
60
+ # clicks (which go to hihat_presence below).
61
+ #
62
+ # Formula:
63
+ # sustained_frac = H_energy(4-10kHz) / total_energy(4-10kHz)
64
+ # band_prom = RMS(mag, 4-10kHz) / RMS(mag, all)
65
+ # cymbal_presence = (sustained_frac Γ— band_prom) / CYMBAL_SAT
66
+ # (clamped to [0, 1])
67
+ #
68
+ # WHY v5 dominance ratio (8-16 kHz) FAILED:
69
+ # The v5 formula = RMS(drum_stem, 8-16kHz) / RMS(mix, 8-16kHz)
70
+ # collapses when a bright guitar or synth pad adds energy
71
+ # at 8-16 kHz alongside real hi-hats. The mix denominator
72
+ # inflates while the drum numerator is unchanged, pushing the
73
+ # ratio below the lane mean even with cymbals throughout.
74
+ # This caused "cymbal presence thinner than what I feature"
75
+ # rejections for tracks verified to have hi-hats and crashes.
76
+ #
77
+ # The HPSS H component captures ring-out / shimmer β€” the
78
+ # characteristic of crashes and rides β€” regardless of what
79
+ # other sources contribute to the full mix at those frequencies.
80
+ #
81
+ # hihat_presence β€” (v6 NEW) Onset density of the HPSS percussive (P)
82
+ # component in 8–14 kHz, as a fraction of HIHAT_ONSET_SAT.
83
+ #
84
+ # Hi-hats are short transients β†’ appear in P component.
85
+ # Guitar chord rings at 8-14 kHz are sustained β†’ appear in H.
86
+ # Soft-masking by P/(mag+EPS) then zeroing non-hihat bins
87
+ # isolates percussive 8-14 kHz content. Onset detection on
88
+ # this signal counts hi-hat events per second.
89
+ #
90
+ # Guitar staccato transients at 8-14 kHz are much weaker
91
+ # than real hi-hat transients and fall below the
92
+ # onset_detect delta threshold.
93
+ #
94
+ # drum_confidence β€” (v6 updated) Composite [0,1] drum detection score:
95
+ # 0.55 Γ— clamp(kick_dom / DRUM_KICK_SAT, 0, 1)
96
+ # + 0.30 Γ— hihat_presence (already in [0,1])
97
+ # + 0.15 Γ— cymbal_presence (already in [0,1])
98
+ #
99
+ # Hi-hat weight raised vs. v5 cymbal weight (0.30 vs 0.40)
100
+ # because onset density at 8-14 kHz is near-impossible to
101
+ # trigger without real drum content. A drumless acoustic
102
+ # track cannot produce rapid regular transients in that band.
103
+ #
104
+ # drum_punch (index 2): drum_crest_raw Γ— drum_confidence (unchanged gate mechanism).
105
+ #
106
+ # 3. bass_tightness (kept) + bass_prominence (new in v3)
107
+ # Identical reasoning to drum_punch/drum_prominence above.
108
+ #
109
+ # 4. guitar_energy (old) β†’ other_prominence (renamed in v3)
110
+ # With htdemucs_6s (v8), the "other" stem is now cleaner: guitar and piano
111
+ # are extracted into dedicated stems (indices 4 and 5). other_prominence now
112
+ # reflects only synths, strings, pads, and miscellaneous non-vocal content.
113
+ #
114
+ # 5. egtr_presence + egtr_level (v8 NEW)
115
+ # Two new dimensions operating on the dedicated guitar stem from htdemucs_6s.
116
+ # See below for full rationale.
117
+ #
118
+ # RENAMES (done in v3):
119
+ # complexity β†’ harmonic_complexity (chroma entropy β‰  general complexity)
120
+ # dynamics β†’ lufs_variation (std(LUFS) β‰  all kinds of dynamics)
121
+ #
122
+ # Index history:
123
+ # Indices 0–25: identical to v3.
124
+ # Indices 26–27: added in v4, semantics updated in v5 (dominance ratio).
125
+ # Index 26: unchanged in v6 (kick dominance ratio).
126
+ # Index 27: v6 redesign β€” was 8-16kHz dominance ratio, now HPSS sustained
127
+ # fraction in 4-10kHz. INCOMPATIBLE with v5 cache entries.
128
+ # Index 28: v6 updated weights (was 0.60Γ—kick+0.40Γ—cymbal).
129
+ # Index 29: v6 NEW β€” hihat_presence.
130
+ # Indices 30–31: v8 NEW β€” egtr_presence, egtr_level.
131
+ # htdemucs switched from 4-stem (htdemucs) to 6-stem (htdemucs_6s).
132
+ # total_rms now includes guitar_rms + piano_rms (more accurate
133
+ # stem-sum denominator). All v7 cache entries are incompatible.
134
+ # ─────────────────────────────────────────────────────────────────────────────
135
+ ENG_DIMS = [
136
+ "lufs", # 0: Integrated loudness (LUFS)
137
+ "crest_factor", # 1: Peak-to-RMS ratio (global mix)
138
+ "drum_punch", # 2: Drum stem crest factor Γ— drum_confidence (gated)
139
+ "drum_prominence", # 3: Drum stem RMS / total mix RMS (mix presence)
140
+ "stereo_width", # 4: Mid-side energy ratio
141
+ "brightness", # 5: Spectral centroid of mix (Hz)
142
+ "harmonic_complexity", # 6: Chroma entropy (harmonic diversity only)
143
+ "clip_num_events", # 7: Number of separate clipping events
144
+ "clip_total", # 8: Total clipped samples
145
+ "clip_ratio", # 9: Clipped samples / total samples
146
+ "clip_max_run", # 10: Longest consecutive clipping run (samples)
147
+ "clip_mean_run", # 11: Mean clipping run length across all events
148
+ "other_prominence", # 12: Other stem (synth/pad/strings) RMS / total mix RMS
149
+ "vocal_level", # 13: Vocal stem RMS / total mix RMS (loudness when present)
150
+ "vocal_presence", # 14: Fraction of frames with active vocal signal (0–1)
151
+ "vocal_brightness", # 15: Spectral centroid of vocal stem (Hz)
152
+ "chorus_lift", # 16: Max short-term LUFS minus median (LU)
153
+ "crispness", # 17: Zero-crossing rate (high-frequency texture)
154
+ "clarity", # 18: Spectral flatness (tonal vs. noise-like character)
155
+ "length_s", # 19: Track length in seconds
156
+ "lufs_variation", # 20: Std of short-term LUFS
157
+ "peak_lufs", # 21: Max short-term LUFS
158
+ "rolloff", # 22: Spectral rolloff (Hz)
159
+ "bass_tightness", # 23: Bass stem crest factor
160
+ "bass_prominence", # 24: Bass stem RMS / total mix RMS
161
+ "phase_corr", # 25: L/R phase correlation
162
+ # ── Drum sub-dimensions (v4+, semantics per version noted) ──────────────��
163
+ "drum_kick_presence", # 26: RMS(drum stem, 40–120 Hz) / RMS(full mix, 40–120 Hz) [v5, unchanged]
164
+ "drum_cymbal_presence",# 27: HPSS sustained fraction Γ— band prominence in 4–10 kHz [v6]
165
+ "drum_confidence", # 28: 0.55Γ—kick_conf + 0.30Γ—hihat_conf + 0.15Γ—cymbal_conf [v6]
166
+ "hihat_presence", # 29: Onset density of percussive 8–14 kHz drum component [v6 NEW]
167
+ # ── Electric guitar dimensions (v8 NEW β€” htdemucs_6s guitar stem) ────────
168
+ "egtr_presence", # 30: Composite electric guitar detection score [0,1] [v8 NEW]
169
+ "egtr_level", # 31: egtr_presence Γ— (guitar_rms / total_rms) β€” guitar mix level [v8 NEW]
170
+ ]
171
+
172
+ # ── Vocal activity detection threshold ───────────────────────────────────────
173
+ # Frames whose RMS exceeds this value are counted as active vocal frames.
174
+ # 0.01 linear β‰ˆ -40 dBFS.
175
+ VOCAL_ACTIVITY_THRESHOLD = 0.01
176
+
177
+ # ── Drum band analysis constants (v6) ────────────────────────────────────────
178
+ #
179
+ # KICK (dominance ratio, unchanged from v5):
180
+ # DRUM_KICK_SAT = 0.55
181
+ # The kick dominance ratio at which kick_conf saturates at 1.0. Real kick
182
+ # drum: drum stem provides 50–90 % of mix kick-band energy β†’ conf = 1.0.
183
+ # Acoustic guitar bleed: kick_dom β‰ˆ 0.05–0.30 β†’ conf ≀ 0.55.
184
+ #
185
+ # CYMBAL (HPSS sustained fraction, 4–10 kHz, v6):
186
+ # CYMBAL_BAND_FMIN / FMAX = 4000–10000 Hz
187
+ # Crash and ride cymbal fundamental energy. Crashes ring in 3–8 kHz;
188
+ # rides shimmer in 5–12 kHz. 4-10 kHz captures the core sustained content
189
+ # of both without extending into the hi-hat domain (8-14 kHz).
190
+ # CYMBAL_SAT = 0.20
191
+ # Value of (sustained_frac Γ— band_prom) at which cymbal_presence = 1.0.
192
+ # Estimated: crash-heavy track β†’ sustained_frac β‰ˆ 0.60, band_prom β‰ˆ 0.35
193
+ # β†’ product β‰ˆ 0.21. Kick+snare only β†’ product β‰ˆ 0.05–0.10.
194
+ # Tune up if lane refs with strong crashes score < 0.70; tune down if
195
+ # kick-only tracks score > 0.40.
196
+ #
197
+ # HI-HAT (HPSS onset density, 8–14 kHz, v6 NEW):
198
+ # HIHAT_BAND_FMIN / FMAX = 8000–14000 Hz
199
+ # Closed and open hi-hat fundamental energy. Concentrated above 8 kHz;
200
+ # stopping at 14 kHz avoids excessive noise-floor sensitivity at the
201
+ # top of the drum stem frequency range.
202
+ # HIHAT_ONSET_SAT = 5.0 (onsets/sec)
203
+ # Onset rate at which hihat_presence = 1.0. 16th notes at 75 bpm = 5/sec;
204
+ # 8th notes at 150 bpm = 5/sec. Most steady hi-hat patterns in rock/pop
205
+ # fall in the 3–8 onsets/sec range. Tune down if lane refs with active
206
+ # hi-hats score < 0.60; tune up if drumless tracks score > 0.15.
207
+ #
208
+ # DRUM CONFIDENCE WEIGHTS (v6):
209
+ # DRUM_KICK_WEIGHT = 0.55 (was 0.60 in v5)
210
+ # DRUM_HIHAT_WEIGHT = 0.30 (new; replaces cymbal_weight contribution)
211
+ # DRUM_CYMBAL_WEIGHT = 0.15 (new; crash/ride as secondary gate input)
212
+ # Sum = 1.0 β€” fully normalized confidence score.
213
+ #
214
+ # Hi-hat raised to 0.30: onset density at 8-14 kHz is nearly impossible to
215
+ # produce without a real drum kit. Acoustic guitars, hand percussion, and
216
+ # electronic pads without a dedicated hi-hat pattern will all score near 0.
217
+ # This makes drum_confidence a much stronger gate against false positives.
218
+ #
219
+ DRUM_KICK_FMIN = 40.0 # Hz β€” kick fundamental lower bound
220
+ DRUM_KICK_FMAX = 120.0 # Hz β€” kick fundamental upper bound
221
+ DRUM_KICK_SAT = 0.55 # kick dominance ratio at kick_conf = 1.0
222
+
223
+ CYMBAL_BAND_FMIN = 4000.0 # Hz β€” crash/ride lower bound
224
+ CYMBAL_BAND_FMAX = 10000.0 # Hz β€” crash/ride upper bound
225
+ CYMBAL_SAT = 0.20 # (sustained_frac Γ— band_prom) at cymbal_presence = 1.0
226
+
227
+ HIHAT_BAND_FMIN = 8000.0 # Hz β€” hi-hat lower bound
228
+ HIHAT_BAND_FMAX = 14000.0 # Hz β€” hi-hat upper bound
229
+ HIHAT_ONSET_SAT = 5.0 # onsets/sec at hihat_presence = 1.0
230
+
231
+ DRUM_KICK_WEIGHT = 0.55 # weight for kick_conf in drum_confidence
232
+ DRUM_HIHAT_WEIGHT = 0.30 # weight for hihat_presence in drum_confidence
233
+ DRUM_CYMBAL_WEIGHT = 0.15 # weight for cymbal_presence in drum_confidence
234
+
235
+ # ── Electric guitar detection constants (v8) ──────────────────────────────────
236
+ #
237
+ # EGTR_MID_FMIN / FMAX = 200–5000 Hz
238
+ # Core range of electric guitar fundamentals and dominant harmonics.
239
+ # Low E string = 82 Hz; high E string = 1318 Hz. Upper harmonics and
240
+ # distortion products extend to 5 kHz. Starting at 200 Hz avoids bleed
241
+ # from the bass stem into the guitar stem. Stopping at 5 kHz keeps the
242
+ # measure out of the hi-hat / pick-noise band (handled by drum dims).
243
+ #
244
+ # EGTR_PICK_ONSET_SAT = 4.0 (onsets/sec)
245
+ # Onset rate at which picking_density = 1.0. 8th notes at 120 bpm = 4/sec;
246
+ # 16th notes at 60 bpm = 4/sec. Steady rhythm guitar in rock/pop lands
247
+ # in the 2–6 onsets/sec range.
248
+ # Tune DOWN if lane refs with active rhythm guitar score < 0.50.
249
+ # Tune UP if piano-only or pad-heavy tracks score above 0.30.
250
+ #
251
+ # EGTR_FLATNESS_SAT = 0.25
252
+ # Mid-band spectral flatness at which mid_saturation = 1.0.
253
+ # Clean electric guitar: 0.05–0.15. Crunchy/overdriven: 0.10–0.30.
254
+ # Full distortion lead: 0.20–0.40. Piano: 0.02–0.08. Acoustic: 0.03–0.12.
255
+ # 0.25 is intentionally generous β€” rewards any electric guitar content,
256
+ # not only distorted playing. Clean electric guitar (Fender clean tone)
257
+ # will score low here; such tracks still score via picking_density.
258
+ #
259
+ # EGTR_PICK_WEIGHT = 0.65 β€” primary gate: rhythmic attack events
260
+ # EGTR_FLAT_WEIGHT = 0.35 β€” secondary: harmonic saturation (rewards dirt)
261
+ #
262
+ # TUNING GUIDANCE (post first training run):
263
+ # Check the printed standards table for egtr_presence and egtr_level.
264
+ # Lane refs with active electric guitar: egtr_presence β‰₯ 0.50 (ideally 0.60–0.90)
265
+ # Non-lane refs without guitar: egtr_presence ≀ 0.20
266
+ # If guitar tracks score 0.20–0.35: lower EGTR_PICK_ONSET_SAT (4.0 β†’ 2.5)
267
+ # and/or lower EGTR_FLATNESS_SAT (0.25 β†’ 0.15).
268
+ # If piano/pad tracks score above 0.30: raise EGTR_PICK_ONSET_SAT or
269
+ # reduce EGTR_FLAT_WEIGHT to down-weight the flatness component.
270
+ #
271
+ EGTR_MID_FMIN = 200.0 # Hz β€” guitar mid-band lower bound
272
+ EGTR_MID_FMAX = 5000.0 # Hz β€” guitar mid-band upper bound
273
+ EGTR_PICK_ONSET_SAT = 4.0 # onsets/sec at picking_density = 1.0
274
+ EGTR_FLATNESS_SAT = 0.25 # flatness value at mid_saturation = 1.0
275
+ EGTR_PICK_WEIGHT = 0.65 # weight for picking_density in egtr_presence
276
+ EGTR_FLAT_WEIGHT = 0.35 # weight for mid_saturation in egtr_presence
277
+
278
+
279
+ device = "cuda" if torch.cuda.is_available() else "cpu"
280
+ demucs_model = get_model('htdemucs_6s') # v8: 6-stem model (adds guitar + piano stems)
281
+ demucs_model.to(device)
282
+
283
+ # htdemucs_6s stem order: 0=drums, 1=bass, 2=other, 3=vocals, 4=guitar, 5=piano
284
+ _STEM_DRUMS = 0
285
+ _STEM_BASS = 1
286
+ _STEM_OTHER = 2
287
+ _STEM_VOCALS = 3
288
+ _STEM_GUITAR = 4 # v8 NEW β€” dedicated electric/acoustic guitar stem
289
+ _STEM_PIANO = 5 # v8 NEW β€” dedicated piano stem
290
+
291
+
292
+ def get_stems(path):
293
+ wav, _ = librosa.load(path, sr=demucs_model.samplerate, mono=False)
294
+ if wav.ndim == 1: wav = np.stack([wav, wav])
295
+ ref = torch.from_numpy(wav).to(device)
296
+ ref_std = ref.std() + EPS
297
+ ref = (ref - ref.mean()) / ref_std
298
+ with torch.no_grad():
299
+ sources = apply_model(demucs_model, ref.unsqueeze(0), shifts=5, split=True)[0]
300
+ return (sources * ref_std.cpu().item()).cpu().numpy()
301
+
302
+
303
+ def _measure_clipping(y, threshold=0.999):
304
+ """
305
+ Returns all five clipping dimensions as individual floats.
306
+
307
+ threshold=0.999 (~-0.009 dBFS) avoids flagging legitimately loud
308
+ transient peaks that just touch near-full-scale without flat-topping.
309
+
310
+ Returns: (num_events, total_samples, ratio, max_run, mean_run)
311
+ """
312
+ clipped = np.abs(y) >= threshold
313
+ total = int(np.sum(clipped))
314
+ if total == 0:
315
+ return 0.0, 0.0, 0.0, 0.0, 0.0
316
+
317
+ padded = np.concatenate(([False], clipped, [False]))
318
+ diff = np.diff(padded.astype(int))
319
+ run_starts = np.where(diff == 1)[0]
320
+ run_ends = np.where(diff == -1)[0]
321
+ run_lengths = run_ends - run_starts
322
+
323
+ return (
324
+ float(len(run_lengths)),
325
+ float(total),
326
+ float(total / len(y)),
327
+ float(np.max(run_lengths)),
328
+ float(np.mean(run_lengths)),
329
+ )
330
+
331
+
332
+ def _band_rms_abs(y: np.ndarray, sr: int, fmin: float, fmax: float) -> float:
333
+ """
334
+ Absolute RMS of y in the [fmin, fmax] Hz band (not normalized).
335
+ Used by _compute_drum_bands for kick dominance ratio.
336
+ Returns 0.0 if signal is silent or band is invalid.
337
+ """
338
+ nyq = sr / 2.0
339
+ lo = max(fmin, 1.0)
340
+ hi = min(fmax, nyq - 1.0)
341
+ if lo >= hi:
342
+ return 0.0
343
+ sos = butter(2, [lo, hi], btype='bandpass', fs=sr, output='sos')
344
+ filtered = sosfilt(sos, y)
345
+ return float(np.sqrt(np.mean(filtered**2)))
346
+
347
+
348
+ # ─────────────────────────────────────────────────────────────────────────────
349
+ # ITU-R BS.1770-4 / BS.1775-2 LOUDNESS AND TRUE PEAK IMPLEMENTATION
350
+ #
351
+ # WHY THE OLD FORMULA WAS WRONG
352
+ # ───────────────────────────────
353
+ # The original approximation used:
354
+ # lufs β‰ˆ -0.691 + 10 Γ— log10(mean(yΒ²))
355
+ # This is RMS-based power in dB, not integrated LUFS. It is missing two
356
+ # mandatory stages defined in ITU-R BS.1770:
357
+ #
358
+ # 1. K-weighting pre-filter β€” a two-stage IIR filter that:
359
+ # Stage 1: high-shelf boost (+4 dB at high frequencies) modelling
360
+ # the acoustic effect of the listener's head on a microphone.
361
+ # Stage 2: high-pass filter (RLB weighting) removing DC and sub-20 Hz
362
+ # content that inflates RMS but carries no perceived loudness.
363
+ # Together these raise typical music by +1 to +5 dB before squaring.
364
+ #
365
+ # 2. Gating β€” BS.1770-4 integrates only "active" blocks:
366
+ # Absolute gate: blocks below -70 LUFS excluded (silence/noise floor).
367
+ # Relative gate: second pass excludes blocks more than -10 LU below
368
+ # the ungated mean (rejects very quiet passages).
369
+ # Without gating, quiet intros/outros and noise floors pull the
370
+ # integrated value down, causing systematic under-reading of 2–5 dB.
371
+ #
372
+ # PRACTICAL IMPACT ON THIS SYSTEM
373
+ # ────────────────────────────────
374
+ # Error: 2–4 dB under-reading of integrated LUFS, varying by program material.
375
+ # β€’ The Spotify normalization gain calculation was off by that same margin.
376
+ # β€’ Lane engineering standards for lufs and peak_lufs were calibrated against
377
+ # the wrong values (consistently wrong, so relative comparisons partially
378
+ # survived β€” but absolute statements about headroom were incorrect).
379
+ # β€’ short-term LUFS (used for peak_lufs, lufs_variation, chorus_lift) had the
380
+ # same error AND lacked K-weighting, so spectral character affected the
381
+ # "loudness" reading in a non-perceptual way.
382
+ # β€’ true_peak_dbfs was never returned by the modal worker, making the
383
+ # tp_headroom constraint in apply_streaming_normalization() dead code.
384
+ #
385
+ # FIX SUMMARY (v7)
386
+ # ────────────────
387
+ # _compute_integrated_lufs() β€” pyloudnorm Meter.integrated_loudness()
388
+ # Full BS.1770-4: K-weighting + gating.
389
+ # _compute_short_term_lufs_array() β€” K-weighted sliding 3s/1s windows, no gate.
390
+ # Same K-weighting as integrated. Replaces the
391
+ # raw librosa.feature.rms()-based st_lufs blocks.
392
+ # _true_peak_dbfs() β€” 4Γ— polyphase oversampling (BS.1775-2) to
393
+ # detect intersample peaks. Returned as third
394
+ # element of extract_mir_features() so the modal
395
+ # worker can include it in the response dict and
396
+ # apply_streaming_normalization() can enforce the
397
+ # -1.0 dBFS true peak ceiling correctly.
398
+ #
399
+ # NOTE: pyloudnorm must be installed: pip install pyloudnorm
400
+ # ─────────────────────────────────────────────────────────────────────────────
401
+
402
+ def _kweight_sos(sr: int) -> np.ndarray:
403
+ """
404
+ BS.1770 K-weighting filter as a stacked SOS array for scipy.signal.sosfilt.
405
+
406
+ Computed directly from the analog prototype parameters in BS.1770 Annex 1
407
+ using the Audio EQ Cookbook (Bristow-Johnson) bilinear transform formulas.
408
+ Does NOT touch pyloudnorm internals β€” meter._filters internal structure
409
+ has changed between pyloudnorm versions and is not part of its public API.
410
+
411
+ Two cascaded stages:
412
+ Stage 1 β€” Pre-filter (high-shelf, +4 dB at high frequencies):
413
+ Models the free-field to diffuse-field acoustic correction for the
414
+ effect of the listener's head on a mounted microphone. Raises
415
+ high-frequency energy before mean-square integration so that HF
416
+ content is weighted appropriately by the loudness measure.
417
+ Parameters: f0 = 1500 Hz, dBgain = +4 dB, Q = 1/sqrt(2)
418
+
419
+ Stage 2 β€” RLB weighting (2nd-order Butterworth high-pass, ~38.1 Hz):
420
+ Removes DC and very low-frequency energy that carries no perceptual
421
+ loudness but would otherwise inflate the mean-square sum.
422
+ Parameters: fc = 38.13494637408312 Hz
423
+
424
+ Returns a (2, 6) ndarray β€” two SOS sections β€” ready for sosfilt().
425
+ """
426
+ # Stage 1: Pre-filter high-shelf (Audio EQ Cookbook high-shelf formula)
427
+ f0 = 1500.0 # Hz β€” shelf midpoint frequency
428
+ dBgain = 4.0 # dB β€” shelf gain
429
+ Q = 1.0 / np.sqrt(2) # = 0.7071... (Butterworth Q)
430
+
431
+ A = 10.0 ** (dBgain / 40.0) # sqrt of linear power gain
432
+ w0 = 2.0 * np.pi * f0 / sr # digital frequency (radians/sample)
433
+ alpha = np.sin(w0) / (2.0 * Q) # bandwidth term
434
+
435
+ # High-shelf numerator / denominator (Audio EQ Cookbook, high-shelf)
436
+ b0 = A * ((A + 1) + (A - 1) * np.cos(w0) + 2 * np.sqrt(A) * alpha)
437
+ b1 = -2 * A * ((A - 1) + (A + 1) * np.cos(w0))
438
+ b2 = A * ((A + 1) + (A - 1) * np.cos(w0) - 2 * np.sqrt(A) * alpha)
439
+ a0 = ((A + 1) - (A - 1) * np.cos(w0) + 2 * np.sqrt(A) * alpha)
440
+ a1 = 2 * ((A - 1) - (A + 1) * np.cos(w0))
441
+ a2 = ((A + 1) - (A - 1) * np.cos(w0) - 2 * np.sqrt(A) * alpha)
442
+
443
+ # Second-order section: [b0/a0, b1/a0, b2/a0, 1.0, a1/a0, a2/a0]
444
+ sos_shelf = np.array([[b0/a0, b1/a0, b2/a0, 1.0, a1/a0, a2/a0]])
445
+
446
+ # Stage 2: RLB high-pass (2nd-order Butterworth)
447
+ fc = 38.13494637408312 # Hz
448
+ sos_hp = butter(2, fc / (sr / 2.0), btype='high', output='sos')
449
+
450
+ return np.vstack([sos_shelf, sos_hp]) # shape (2, 6)
451
+
452
+
453
+ def _compute_integrated_lufs(y_stereo: np.ndarray, sr: int) -> float:
454
+ """
455
+ ITU-R BS.1770-4 integrated LUFS (K-weighted, gated).
456
+
457
+ y_stereo: (2, n_samples) stereo array at sr Hz.
458
+ Returns integrated LUFS as float; returns -70.0 for silent / fully-gated
459
+ signals (pyloudnorm returns -inf in those cases).
460
+ """
461
+ meter = pyln.Meter(sr)
462
+ data = y_stereo.T.astype(np.float64) # pyloudnorm expects (n_samples, n_ch)
463
+ result = meter.integrated_loudness(data)
464
+ return float(result) if np.isfinite(result) else -70.0
465
+
466
+
467
+ def _compute_short_term_lufs_array(y_stereo: np.ndarray, sr: int,
468
+ window_s: float = 3.0,
469
+ hop_s: float = 1.0) -> np.ndarray:
470
+ """
471
+ K-weighted short-term loudness in sliding windows (BS.1770, no gating).
472
+
473
+ Applies the same K-weighting filters as integrated LUFS but without the
474
+ absolute/relative gating step, matching the BS.1770 short-term definition.
475
+ Uses the BS.1770 stereo channel-sum formula (G_L = G_R = 1.0):
476
+ L_window = -0.691 + 10 Γ— log10(mean_sq_L + mean_sq_R)
477
+
478
+ window_s = 3.0, hop_s = 1.0 matches the original librosa.feature.rms()
479
+ block framing that peak_lufs, lufs_variation, and chorus_lift are based on.
480
+
481
+ y_stereo: (2, n_samples) stereo array at sr Hz.
482
+ Returns array of per-window LUFS values (float64).
483
+ """
484
+ # Apply K-weighting (BS.1770 pre-filter + RLB high-pass) to each channel.
485
+ # _kweight_sos() returns a single stacked (2, 6) SOS array computed
486
+ # directly from the BS.1770 analog prototype β€” no pyloudnorm internals used.
487
+ sos = _kweight_sos(sr)
488
+ data = y_stereo.astype(np.float64).copy() # (2, n_samples)
489
+ for ch in range(data.shape[0]):
490
+ data[ch] = sosfilt(sos, data[ch])
491
+
492
+ n_samples = data.shape[1]
493
+ win_len = int(window_s * sr)
494
+ hop_len = max(int(hop_s * sr), 1)
495
+
496
+ results = []
497
+ for start in range(0, max(n_samples - win_len + 1, 1), hop_len):
498
+ block = data[:, start : start + win_len] # (2, win_len)
499
+ ms_sum = float(np.sum(np.mean(block ** 2, axis=1))) # sum over L, R
500
+ results.append(-0.691 + 10.0 * np.log10(ms_sum + EPS))
501
+
502
+ return np.array(results, dtype=np.float64) if results else np.array([-70.0])
503
+
504
+
505
+ def _true_peak_dbfs(y_stereo: np.ndarray) -> float:
506
+ """
507
+ ITU-R BS.1775-2 true peak level (dBFS), measured via 4Γ— oversampling.
508
+
509
+ Standard sample-domain peak measurement misses intersample peaks β€” peaks
510
+ that occur between samples and can exceed the sample-domain maximum after
511
+ reconstruction. 4Γ— oversampling with a polyphase anti-aliasing filter
512
+ (scipy.signal.resample_poly) captures these peaks reliably.
513
+
514
+ Spotify, Apple Music, YouTube, and Tidal all measure true peak at β‰₯ 4Γ—
515
+ oversampling when deciding whether and by how much to apply limiting during
516
+ normalization. -1.0 dBFS TP is the standard limiting ceiling for most DSPs.
517
+
518
+ y_stereo: (2, n_samples) stereo or (n_samples,) mono array.
519
+ Returns peak in dBFS. Typical brick-wall limited master: -0.5 to -0.1 dBFS.
520
+ Silent signal: returns -120.0.
521
+ """
522
+ oversampled = resample_poly(y_stereo, 4, 1)
523
+ peak_linear = float(np.max(np.abs(oversampled)))
524
+ if peak_linear < EPS:
525
+ return -120.0
526
+ return float(20.0 * np.log10(peak_linear))
527
+
528
+
529
+ # ─────────────────────────────────────────────────────────────────────────────
530
+ # CANONICAL STREAMING NORMALIZATION
531
+ #
532
+ # Single source of truth used by both lane_model.py (training) and
533
+ # analyze_submissions.py (inference). Having one implementation guarantees
534
+ # that training and scoring apply identical gain logic β€” any future change to
535
+ # normalization behaviour only needs to be made in one place.
536
+ #
537
+ # Previously duplicated as:
538
+ # _norm_eng() in lane_model.py (training)
539
+ # apply_streaming_normalization() in analyze_submissions.py (inference)
540
+ # Those two copies were kept manually in sync. They are now replaced by this
541
+ # single function, imported by both callers.
542
+ # ─────────────────────────────────────────────────────────────────────────────
543
+
544
+ def apply_streaming_normalization(
545
+ eng: np.ndarray,
546
+ target_lufs: float,
547
+ true_peak_dbfs: "float | None" = None,
548
+ ) -> "tuple[np.ndarray, float]":
549
+ """
550
+ Simulate streaming-platform loudness normalization on an eng vector.
551
+
552
+ Computes the gain the platform would apply to reach target_lufs, then
553
+ shifts the lufs and peak_lufs dimensions by that gain.
554
+
555
+ Only attenuates: gain is capped at 0.0 dB (platforms never boost quiet
556
+ masters β€” Spotify, Apple Music, etc. leave them quieter than the target).
557
+
558
+ true_peak_dbfs (BS.1775-2, 4Γ— oversampled):
559
+ If provided and finite, the gain is also capped so that true peak
560
+ never exceeds -1.0 dBFS post-normalization, matching the hardware
561
+ limiter ceiling used by most streaming DSPs. This cap only activates
562
+ for masters with intersample peaks above -1.0 dBFS. When None or
563
+ non-finite, the cap is skipped (equivalent to tp_headroom = 0.0,
564
+ which is harmless because the 0.0 attenuation cap already applies).
565
+
566
+ Raises ValueError if "lufs" or "peak_lufs" are absent from ENG_DIMS.
567
+ Callers must treat this as a fatal configuration error β€” it means
568
+ ENG_DIMS was modified without updating this function.
569
+
570
+ Parameters
571
+ ----------
572
+ eng : np.ndarray
573
+ Raw engineering feature vector (32 elements, pre-normalization).
574
+ target_lufs : float
575
+ Streaming loudness target in LUFS (e.g. -14.0 for Spotify).
576
+ true_peak_dbfs : float | None
577
+ BS.1775-2 true peak level of the master in dBFS, or None if
578
+ unavailable (true peak cap is skipped when None).
579
+
580
+ Returns
581
+ -------
582
+ eng_normed : np.ndarray
583
+ Copy of eng with lufs and peak_lufs shifted by gain_db.
584
+ gain_db : float
585
+ Gain applied (always <= 0.0).
586
+ """
587
+ try:
588
+ lufs_idx = ENG_DIMS.index("lufs")
589
+ peak_idx = ENG_DIMS.index("peak_lufs")
590
+ except ValueError as exc:
591
+ raise ValueError(
592
+ f"apply_streaming_normalization: required dimension not found in ENG_DIMS: {exc}"
593
+ ) from exc
594
+
595
+ # True peak cap: only active when true_peak_dbfs is provided and finite.
596
+ # When true_peak > -1.0 dBFS (intersample peak above limiter ceiling),
597
+ # tp_headroom is negative, further restricting the gain so the normalized
598
+ # true peak lands at -1.0 dBFS. When true_peak <= -1.0 dBFS, tp_headroom
599
+ # is non-negative and the existing 0.0 cap is the binding constraint.
600
+ tp_headroom = (
601
+ (-1.0 - true_peak_dbfs)
602
+ if (true_peak_dbfs is not None and np.isfinite(true_peak_dbfs))
603
+ else 0.0
604
+ )
605
+ gain_db = min(target_lufs - eng[lufs_idx], tp_headroom, 0.0)
606
+ out = eng.copy()
607
+ out[lufs_idx] += gain_db
608
+ out[peak_idx] += gain_db
609
+ return out, gain_db
610
+
611
+
612
+ def _compute_drum_bands(drum_mono: np.ndarray, mix_mono: np.ndarray, sr: int) -> tuple:
613
+ """
614
+ Returns (kick_dom, cymbal_presence, hihat_presence).
615
+
616
+ kick_dom β€” dominance ratio (unchanged from v5):
617
+ RMS(drum_stem, 40–120 Hz) / RMS(mix, 40–120 Hz)
618
+ cymbal_presence β€” HPSS sustained fraction Γ— band prominence in 4–10 kHz [0,1]
619
+ hihat_presence β€” onset density of HPSS percussive 8–14 kHz component [0,1]
620
+
621
+ WHY HPSS INSTEAD OF DOMINANCE RATIO FOR CYMBAL / HI-HAT (v6):
622
+
623
+ The v5 formula drum_cymbal_presence = RMS(drum_8-16kHz) / RMS(mix_8-16kHz)
624
+ conflated hi-hats and crashes and failed in mixed-source tracks. When a bright
625
+ guitar or synth pad adds energy at 8-16 kHz alongside real drum content, the
626
+ mix denominator inflates while the drum numerator is unchanged β€” the ratio drops
627
+ below the lane mean even with cymbals audibly present throughout the track.
628
+
629
+ HPSS (Harmonic-Percussive Source Separation, librosa.decompose.hpss) applies
630
+ a horizontal median filter to the magnitude spectrogram to extract sustained
631
+ tonal content (H) and a vertical median filter for transient broadband content (P):
632
+
633
+ H (harmonic/sustained): crash ring-out, ride shimmer, guitar chord sustain.
634
+ P (percussive/transient): hi-hat clicks, kick/snare attacks.
635
+
636
+ The H component is not affected by other instruments' level in the full mix β€”
637
+ it measures what is sustained within the drum stem's own spectrogram. A crash
638
+ ring-out is always captured in H regardless of guitar/synth in the full mix.
639
+
640
+ CYMBAL (crash/ride): measured via sustained fraction Γ— band prominence.
641
+ sustained_frac = H_energy(4-10kHz) / total_energy(4-10kHz)
642
+ β†’ high when crash ring-out dominates 4-10kHz drum content
643
+ β†’ lower for snare-only (snare ring has less sustained relative energy in band)
644
+ band_prom = RMS(mag_4-10kHz) / RMS(mag_all_freqs)
645
+ β†’ scales by how much 4-10kHz energy the drum stem actually carries
646
+ β†’ near-silent drum stem (no drums) β†’ band_prom β‰ˆ 0 β†’ product β‰ˆ 0
647
+ Combined: cymbal_presence = (sustained_frac Γ— band_prom) / CYMBAL_SAT
648
+
649
+ HI-HAT: measured via onset density of percussive 8-14 kHz signal.
650
+ 1. Apply soft percussive mask P/(mag+EPS) to complex STFT D.
651
+ 2. Zero all frequency bins outside 8-14 kHz.
652
+ 3. Reconstruct time-domain hi-hat signal via iSTFT.
653
+ 4. Run librosa onset detection on reconstructed signal.
654
+ 5. onset_rate (onsets/sec) / HIHAT_ONSET_SAT β†’ [0,1].
655
+
656
+ Guitar chord rings at 8-14 kHz are sustained β†’ H component β†’ soft_p low
657
+ for those bins β†’ soft-masked out. Guitar staccato transients at 8-14 kHz
658
+ are much weaker than real hi-hat transients β†’ below onset_detect threshold.
659
+ """
660
+ # ── Kick dominance (unchanged from v5) ────────────────────────────────────
661
+ kick_mix = _band_rms_abs(mix_mono, sr, DRUM_KICK_FMIN, DRUM_KICK_FMAX)
662
+ kick_drum = _band_rms_abs(drum_mono, sr, DRUM_KICK_FMIN, DRUM_KICK_FMAX)
663
+ kick_dom = float(min(kick_drum / (kick_mix + EPS), 1.0)) if kick_mix >= EPS else 0.0
664
+
665
+ # ── STFT + HPSS on drum stem ──────────────────────────────────────────────
666
+ n_fft = 2048
667
+ hop = 512
668
+ D = librosa.stft(drum_mono, n_fft=n_fft, hop_length=hop) # complex (1025, n_frames)
669
+ mag = np.abs(D) # magnitude spectrogram
670
+ H, P = librosa.decompose.hpss(mag, kernel_size=31, power=2.0) # H=sustained, P=transient
671
+ freqs = librosa.fft_frequencies(sr=sr, n_fft=n_fft) # shape (1025,)
672
+
673
+ # ── Cymbal presence: HPSS sustained energy in 4–10 kHz ───────────────────
674
+ cymbal_mask = (freqs >= CYMBAL_BAND_FMIN) & (freqs <= CYMBAL_BAND_FMAX)
675
+ H_cym = H[cymbal_mask, :] # sustained component in cymbal band
676
+ mag_cym = mag[cymbal_mask, :] # total magnitude in cymbal band
677
+
678
+ H_cym_energy = float(np.mean(H_cym ** 2))
679
+ band_energy = float(np.mean(mag_cym ** 2)) + EPS
680
+ sustained_frac = H_cym_energy / band_energy # fraction of cymbal-band energy that is sustained
681
+
682
+ band_rms = float(np.sqrt(np.mean(mag_cym ** 2)))
683
+ total_mag_rms = float(np.sqrt(np.mean(mag ** 2))) + EPS
684
+ band_prom = band_rms / total_mag_rms # cymbal band's prominence in overall drum stem
685
+
686
+ cymbal_presence = float(min((sustained_frac * band_prom) / CYMBAL_SAT, 1.0))
687
+
688
+ # ── Hi-hat presence: onset density of percussive 8–14 kHz component ──────
689
+ hihat_mask = (freqs >= HIHAT_BAND_FMIN) & (freqs <= HIHAT_BAND_FMAX)
690
+ # Soft-mask: amplify percussive content, suppress sustained content per bin.
691
+ # soft_p[f,t] = P[f,t] / (mag[f,t] + EPS) ∈ [0,1] β€” how percussive each bin is.
692
+ soft_p = P / (mag + EPS)
693
+ D_hihat = D * soft_p # complex STFT weighted toward percussive content
694
+ D_hihat[~hihat_mask, :] = 0.0 # zero non-hihat frequencies
695
+ # iSTFT reconstructs time-domain signal containing only percussive 8-14 kHz content.
696
+ hihat_sig = librosa.istft(D_hihat, hop_length=hop, length=len(drum_mono))
697
+
698
+ oenv = librosa.onset.onset_strength(y=hihat_sig, sr=sr, hop_length=hop)
699
+ onsets = librosa.onset.onset_detect(onset_envelope=oenv, sr=sr, hop_length=hop)
700
+ duration = len(drum_mono) / sr
701
+ onset_rate = len(onsets) / max(duration, 1.0)
702
+ hihat_presence = float(min(onset_rate / HIHAT_ONSET_SAT, 1.0))
703
+
704
+ return kick_dom, cymbal_presence, hihat_presence
705
+
706
+
707
+ def _compute_guitar_dims(guitar_mono: np.ndarray, sr: int) -> tuple:
708
+ """
709
+ Returns (egtr_presence, picking_density, mid_saturation).
710
+ All values in [0, 1].
711
+
712
+ egtr_presence β€” composite [0,1] electric guitar detection score:
713
+ EGTR_PICK_WEIGHT Γ— picking_density
714
+ + EGTR_FLAT_WEIGHT Γ— mid_saturation
715
+
716
+ picking_density β€” onset density of the HPSS percussive component in
717
+ 200–5000 Hz, normalized to EGTR_PICK_ONSET_SAT.
718
+ Detects pick/strum attack events per second.
719
+ Electric guitar: 2–6 onsets/sec. Sustained pads: 0–0.5.
720
+ Piano also produces picking onsets; the mid_saturation
721
+ gate separates piano (low flatness) from guitar (higher).
722
+
723
+ mid_saturation β€” spectral flatness of the guitar stem in 200–5000 Hz,
724
+ normalized to EGTR_FLATNESS_SAT.
725
+ Electric guitar (any gain): 0.05–0.40. Piano: 0.02–0.08.
726
+ Acoustic guitar (clean): 0.03–0.12.
727
+ Rewards harmonic saturation from magnetic pickups and
728
+ overdrive β€” the primary acoustic difference between
729
+ electric and acoustic guitar. Clean electric guitar
730
+ (e.g. Fender clean tone) scores low here; such tracks
731
+ still contribute via picking_density.
732
+
733
+ WHY HPSS ON GUITAR STEM (v8):
734
+ Even with htdemucs_6s providing a dedicated guitar stem, there is residual
735
+ bleed from other instruments. HPSS separates the percussive (transient
736
+ attack) component from the harmonic (sustained) component within the guitar
737
+ stem itself. Extracting the P component in 200-5000 Hz and counting onsets
738
+ isolates picking/strumming events even with stem bleed, because instrument
739
+ bleed (synths, pads, vocals) tends to be sustained (β†’ H component) rather
740
+ than transient (β†’ P component) in this frequency range.
741
+
742
+ ACOUSTIC VS ELECTRIC LIMITATION:
743
+ Clean acoustic and clean electric guitar will score similarly on both
744
+ sub-dimensions. The only reliable further discriminators without a
745
+ secondary model are:
746
+ 1. Body resonance: acoustic guitar has a pronounced 150–400 Hz peak
747
+ in the guitar stem. This could be added as a third sub-dimension
748
+ in a future version.
749
+ 2. CLAP embedding: already in the feature vector; the SVM implicitly
750
+ leverages CLAP's semantic understanding of guitar timbre. If your
751
+ lane refs consistently contain electric guitar and non_lane_refs
752
+ contain acoustic-only tracks, the SVM learns the residual difference
753
+ from CLAP + other dimensions even if egtr_presence alone cannot
754
+ fully separate them.
755
+
756
+ Intermediate values (picking_density, mid_saturation) are returned for
757
+ debug/tuning but are NOT added to ENG_DIMS β€” only egtr_presence and
758
+ egtr_level are scoreable dimensions, preserving the atomicity principle.
759
+ """
760
+ if np.sqrt(np.mean(guitar_mono ** 2)) < EPS:
761
+ # Silent guitar stem β€” no guitar in the track.
762
+ return 0.0, 0.0, 0.0
763
+
764
+ n_fft = 2048
765
+ hop = 512
766
+ D = librosa.stft(guitar_mono, n_fft=n_fft, hop_length=hop)
767
+ mag = np.abs(D)
768
+ H, P = librosa.decompose.hpss(mag, kernel_size=31, power=2.0)
769
+ freqs = librosa.fft_frequencies(sr=sr, n_fft=n_fft)
770
+ mid_mask = (freqs >= EGTR_MID_FMIN) & (freqs <= EGTR_MID_FMAX)
771
+
772
+ # ── Picking density: onset rate of percussive mid-band content ────────────
773
+ # Soft-mask the complex STFT toward percussive content, isolate mid band.
774
+ soft_p = P / (mag + EPS) # per-bin percussiveness score [0,1]
775
+ D_pick = D * soft_p # complex STFT weighted toward transients
776
+ D_pick[~mid_mask, :] = 0.0 # zero bins outside guitar mid band
777
+ pick_sig = librosa.istft(D_pick, hop_length=hop, length=len(guitar_mono))
778
+ oenv = librosa.onset.onset_strength(y=pick_sig, sr=sr, hop_length=hop)
779
+ onsets = librosa.onset.onset_detect(onset_envelope=oenv, sr=sr, hop_length=hop)
780
+ duration = len(guitar_mono) / sr
781
+ onset_rate = len(onsets) / max(duration, 1.0)
782
+ picking_density = float(min(onset_rate / EGTR_PICK_ONSET_SAT, 1.0))
783
+
784
+ # ── Mid-band spectral flatness: harmonic saturation from pickups/overdrive ─
785
+ # Flatness = geometric_mean / arithmetic_mean of magnitude spectrum.
786
+ # Ranges [0, 1]: 0 = perfectly tonal (sine-like), 1 = flat noise.
787
+ # Electric guitar pickups and overdrive add intermodulation products that
788
+ # raise flatness compared to clean acoustic guitar or piano.
789
+ mag_mid = mag[mid_mask, :]
790
+ if mag_mid.size == 0 or mag_mid.shape[0] == 0:
791
+ mid_saturation = 0.0
792
+ else:
793
+ geo_mean = np.exp(np.mean(np.log(mag_mid + EPS), axis=0)) # per-frame geo mean
794
+ arith_mean = np.mean(mag_mid, axis=0) + EPS # per-frame arith mean
795
+ frame_flat = geo_mean / arith_mean # flatness [0,1] per frame
796
+ raw_flatness = float(np.mean(frame_flat))
797
+ mid_saturation = float(min(raw_flatness / EGTR_FLATNESS_SAT, 1.0))
798
+
799
+ egtr_presence = float(
800
+ EGTR_PICK_WEIGHT * picking_density +
801
+ EGTR_FLAT_WEIGHT * mid_saturation
802
+ )
803
+
804
+ return egtr_presence, picking_density, mid_saturation
805
+
806
+
807
+ def extract_mir_features(path):
808
+ """
809
+ Returns (l3_vec, eng_vec, true_peak_dbfs).
810
+
811
+ eng_vec is a 32-element array whose indices correspond to ENG_DIMS.
812
+ Indices 0–25 are unchanged from cache version 3.
813
+ Indices 26–29 are drum sub-dimensions (v4–v6, unchanged in v8):
814
+ 26 drum_kick_presence β€” kick dominance ratio (v5, unchanged in v6)
815
+ 27 drum_cymbal_presence β€” HPSS sustained fraction Γ— band prom in 4–10 kHz (v6)
816
+ 28 drum_confidence β€” 0.55Γ—kick + 0.30Γ—hihat + 0.15Γ—cymbal (v6)
817
+ 29 hihat_presence β€” onset density of percussive 8–14 kHz (v6 NEW)
818
+ Indices 30–31 are electric guitar dimensions (v8 NEW):
819
+ 30 egtr_presence β€” composite guitar detection score [0,1]
820
+ 31 egtr_level β€” egtr_presence Γ— (guitar_rms / total_rms)
821
+
822
+ drum_punch (index 2) is gated by drum_confidence:
823
+ drum_punch = crest_factor(drum_stem) Γ— drum_confidence
824
+
825
+ v8 changes vs v7:
826
+ htdemucs_6s: model switched from htdemucs (4-stem) to htdemucs_6s (6-stem).
827
+ New stems: guitar (_STEM_GUITAR=4), piano (_STEM_PIANO=5).
828
+ other_prominence (idx 12) now reflects only synths/pads/strings β€”
829
+ guitar and piano are no longer blended into the other stem.
830
+ total_rms: now includes guitar_rms + piano_rms (previously only 4 stems).
831
+ This makes all stem-share ratios (drum_prominence, bass_prominence,
832
+ other_prominence, vocal_level) slightly different from v7.
833
+ egtr_presence (idx 30): HPSS-based picking density + mid-band spectral
834
+ flatness on the dedicated guitar stem. See _compute_guitar_dims().
835
+ egtr_level (idx 31): egtr_presence Γ— (guitar_rms / total_rms).
836
+ Scales presence by the guitar stem's share of the total mix RMS.
837
+ A guitar that is present but buried low in the mix scores lower than
838
+ one that is both detectable and prominent.
839
+ eng_vec now has 32 elements. All v7 cache entries are incompatible.
840
+
841
+ true_peak_dbfs is the ITU-R BS.1775-2 true peak level of the full mix
842
+ (4Γ— oversampled, in dBFS). Returned so the caller can enforce the -1.0 dBFS
843
+ true peak ceiling in apply_streaming_normalization() correctly.
844
+
845
+ Expected outcomes by source type (approximate):
846
+ Distorted electric guitar prominent in mix:
847
+ egtr_presence β‰ˆ 0.55–0.95, egtr_level β‰ˆ 0.15–0.45
848
+ Clean electric guitar (Fender/Strat clean tone):
849
+ egtr_presence β‰ˆ 0.25–0.55, egtr_level β‰ˆ 0.05–0.25
850
+ Acoustic guitar only (no electric):
851
+ egtr_presence β‰ˆ 0.10–0.35, egtr_level β‰ˆ 0.02–0.15
852
+ Piano-led track (no guitar):
853
+ egtr_presence β‰ˆ 0.05–0.20, egtr_level β‰ˆ 0.01–0.08
854
+ Pad/synth-led track (no guitar):
855
+ egtr_presence β‰ˆ 0.00–0.15, egtr_level β‰ˆ 0.00–0.05
856
+ """
857
+ y, sr = librosa.load(path, sr=44100, mono=True)
858
+ y_stereo, _ = librosa.load(path, sr=sr, mono=False)
859
+ if y_stereo.ndim == 1: # guard: mono input
860
+ y_stereo = np.stack([y_stereo, y_stereo])
861
+ emb_l3, _ = openl3.get_audio_embedding(
862
+ y, sr, content_type="music", embedding_size=512, verbose=False
863
+ )
864
+ l3_vec = np.mean(emb_l3, axis=0)
865
+ src = get_stems(path) # shape: (6, 2, n_samples) β€” stereo stems at 44100 Hz
866
+
867
+ # ── Stem RMS values ───────────────────────────────────────────────────────
868
+ drum_rms = np.sqrt(np.mean(src[_STEM_DRUMS] **2))
869
+ bass_rms = np.sqrt(np.mean(src[_STEM_BASS] **2))
870
+ other_rms = np.sqrt(np.mean(src[_STEM_OTHER] **2))
871
+ voc_rms = np.sqrt(np.mean(src[_STEM_VOCALS]**2))
872
+ guitar_rms = np.sqrt(np.mean(src[_STEM_GUITAR]**2))
873
+ piano_rms = np.sqrt(np.mean(src[_STEM_PIANO] **2))
874
+ # v8: total_rms includes all 6 stems for accurate mix-share ratios.
875
+ total_rms = drum_rms + bass_rms + other_rms + voc_rms + guitar_rms + piano_rms + EPS
876
+
877
+ # ── Drum crest factor (raw, before gating) ────────────────────────────────
878
+ drum_crest_raw = np.max(np.abs(src[_STEM_DRUMS])) / (drum_rms + EPS)
879
+
880
+ # ── Drum sub-band analysis (v6, unchanged in v8) ──────────────────────────
881
+ drum_mono = (src[_STEM_DRUMS][0] + src[_STEM_DRUMS][1]) / 2.0
882
+
883
+ drum_kick_pres, drum_cymbal_pres, hihat_pres = _compute_drum_bands(drum_mono, y, sr)
884
+
885
+ # drum_confidence: weighted sum of three [0,1] drum indicators.
886
+ kick_conf = min(drum_kick_pres / DRUM_KICK_SAT, 1.0)
887
+ hihat_conf = hihat_pres # already in [0, 1]
888
+ cymbal_conf = drum_cymbal_pres # already in [0, 1]
889
+ drum_conf = float(
890
+ DRUM_KICK_WEIGHT * kick_conf +
891
+ DRUM_HIHAT_WEIGHT * hihat_conf +
892
+ DRUM_CYMBAL_WEIGHT * cymbal_conf
893
+ )
894
+
895
+ # Gate drum_punch by confidence: suppresses false positives from guitar bleed.
896
+ drum_punch_val = drum_crest_raw * drum_conf
897
+
898
+ # ── Bass crest factor ─────────────────────────────────────────────────────
899
+ bass_tightness_val = np.max(np.abs(src[_STEM_BASS])) / (bass_rms + EPS)
900
+
901
+ # ── Mix-presence ratios ───────────────────────────────────────────────────
902
+ drum_prom = drum_rms / total_rms
903
+ bass_prom = bass_rms / total_rms
904
+ other_prom = other_rms / total_rms
905
+ voc_level = voc_rms / total_rms
906
+
907
+ # ── Vocal sub-dimensions ──────────────────────────────────────────────────
908
+ voc_mono = (src[_STEM_VOCALS][0] + src[_STEM_VOCALS][1]) / 2.0
909
+ voc_rms_frames = librosa.feature.rms(
910
+ y=voc_mono, frame_length=2048, hop_length=512
911
+ )[0]
912
+ voc_presence = float(np.mean(voc_rms_frames > VOCAL_ACTIVITY_THRESHOLD))
913
+ voc_brightness = float(np.mean(
914
+ librosa.feature.spectral_centroid(y=voc_mono, sr=sr)
915
+ ))
916
+
917
+ # ── Electric guitar sub-dimensions (v8 NEW) ───────────────────────────────
918
+ guitar_mono = (src[_STEM_GUITAR][0] + src[_STEM_GUITAR][1]) / 2.0
919
+ egtr_pres, _picking, _flatness = _compute_guitar_dims(guitar_mono, sr)
920
+ # egtr_level: scale presence by the guitar's share of total mix RMS.
921
+ # Rewards tracks where guitar is both detectable AND prominent in the mix.
922
+ egtr_level_val = float(egtr_pres * (guitar_rms / total_rms))
923
+
924
+ # ── Mid-side stereo width ─────────────────────────────────────────────────
925
+ mid, side = (y_stereo[0] + y_stereo[1]) / 2, (y_stereo[0] - y_stereo[1]) / 2
926
+ ms_ratio = (np.sqrt(np.mean(side**2)) /
927
+ (np.sqrt(np.mean(mid**2)) + np.sqrt(np.mean(side**2)) + EPS))
928
+
929
+ # ── Short-term loudness (K-weighted, BS.1770 β€” no gating) ────────────────
930
+ # Uses the same K-weighting filter as integrated LUFS. window=3s, hop=1s
931
+ # matches the original librosa.feature.rms() block framing.
932
+ st_lufs = _compute_short_term_lufs_array(y_stereo, sr, window_s=3.0, hop_s=1.0)
933
+
934
+ # ── Clipping ─────────────────────────────────────────────────────────────
935
+ c_events, c_total, c_ratio, c_max, c_mean = _measure_clipping(y)
936
+
937
+ # ── Assemble eng_vec β€” must match ENG_DIMS order exactly ─────────────────
938
+ eng_vec = np.array([
939
+ _compute_integrated_lufs(y_stereo, sr), # 0 lufs (ITU-R BS.1770-4 β€” K-weighted, gated)
940
+ np.max(np.abs(y)) / (np.sqrt(np.mean(y**2)) + EPS), # 1 crest_factor
941
+ drum_punch_val, # 2 drum_punch (gated)
942
+ drum_prom, # 3 drum_prominence
943
+ ms_ratio, # 4 stereo_width
944
+ librosa.feature.spectral_centroid(y=y, sr=sr).mean(), # 5 brightness
945
+ entropy(np.mean(librosa.feature.chroma_cqt(y=y, sr=sr), axis=1)), # 6 harmonic_complexity
946
+ c_events, # 7 clip_num_events
947
+ c_total, # 8 clip_total
948
+ c_ratio, # 9 clip_ratio
949
+ c_max, # 10 clip_max_run
950
+ c_mean, # 11 clip_mean_run
951
+ other_prom, # 12 other_prominence
952
+ voc_level, # 13 vocal_level
953
+ voc_presence, # 14 vocal_presence
954
+ voc_brightness, # 15 vocal_brightness
955
+ np.max(st_lufs) - np.median(st_lufs), # 16 chorus_lift
956
+ np.mean(librosa.feature.zero_crossing_rate(y)), # 17 crispness
957
+ np.mean(librosa.feature.spectral_flatness(y=y)), # 18 clarity
958
+ len(y) / sr, # 19 length_s
959
+ np.std(st_lufs), # 20 lufs_variation
960
+ np.max(st_lufs), # 21 peak_lufs
961
+ np.mean(librosa.feature.spectral_rolloff(y=y, sr=sr)), # 22 rolloff
962
+ bass_tightness_val, # 23 bass_tightness
963
+ bass_prom, # 24 bass_prominence
964
+ np.nan_to_num(np.corrcoef(y_stereo[0], y_stereo[1])[0, 1]), # 25 phase_corr
965
+ drum_kick_pres, # 26 drum_kick_presence
966
+ drum_cymbal_pres, # 27 drum_cymbal_presence (v6)
967
+ drum_conf, # 28 drum_confidence (v6)
968
+ hihat_pres, # 29 hihat_presence (v6)
969
+ egtr_pres, # 30 egtr_presence (v8 NEW)
970
+ egtr_level_val, # 31 egtr_level (v8 NEW)
971
+ ])
972
+
973
+ # ── True peak (BS.1775-2, 4Γ— oversampled) ────────────────────────────────
974
+ # Returned as third element so the modal worker / analyze_submissions.py
975
+ # can enforce the -1.0 dBFS true peak ceiling in normalization simulation.
976
+ true_peak = _true_peak_dbfs(y_stereo)
977
+
978
+ return l3_vec, eng_vec, true_peak