Melati-AudioSet-Ensemble

Six AudioSet taggers run on the same clip, their probabilities averaged: 0.5219 mAP on AudioSet's evaluation set, ahead of every single model we scored on the same clips, the best of them our own Melati-AudioSet-EAT-Large (0.5124). Four members are released by others (CED-base, EAT-large, SSLAM and BEATs) and two are Melati-AudioSet-EAT-Large's v3 and v4. It names which of AudioSet's 527 sounds are in a clip, for six forward passes (1.20B parameters in all); Melati-AudioSet-EAT-Large is one model of 309M, distilled from three of these members' average. Part of Melati, Scicom's speech-data pipeline.

model	mAP
this ensemble	0.5219	the six below, averaged
Melati-AudioSet-EAT-Large v4	0.5124	ours
Melati-AudioSet-EAT-Large v3	0.5105	ours
CED-base	0.4993	Dinkel et al., ICASSP 2024
EAT-large	0.4947	Chen et al., IJCAI 2024
SSLAM	0.4945	Alex et al., ICLR 2025
BEATs	0.4835	Chen et al., ICML 2023

All on the same 17,141 clips: AudioSet's evaluation set as agkphysics/AudioSet holds it today (YouTube removes clips over time, so every paper's copy differs and published numbers are not comparable across papers), each model in fp32 with its own preprocessing, mAP the mean over the 527 classes of their average precision. Every number here is this repo's, loaded from the Hub as the Use section loads it. CED-base reads 0.4993 here and 0.4997 on Melati-AudioSet-EAT-Large's card, where it was given 64 clips at a time (see Use); the other numbers there differ from these by 0.0001 at most (that card scores probabilities stored in 16 bits).

Use

pip install torch numpy huggingface_hub safetensors soundfile

import sys
from huggingface_hub import snapshot_download

path = snapshot_download("Scicom-intl/Melati-AudioSet-Ensemble")
sys.path.insert(0, path)
from melati_ensemble import Ensemble, read

model = Ensemble(path)             # the six members, only their weights fetched, each at a pinned revision; the GPU if any

probs = model([read("clip.wav")])[0]    # any sample rate or channels, read as 16 kHz mono; AudioSet's 527 classes
for i in probs.argsort()[::-1][:5]:
    print(model.labels[i], round(float(probs[i]), 3))

A clip is 16 kHz mono, cut or zero-padded to 10 s, the length every member was scored on. A longer recording goes through model.windows(wav, hop_s=5.0), which gives each 10 s window's start in seconds and its probabilities; the highest over the windows tags the whole recording. model(clips, each=True) also returns every member's own probabilities, and Ensemble(path, members=[...]) runs only some of them (their names are in ensemble.json).

Every member is built in melati_ensemble.py in plain PyTorch, and only its config.json and model.safetensors are fetched from its repo at the pinned revision: no repo's own code is run, so it needs neither transformers nor timm nor torchaudio (soundfile only for read). Each member is fed its own preprocessing, as it was scored, computed for the whole batch on the GPU. On 512 evaluation clips, each member's probabilities match its own repo's code to within 3e-05 in fp32. CED-base's decibel floor is set under each clip's own loudest value, as its feature extractor sets it for a clip given alone, so a clip's tags never depend on the clips batched with it. Revisions before this one ran each member's own code through transformers' trust_remote_code, which needed timm and did not load without it.

On one GPU, in batches of 16, it tags 18 ten-second clips a second in fp32; Ensemble(path, bf16=True) tags 65 a second at 0.5219 mAP. Tested with torch 2.8.0, numpy 1.26.4, huggingface_hub 0.36.2 and safetensors 0.8.0, and loaded into an empty cache in a new environment holding only the five packages above, at torch 2.15.0, numpy 2.5.4, huggingface_hub 2.3.0, safetensors 0.8.0 and soundfile 0.14.0.

Inference engine

melati_engine.py serves the ensemble on CUDA GPUs: every GPU holds all six members, their weights in bf16, and runs a CUDA graph captured for each batch size, and the 10 s windows of every request share one queue, so concurrent requests fill the same batches and every GPU takes its share. A recording longer than 10 s is cut into 10 s windows every hop_s seconds (5 by default), and a sound's probability in it is its highest over the windows.

pip install torch numpy huggingface_hub safetensors soundfile fastapi uvicorn
hf download Scicom-intl/Melati-AudioSet-Ensemble --local-dir melati-ensemble
python melati-ensemble/melati_engine.py serve --gpus 0,1 --port 8000
curl -s --data-binary @clip.wav "localhost:8000/tag?top_k=5"
python melati-ensemble/melati_engine.py tag clips/ --gpus 0,1 --out tags.jsonl    # every audio file under clips/

POST /tag takes an audio file as its body (anything soundfile reads), or JSON {"pcm16": base64, "sample_rate": 16000}, and answers the top_k most probable sounds (threshold sets a floor); windows=true adds each window's, members=true each member's and probs=true all 527. In Python, Engine(gpus=[0]).tag(wave) gives each window's start and probabilities. GET /stats says how full its batches are and how busy its GPUs.

Through the engine on 4 GPUs the evaluation set scores 0.5220 mAP (Ensemble(path, bf16=True): 0.5219), at 282 ten-second clips a second, 2,820 times real time and 3.9 times one GPU's 72. One GPU runs about as fast as Ensemble(path, bf16=True) in batches of 64 (69 a second): a clip costs six forward passes, 1.25 TFLOP, so the GPU's compute is the limit, and what the engine adds is the work spread over GPUs and concurrent requests batched together. An idle engine answers a clip in 38 ms (45 ms without CUDA graphs). Over HTTP on 4 GPUs (the server in one process, clients posting 10 s WAV files), one client gets each answer in 40 ms; 16 clients at once get 152 answers a second (88 ms at the median), and 256 at once 201 a second.

What each member adds

How much the ensemble loses without each member, on the same clips. The three released taggers Melati-AudioSet-EAT-Large v4 was distilled from average 0.5166; with v3 and v4 that is 0.5189, with BEATs 0.5201, and with all three 0.5219. It misses BEATs most (+0.0030), the weakest member alone (0.4835): it was pre-trained differently from the rest. v3 and v4 are the same stage of Melati-AudioSet-EAT-Large trained twice, with and without a frame-level loss, so each adds little once the other is in (+0.0004 and +0.0002). EAT-large adds nothing on top of the rest (-0.0001); v3 and v4 were trained from it.

Members

member	where it is loaded from	licence	parameters	mAP alone
Melati-AudioSet-EAT-Large v4 (ours)	Scicom-intl/Melati-AudioSet-EAT-Large at 78248ff	MIT	309M	0.5124
Melati-AudioSet-EAT-Large v3 (ours)	Scicom-intl/Melati-AudioSet-EAT-Large at 06a9aef	MIT	309M	0.5105
CED-base (Dinkel et al., ICASSP 2024)	mispeech/ced-base at db3e14a	Apache-2.0	86M	0.4993
EAT-large (Chen et al., IJCAI 2024)	worstchan/EAT-large_epoch20_finetune_AS2M at 53b1251	MIT	309M	0.4947
SSLAM (Alex et al., ICLR 2025)	ta012/SSLAM_AS2M_Finetuned at 4500c5d	MIT	90M	0.4945
BEATs (Chen et al., ICML 2023)	this repo: beats.safetensors (microsoft/unilm's iter3+ checkpoint fine-tuned on AudioSet-2M, cpt2) and melati_beats/	MIT	91M	0.4835

Labels

All 527 of AudioSet's classes, as model.labels names them (the index is the output's column), under the 7 top-level categories of AudioSet's ontology (Google, CC BY-SA 4.0); the 29 classes it places under two categories are listed under the first. AP is this model's average precision on the class over the 17,141 evaluation clips in fp32, and clips is how many of them carry the label; the mAP above is the mean of the AP column.

Human sounds: 72 labels, mean AP 0.513

index	label	AP	clips
0	Speech	0.863	4,419
1	Male speech, man speaking	0.122	58
2	Female speech, woman speaking	0.166	59
3	Child speech, kid speaking	0.521	159
4	Conversation	0.502	50
5	Narration, monologue	0.153	55
6	Babbling	0.680	50
7	Speech synthesizer	0.542	54
8	Shout	0.232	52
9	Bellow	0.208	52
10	Whoop	0.255	54
11	Yell	0.311	45
12	Battle cry	0.891	54
13	Children shouting	0.618	53
14	Screaming	0.273	38
15	Whispering	0.861	51
16	Laughter	0.493	96
17	Baby laughter	0.698	56
18	Giggle	0.388	54
19	Snicker	0.312	54
20	Belly laugh	0.332	52
21	Chuckle, chortle	0.262	54
22	Crying, sobbing	0.576	40
23	Baby cry, infant cry	0.562	36
24	Whimper	0.404	45
25	Wail, moan	0.239	41
26	Sigh	0.431	46
27	Singing	0.380	292
28	Choir	0.732	84
29	Yodeling	0.677	53
30	Chant	0.505	48
31	Mantra	0.395	54
32	Male singing	0.274	55
33	Female singing	0.319	47
34	Child singing	0.637	50
35	Synthetic singing	0.572	45
36	Rapping	0.536	55
37	Humming	0.580	42
38	Groan	0.083	43
39	Grunt	0.421	43
40	Whistling	0.787	46
41	Breathing	0.193	49
42	Wheeze	0.425	51
43	Snoring	0.884	51
44	Gasp	0.314	47
45	Pant	0.531	47
46	Snort	0.272	50
47	Cough	0.637	42
48	Throat clearing	0.632	50
49	Sneeze	0.604	37
50	Sniff	0.437	52
51	Run	0.410	50
52	Shuffle	0.581	56
53	Walk, footsteps	0.505	49
54	Chewing, mastication	0.611	42
55	Biting	0.363	54
56	Gargling	0.805	41
57	Stomach rumble	0.850	41
58	Burping, eructation	0.748	48
59	Hiccup	0.800	53
60	Fart	0.854	48
61	Hands	0.581	49
62	Finger snapping	0.639	47
63	Clapping	0.547	52
64	Heart sounds, heartbeat	0.666	57
65	Heart murmur	0.401	49
66	Cheering	0.586	52
67	Applause	0.620	51
68	Chatter	0.772	46
69	Crowd	0.752	126
70	Hubbub, speech noise, speech babble	0.393	50
71	Children playing	0.734	48

Animal: 65 labels, mean AP 0.664

index	label	AP	clips
72	Animal	0.686	586
73	Domestic animals, pets	0.717	295
74	Dog	0.794	193
75	Bark	0.493	51
76	Yip	0.391	54
77	Howl	0.742	51
78	Bow-wow	0.530	62
79	Growling	0.417	56
80	Whimper (dog)	0.298	48
81	Cat	0.819	96
82	Purr	0.844	56
83	Meow	0.732	56
84	Hiss	0.461	78
85	Caterwaul	0.753	56
86	Livestock, farm animals, working animals	0.783	99
87	Horse	0.629	73
88	Clip-clop	0.553	55
89	Neigh, whinny	0.898	44
90	Cattle, bovinae	0.953	53
91	Moo	0.948	51
92	Cowbell	0.774	57
93	Pig	0.716	45
94	Oink	0.491	48
95	Goat	0.677	48
96	Bleat	0.551	49
97	Sheep	0.656	52
98	Fowl	0.908	179
99	Chicken, rooster	0.859	62
100	Cluck	0.722	42
101	Crowing, cock-a-doodle-doo	0.939	45
102	Turkey	0.963	56
103	Gobble	0.957	56
104	Duck	0.523	50
105	Quack	0.671	55
106	Goose	0.823	61
107	Honk	0.766	53
108	Wild animals	0.373	42
109	Roaring cats (lions, tigers)	0.550	44
110	Roar	0.659	50
111	Bird	0.501	187
112	Bird vocalization, bird call, bird song	0.355	151
113	Chirp, tweet	0.156	62
114	Squawk	0.343	50
115	Pigeon, dove	0.830	50
116	Coo	0.855	50
117	Crow	0.742	59
118	Caw	0.740	57
119	Owl	0.858	54
120	Hoot	0.965	53
121	Bird flight, flapping wings	0.756	46
122	Canidae, dogs, wolves	0.301	52
123	Rodents, rats, mice	0.493	51
124	Mouse	0.206	51
125	Patter	0.418	56
126	Insect	0.883	141
127	Cricket	0.723	55
128	Mosquito	0.840	47
129	Fly, housefly	0.913	94
130	Buzz	0.102	52
131	Bee, wasp, etc.	0.852	50
132	Frog	0.955	50
133	Croak	0.942	49
134	Snake	0.582	37
135	Rattle	0.087	52
136	Whale vocalization	0.746	48

Music: 146 labels, mean AP 0.495

index	label	AP	clips
137	Music	0.883	4,654
138	Musical instrument	0.451	369
139	Plucked string instrument	0.558	232
140	Guitar	0.706	274
141	Electric guitar	0.622	115
142	Bass guitar	0.543	52
143	Acoustic guitar	0.528	84
144	Steel guitar, slide guitar	0.381	56
145	Tapping (guitar technique)	0.501	54
146	Strum	0.294	56
147	Banjo	0.813	57
148	Sitar	0.539	49
149	Mandolin	0.501	55
150	Zither	0.690	49
151	Ukulele	0.607	53
152	Keyboard (musical)	0.555	134
153	Piano	0.446	83
154	Electric piano	0.490	55
155	Organ	0.849	115
156	Electronic organ	0.521	59
157	Hammond organ	0.588	57
158	Synthesizer	0.241	49
159	Sampler	0.188	54
160	Harpsichord	0.832	56
161	Percussion	0.636	215
162	Drum kit	0.664	73
163	Drum machine	0.338	60
164	Drum	0.580	171
165	Snare drum	0.665	72
166	Rimshot	0.661	69
167	Drum roll	0.647	56
168	Bass drum	0.629	81
169	Timpani	0.791	51
170	Tabla	0.680	54
171	Cymbal	0.609	60
172	Hi-hat	0.663	53
173	Wood block	0.690	52
174	Tambourine	0.741	54
175	Rattle (instrument)	0.463	49
176	Maraca	0.433	48
177	Gong	0.804	51
178	Tubular bells	0.372	53
179	Mallet percussion	0.349	46
180	Marimba, xylophone	0.419	57
181	Glockenspiel	0.445	48
182	Vibraphone	0.533	54
183	Steelpan	0.822	53
184	Orchestra	0.410	53
185	Brass instrument	0.825	179
186	French horn	0.821	51
187	Trumpet	0.614	50
188	Trombone	0.654	53
189	Bowed string instrument	0.597	101
190	String section	0.564	54
191	Violin, fiddle	0.456	53
192	Pizzicato	0.743	115
193	Cello	0.679	59
194	Double bass	0.713	52
195	Wind instrument, woodwind instrument	0.815	224
196	Flute	0.627	46
197	Saxophone	0.713	53
198	Clarinet	0.683	55
199	Harp	0.793	50
200	Bell	0.265	52
201	Church bell	0.711	54
202	Jingle bell	0.691	50
203	Bicycle bell	0.801	50
204	Tuning fork	0.788	53
205	Chime	0.780	49
206	Wind chime	0.843	47
207	Change ringing (campanology)	0.943	58
208	Harmonica	0.935	50
209	Accordion	0.886	55
210	Bagpipes	0.971	58
211	Didgeridoo	0.914	54
212	Shofar	0.914	52
213	Theremin	0.566	51
214	Singing bowl	0.644	51
215	Scratching (performance technique)	0.610	56
216	Pop music	0.231	70
217	Hip hop music	0.342	51
218	Beatboxing	0.650	46
219	Rock music	0.211	81
220	Heavy metal	0.303	60
221	Punk rock	0.255	58
222	Grunge	0.293	51
223	Progressive rock	0.377	64
224	Rock and roll	0.388	75
225	Psychedelic rock	0.280	51
226	Rhythm and blues	0.140	52
227	Soul music	0.266	56
228	Reggae	0.426	45
229	Country	0.271	56
230	Swing music	0.386	50
231	Bluegrass	0.526	60
232	Funk	0.263	49
233	Folk music	0.151	47
234	Middle Eastern music	0.299	43
235	Jazz	0.111	53
236	Disco	0.326	51
237	Classical music	0.123	51
238	Opera	0.606	54
239	Electronic music	0.271	52
240	House music	0.363	50
241	Techno	0.533	78
242	Dubstep	0.499	83
243	Drum and bass	0.378	66
244	Electronica	0.370	70
245	Electronic dance music	0.525	72
246	Ambient music	0.343	49
247	Trance music	0.394	52
248	Music of Latin America	0.267	49
249	Salsa music	0.629	53
250	Flamenco	0.458	54
251	Blues	0.265	55
252	Music for children	0.259	50
253	New-age music	0.320	52
254	Vocal music	0.302	66
255	A capella	0.552	54
256	Music of Africa	0.356	63
257	Afrobeat	0.484	51
258	Christian music	0.141	36
259	Gospel music	0.479	46
260	Music of Asia	0.169	50
261	Carnatic music	0.631	56
262	Music of Bollywood	0.487	36
263	Ska	0.433	54
264	Traditional music	0.267	54
265	Independent music	0.196	56
266	Song	0.133	135
267	Background music	0.373	54
268	Theme music	0.311	53
269	Jingle (music)	0.181	50
270	Soundtrack music	0.095	48
271	Lullaby	0.471	50
272	Video game music	0.530	47
273	Christmas music	0.104	44
274	Dance music	0.304	71
275	Wedding music	0.127	53
276	Happy music	0.211	53
277	Funny music	0.139	45
278	Sad music	0.321	47
279	Tender music	0.132	55
280	Exciting music	0.106	47
281	Angry music	0.772	53
282	Scary music	0.462	48

Natural sounds: 17 labels, mean AP 0.511

index	label	AP	clips
283	Wind	0.541	170
284	Rustling leaves	0.206	54
285	Wind noise (microphone)	0.570	139
286	Thunderstorm	0.811	53
287	Thunder	0.807	52
288	Water	0.600	239
289	Rain	0.776	86
290	Raindrop	0.603	54
291	Rain on surface	0.645	52
292	Stream	0.461	52
293	Waterfall	0.605	53
294	Ocean	0.487	59
295	Waves, surf	0.447	54
296	Steam	0.489	55
297	Gurgling	0.163	52
298	Fire	0.410	54
299	Crackle	0.072	50

Sounds of things: 166 labels, mean AP 0.551

index	label	AP	clips
300	Vehicle	0.583	698
301	Boat, Water vehicle	0.692	223
302	Sailboat, sailing ship	0.603	57
303	Rowboat, canoe, kayak	0.615	58
304	Motorboat, speedboat	0.478	56
305	Ship	0.416	56
306	Motor vehicle (road)	0.144	77
307	Car	0.440	250
308	Vehicle horn, car horn, honking	0.493	54
309	Toot	0.572	53
310	Car alarm	0.758	50
311	Power windows, electric windows	0.728	54
312	Skidding	0.656	49
313	Tire squeal	0.318	50
314	Car passing by	0.447	53
315	Race car, auto racing	0.470	64
316	Truck	0.384	89
317	Air brake	0.400	56
318	Air horn, truck horn	0.684	51
319	Reversing beeps	0.645	51
320	Ice cream truck, ice cream van	0.820	52
321	Bus	0.508	54
322	Emergency vehicle	0.964	121
323	Police car (siren)	0.810	52
324	Ambulance (siren)	0.791	54
325	Fire engine, fire truck (siren)	0.907	51
326	Motorcycle	0.319	50
327	Traffic noise, roadway noise	0.145	61
328	Rail transport	0.838	161
329	Train	0.835	160
330	Train whistle	0.509	56
331	Train horn	0.700	54
332	Railroad car, train wagon	0.839	155
333	Train wheels squealing	0.881	56
334	Subway, metro, underground	0.764	58
335	Aircraft	0.630	99
336	Aircraft engine	0.672	51
337	Jet engine	0.698	58
338	Propeller, airscrew	0.293	55
339	Helicopter	0.651	52
340	Fixed-wing aircraft, airplane	0.596	59
341	Bicycle	0.363	56
342	Skateboard	0.829	55
343	Engine	0.437	173
344	Light engine (high frequency)	0.607	53
345	Dental drill, dentist's drill	0.416	52
346	Lawn mower	0.601	56
347	Chainsaw	0.648	52
348	Medium engine (mid frequency)	0.116	53
349	Heavy engine (low frequency)	0.120	56
350	Engine knocking	0.275	56
351	Engine starting	0.483	56
352	Idling	0.231	80
353	Accelerating, revving, vroom	0.287	131
354	Door	0.294	57
355	Doorbell	0.632	49
356	Ding-dong	0.278	42
357	Sliding door	0.344	54
358	Slam	0.243	51
359	Knock	0.620	45
360	Tap	0.510	55
361	Squeak	0.514	51
362	Cupboard open or close	0.484	48
363	Drawer open or close	0.513	54
364	Dishes, pots, and pans	0.383	51
365	Cutlery, silverware	0.498	51
366	Chopping (food)	0.702	47
367	Frying (food)	0.828	58
368	Microwave oven	0.418	52
369	Blender	0.831	51
370	Water tap, faucet	0.466	66
371	Sink (filling or washing)	0.598	52
372	Bathtub (filling or washing)	0.652	42
373	Hair dryer	0.910	50
374	Toilet flush	0.869	46
375	Toothbrush	0.373	50
376	Electric toothbrush	0.427	49
377	Vacuum cleaner	0.783	47
378	Zipper (clothing)	0.710	52
379	Keys jangling	0.375	52
380	Coin (dropping)	0.574	55
381	Scissors	0.550	52
382	Electric shaver, electric razor	0.534	49
383	Shuffling cards	0.844	57
384	Typing	0.658	52
385	Typewriter	0.690	53
386	Computer keyboard	0.717	51
387	Writing	0.582	51
388	Alarm	0.321	53
389	Telephone	0.339	45
390	Telephone bell ringing	0.701	47
391	Ringtone	0.612	46
392	Telephone dialing, DTMF	0.694	47
393	Dial tone	0.656	50
394	Busy signal	0.636	46
395	Alarm clock	0.469	54
396	Siren	0.932	181
397	Civil defense siren	0.934	52
398	Buzzer	0.392	55
399	Smoke detector, smoke alarm	0.771	48
400	Fire alarm	0.755	50
401	Foghorn	0.539	54
402	Whistle	0.477	49
403	Steam whistle	0.713	66
404	Mechanisms	0.370	91
405	Ratchet, pawl	0.358	48
406	Clock	0.515	54
407	Tick	0.227	55
408	Tick-tock	0.467	56
409	Gears	0.377	47
410	Pulleys	0.444	51
411	Sewing machine	0.899	56
412	Mechanical fan	0.440	55
413	Air conditioning	0.339	55
414	Cash register	0.862	53
415	Printer	0.838	50
416	Camera	0.570	51
417	Single-lens reflex camera	0.635	51
418	Tools	0.448	93
419	Hammer	0.631	58
420	Jackhammer	0.647	55
421	Sawing	0.591	55
422	Filing (rasp)	0.730	56
423	Sanding	0.887	55
424	Power tool	0.419	72
425	Drill	0.581	55
426	Explosion	0.290	46
427	Gunshot, gunfire	0.801	138
428	Machine gun	0.784	49
429	Fusillade	0.618	44
430	Artillery fire	0.826	48
431	Cap gun	0.882	45
432	Fireworks	0.768	48
433	Firecracker	0.658	42
434	Burst, pop	0.052	52
435	Eruption	0.433	52
436	Boom	0.545	53
437	Wood	0.534	151
438	Chop	0.520	49
439	Splinter	0.528	54
440	Crack	0.251	46
441	Glass	0.425	54
442	Chink, clink	0.205	48
443	Shatter	0.372	47
444	Liquid	0.140	53
445	Splash, splatter	0.508	42
446	Slosh	0.377	48
447	Squish	0.106	53
448	Drip	0.324	52
449	Pour	0.534	49
450	Trickle, dribble	0.218	50
451	Gush	0.347	50
452	Fill (with liquid)	0.223	53
453	Spray	0.583	54
454	Pump (liquid)	0.435	53
455	Stir	0.818	57
456	Boiling	0.316	55
457	Sonar	0.675	48
458	Arrow	0.791	55
459	Whoosh, swoosh, swish	0.425	51
460	Thump, thud	0.313	46
461	Thunk	0.402	47
462	Electronic tuner	0.698	53
463	Effects unit	0.648	102
464	Chorus effect	0.626	52
465	Basketball bounce	0.681	52

Source-ambiguous sounds: 40 labels, mean AP 0.400

index	label	AP	clips
466	Bang	0.157	51
467	Slap, smack	0.742	50
468	Whack, thwack	0.278	46
469	Smash, crash	0.506	50
470	Breaking	0.488	47
471	Bouncing	0.242	52
472	Whip	0.773	50
473	Flap	0.152	51
474	Scratch	0.571	53
475	Scrape	0.052	47
476	Rub	0.803	127
477	Roll	0.379	50
478	Crushing	0.448	54
479	Crumpling, crinkling	0.546	51
480	Tearing	0.429	49
481	Beep, bleep	0.381	49
482	Ping	0.461	51
483	Ding	0.359	54
484	Clang	0.386	50
485	Squeal	0.255	51
486	Creak	0.188	50
487	Rustle	0.124	51
488	Whir	0.132	54
489	Clatter	0.145	58
490	Sizzle	0.768	64
491	Clicking	0.691	48
492	Clickety-clack	0.589	57
493	Rumble	0.396	50
494	Plop	0.874	45
495	Jingle, tinkle	0.287	48
496	Hum	0.398	71
497	Zing	0.321	43
498	Boing	0.842	40
499	Crunch	0.546	46
500	Silence	0.062	47
501	Sine wave	0.606	53
502	Harmonic	0.082	50
503	Chirp tone	0.337	52
504	Sound effect	0.077	40
505	Pulse	0.134	48

Channel, environment and background: 21 labels, mean AP 0.310

index	label	AP	clips
506	Inside, small room	0.223	479
507	Inside, large room or hall	0.134	133
508	Inside, public space	0.157	45
509	Outside, urban or manmade	0.128	177
510	Outside, rural or natural	0.076	157
511	Reverberation	0.321	55
512	Echo	0.408	53
513	Noise	0.134	56
514	Environmental noise	0.581	48
515	Static	0.574	50
516	Mains hum	0.639	56
517	Distortion	0.548	107
518	Sidetone	0.255	53
519	Cacophony	0.180	53
520	White noise	0.128	54
521	Pink noise	0.411	50
522	Throbbing	0.360	52
523	Vibration	0.179	56
524	Television	0.309	49
525	Radio	0.447	52
526	Field recording	0.315	55

Licence

melati_ensemble.py and this card are MIT. The member networks in melati_ensemble.py are written after their repos' modeling code (EAT's, MIT; CED's, Apache-2.0), and its preprocessing and resampler after torchaudio's (BSD-2-Clause). BEATs's weights and code are Microsoft's, MIT (melati_beats/LICENSE); the code is unilm's, unchanged but for its imports of itself, made relative, and its torchaudio import, moved into the one method that uses it. The other members' weights are fetched from their own repos under their own licences: Apache-2.0 (CED-base) and MIT (EAT-large, SSLAM, Melati-AudioSet-EAT-Large). AudioSet's labels are CC BY 4.0 (Google); its audio is YouTube's and is not redistributed here.

Citation

@misc{zolkepli2026melati,
  title  = {Melati: Multilingual Expressive speech, Labelled for Affect, Tags and Interjections},
  author = {Husein Zolkepli},
  year   = {2026},
  note   = {Scicom (MSC) Berhad}
}

We thank Scicom (MSC) Berhad for the compute.