aadd / README.md

Model card: install line

7fe41e7 verified about 16 hours ago

9.2 kB

---
license: mit
language:
  - en
  - ru
library_name: pytorch
pipeline_tag: audio-classification
base_model: openai/whisper-small
tags:
  - audio
  - speech
  - audio-quality
  - degradation-detection
  - data-curation
  - sound-event-detection
  - whisper
datasets:
  - openslr/librispeech_asr
  - istupakov/russian_librispeech
metrics:
  - average_precision
  - roc_auc
---
# AADD v3.1: which audio degradation, and where

[![GitHub](https://img.shields.io/badge/GitHub-gertelrina%2Faadd-181717?logo=github)](https://github.com/gertelrina/aadd)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://github.com/gertelrina/aadd/blob/main/LICENSE)

**Code, CLI, Python API and interactive dataset report: [github.com/gertelrina/aadd](https://github.com/gertelrina/aadd).**

**AADD** (Automatic Audio Degradation Detection) finds **12 kinds of degradation in
speech recordings every 20 ms**: background noise, music, reverberation, clipping,
low-pass / narrow band, high-pass, codec artefacts, packet loss, mains hum, clicks,
pitch shift and time stretch. Several can be present at once. It is meant for
cleaning speech datasets (ASR, TTS) and for auditing recordings.

<p align="center"><img src="demo.gif" width="860" alt="AADD v3 localising noise, reverberation, clipping, codec artefacts and clicks in a 14-second clip"></p>

* Backbone: Whisper-small encoder (88 M parameters), fine-tuned end to end
* Output: one sigmoid per class per 20 ms frame on 8-s windows; any length via sliding windows
* Languages seen in training: English and Russian

## What changed in v3.1 (2026-11-06)

v3.0 called real background babble (cafés, streets, buses) "music". v3.1 is v3.0
fine-tuned for 4 000 steps with 36 h of real environmental noise from FSD50K
(CC0 / CC BY clips only, no music or voice labels) and synthetic crowd babble
added to the noise class. On the VoiceBank-DEMAND test set, music detections per
noise type (v3.0 → v3.1): café 41 % → 16 %, public square 49 % → 25 %, bus 39 % →
18 %, office 5 % → 4 %; noise recall rose (café 84 % → 95 %, square 55 % → 78 %).
The DEMAND "living room" noise keeps 86 % music: it contains sustained harmonic
tones (music / TV), so this is probably correct. Synthetic held-out tests are
unchanged or better (frame macro-AP 0.955 / 0.960 en / ru).

`Detector("v3")` and `--model v3` now load v3.1; `v3.0` stays available.

## Usage

```bash
pip install "aadd @ git+https://github.com/gertelrina/aadd"
aadd scan my_dataset/ --model v3 --report report.html   # downloads this checkpoint on first use
aadd scan call.wav --model v3 --resolution 0.1 --segments per_100ms.csv
```

```python
from aadd import Detector
det = Detector("v3", resolution=1.0)       # per-second summary; frame probabilities in res.frame_probs
res = det.predict_file("call.wav")
res.detected                                # {'noise': 0.97, 'reverb': 0.88}
res.frame_probs.shape                       # (n_frames_20ms, 12)
```

Or download the file directly:

```python
from huggingface_hub import hf_hub_download
path = hf_hub_download("agertel/aadd", "aadd_v3_1_whisper_small_frames.pt")
det = Detector(path)
```

## Results

**Out-of-domain, real recordings** (never seen in training). Clip score = 95th
percentile of frame probabilities.

| set | class | ROC-AUC | recall at p ≥ 0.5 | false alarms on clean | music detections (no music present) |
|---|---|---|---|---|---|
| VoiceBank-DEMAND test, 824 noisy vs 824 clean twins | noise | 0.992 | 0.60 | 0.0 % | 13.9 % of clips (v3.0: 21.1 %) |
| DNS Challenge 2020 synthetic dev-test, 600 clips | noise | 0.998 | 0.91 | 0.0 % | 8.2 % (v3.0: 11.3 %) |
| DNS Challenge 2020 synthetic dev-test, 600 clips | reverberation | 1.000 | 1.00 | 0.0 % | — |
| Dynamic-SUPERB noise detection (VCTK + MUSAN) | noise | 0.992 | 0.96 | 2.2 % | 0.5 % |
| Dynamic-SUPERB reverberation, small / medium / large room | reverberation | 0.994 / 1.000 / 0.996 | 0.98 / 1.00 / 0.97 | 0 % | 0–1.5 % |

Ranking is near perfect; at the default threshold the model is conservative on
mild noise (VoiceBank-DEMAND SNR 2.5–17.5 dB), so lower the threshold when
filtering data.

**Held-out synthetic test sets** (unseen speakers, noises and rooms; up to three
overlapping degradations with random onsets):

| test set | frame macro-AP (20 ms) | 1-s segment macro-F1 | clip macro-AUC |
|---|---|---|---|
| English (LibriSpeech test-clean), 3 000 × 8 s | 0.955 | 0.905 | 0.989 |
| Russian (Russian LibriSpeech test), 3 000 × 8 s | 0.960 | 0.881 | 0.987 |
| English, real measured RIRs | 0.954 | 0.902 | 0.991 |
| English, MIT IR survey RIRs | 0.946 | 0.883 | 0.990 |

Frame AP per class (English / Russian): pitch shift 0.999 / 0.993, time stretch 0.999 / 0.994, low-pass 0.996 / 0.994, clipping 0.993 / 0.997, packet loss 0.991 / 0.992, high-pass 0.991 / 1.000, music 0.985 / 0.990, reverberation 0.985 / 0.991, clicks 0.976 / 0.979, codec 0.925 / 0.961, background noise 0.898 / 0.799, mains hum 0.726 / 0.825.

## Training

* **Speech**: LibriSpeech train-clean-100 (English, 100 h) and Russian LibriSpeech
  train (93 h); validation and test speakers are disjoint.
* **Degradations**, generated on the fly in 8-s windows, 0–3 events per window,
  whole-window or 0.5–4 s intervals with random onsets, fixed signal-chain order:
  MUSAN noise (−5…30 dB SNR) and music (0…25 dB), RIRS_NOISES room impulse responses,
  hard / soft clipping (2–30 % samples), Butterworth or resampling low-pass (1–5 kHz),
  high-pass (200–1000 Hz), MP3 8–41 kbit/s, Opus 10–32 kbit/s, Vorbis, GSM, µ-law,
  A-law, G.721, IMA-ADPCM, Gilbert–Elliott packet loss (5–30 %), 50/60 Hz hum,
  clicks, pitch shift (±2–8 semitones), time stretch (0.7–0.9, 1.15–1.5).
  Labels are per 20 ms; frames where a degradation is physically absent are masked.
* **Model**: Whisper-small encoder, learnable weighted sum of all 13 hidden states,
  two 1-D convolutions and a linear head, 12 sigmoids per frame.
* **Recipe**: AdamW, lr 3e-5 (encoder) / 1e-3 (head), 1 000 warm-up steps, cosine,
  bf16, batch 64 windows, 30 000 steps (≈ 4 300 h of generated audio), one H100,
  2.4 h. Best validation checkpoint at step 16 000 (frame macro-AP 0.969).
## Files in this repository
| file | model | size |
|---|---|---|
| `aadd_v3_1_whisper_small_frames.pt` | **v3.1** (default `v3`): 12 realistic classes every 20 ms (this card) | 177 MB |
| `aadd_v3_whisper_small_frames.pt` | v3.0: the first release of v3 | 177 MB |
| `aadd_v2_efficientnet_b0_mel.pt` | v2: EfficientNet-B0 on log-mel, 7 synthetic classes per second | 16 MB |
| `aadd_v1_efficientnet_b0_png.pt` | v1: EfficientNet-B0 on rendered mel images, 7 synthetic classes per second | 16 MB |
| `eval/` | evaluation reports, training logs, speed benchmark | — |

All are loaded the same way: `Detector("v1" | "v2" | "v3")`, or `Detector(path)`.

### v1 and v2

Earlier, smaller models for seven synthetic `audiomentations` degradations
(Gaussian noise, pitch shift, time stretch, circular shift, clipping, low-pass
150–750 Hz, high-pass 2–8 kHz) on 1-s chunks of English read speech; trained on
LibriSpeech dev-clean with one random degradation per second (p = 0.5). Inputs:
v1 renders a 128-band mel spectrogram to a 310×385 PNG with matplotlib (kept
bit-exact in the package), v2 uses the log-mel tensor directly (1×128×63).
CPU-friendly (4 M parameters).

Regenerated LibriSpeech test-clean set, 108 936 one-second chunks, threshold 0.5:

| | v1 | v2 |
|---|---|---|
| F1 macro (8 outputs) | 0.831 | **0.919** |
| F1 micro | 0.908 | **0.930** |
| exact match per second | 0.906 | **0.927** |
| file-level Spearman (true vs predicted number of degraded seconds) | 0.932 | **0.938** |
| CPU, ms per second of audio (Apple M3 Pro) | 133 | **39** |

Per-class F1 (v1 / v2): Gaussian noise 1.000 / 0.999, pitch shift 0.934 / 0.912,
time stretch 0.941 / 0.968, circular shift 0.000 / 0.726, clipping 0.864 / 0.818,
low-pass 0.991 / 0.997, high-pass 0.992 / 0.992, none 0.926 / 0.942. These models
only know their seven synthetic classes; for real recordings use v3.

## Limitations

* Degradations are synthetic. Real-world tests above cover noise and reverberation;
  the other classes are validated only on held-out synthetic data.
* The "clean" training speech comes from LibriVox audiobooks, which themselves
  contain low-level noise, hum and MP3 history. The model therefore flags some of it
  (22 % of English and 44 % of Russian synthetic "clean" windows have a frame above
  0.5 with v3.1, which is more sensitive to noise than v3.0), which caps the measured AP for noise and hum. On studio-clean speech
  (VoiceBank clean) the false-alarm rate is 0.0 % (DNS clean: 0.7 %).
* Read speech only; music, far-field meetings and TTS output were not evaluated.
* Inputs are resampled to 16 kHz, so content above 8 kHz is not seen.

## Licence and data

Weights: MIT. Base model: Whisper-small (Apache 2.0). Training data: LibriSpeech
(CC BY 4.0), Russian LibriSpeech (CC BY 4.0 card, public-domain source recordings),
MUSAN (CC BY 4.0), RIRS_NOISES (Apache 2.0). Evaluation data: MIT IR survey,
VoiceBank-DEMAND, DNS Challenge 2020, Dynamic-SUPERB.

	---
	license: mit
	language:
	- en
	- ru
	library_name: pytorch
	pipeline_tag: audio-classification
	base_model: openai/whisper-small
	tags:
	- audio
	- speech
	- audio-quality
	- degradation-detection
	- data-curation
	- sound-event-detection
	- whisper
	datasets:
	- openslr/librispeech_asr
	- istupakov/russian_librispeech
	metrics:
	- average_precision
	- roc_auc
	---

	# AADD v3.1: which audio degradation, and where

	[![GitHub](https://img.shields.io/badge/GitHub-gertelrina%2Faadd-181717?logo=github)](https://github.com/gertelrina/aadd)
	[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://github.com/gertelrina/aadd/blob/main/LICENSE)

	Code, CLI, Python API and interactive dataset report: [github.com/gertelrina/aadd](https://github.com/gertelrina/aadd).

	AADD (Automatic Audio Degradation Detection) finds **12 kinds of degradation in
	speech recordings every 20 ms**: background noise, music, reverberation, clipping,
	low-pass / narrow band, high-pass, codec artefacts, packet loss, mains hum, clicks,
	pitch shift and time stretch. Several can be present at once. It is meant for
	cleaning speech datasets (ASR, TTS) and for auditing recordings.

	<p align="center"><img src="demo.gif" width="860" alt="AADD v3 localising noise, reverberation, clipping, codec artefacts and clicks in a 14-second clip"></p>

	* Backbone: Whisper-small encoder (88 M parameters), fine-tuned end to end
	* Output: one sigmoid per class per 20 ms frame on 8-s windows; any length via sliding windows
	* Languages seen in training: English and Russian

	## What changed in v3.1 (2026-11-06)

	v3.0 called real background babble (cafés, streets, buses) "music". v3.1 is v3.0
	fine-tuned for 4 000 steps with 36 h of real environmental noise from FSD50K
	(CC0 / CC BY clips only, no music or voice labels) and synthetic crowd babble
	added to the noise class. On the VoiceBank-DEMAND test set, music detections per
	noise type (v3.0 → v3.1): café 41 % → 16 %, public square 49 % → 25 %, bus 39 % →
	18 %, office 5 % → 4 %; noise recall rose (café 84 % → 95 %, square 55 % → 78 %).
	The DEMAND "living room" noise keeps 86 % music: it contains sustained harmonic
	tones (music / TV), so this is probably correct. Synthetic held-out tests are
	unchanged or better (frame macro-AP 0.955 / 0.960 en / ru).

	`Detector("v3")` and `--model v3` now load v3.1; `v3.0` stays available.

	## Usage

	```bash
	pip install "aadd @ git+https://github.com/gertelrina/aadd"

	aadd scan my_dataset/ --model v3 --report report.html # downloads this checkpoint on first use
	aadd scan call.wav --model v3 --resolution 0.1 --segments per_100ms.csv
	```

	```python
	from aadd import Detector

	det = Detector("v3", resolution=1.0) # per-second summary; frame probabilities in res.frame_probs
	res = det.predict_file("call.wav")
	res.detected # {'noise': 0.97, 'reverb': 0.88}
	res.frame_probs.shape # (n_frames_20ms, 12)
	```

	Or download the file directly:

	```python
	from huggingface_hub import hf_hub_download
	path = hf_hub_download("agertel/aadd", "aadd_v3_1_whisper_small_frames.pt")
	det = Detector(path)
	```

	## Results

	Out-of-domain, real recordings (never seen in training). Clip score = 95th
	percentile of frame probabilities.

	\| set \| class \| ROC-AUC \| recall at p ≥ 0.5 \| false alarms on clean \| music detections (no music present) \|
	\|---\|---\|---\|---\|---\|---\|
	\| VoiceBank-DEMAND test, 824 noisy vs 824 clean twins \| noise \| 0.992 \| 0.60 \| 0.0 % \| 13.9 % of clips (v3.0: 21.1 %) \|
	\| DNS Challenge 2020 synthetic dev-test, 600 clips \| noise \| 0.998 \| 0.91 \| 0.0 % \| 8.2 % (v3.0: 11.3 %) \|
	\| DNS Challenge 2020 synthetic dev-test, 600 clips \| reverberation \| 1.000 \| 1.00 \| 0.0 % \| — \|
	\| Dynamic-SUPERB noise detection (VCTK + MUSAN) \| noise \| 0.992 \| 0.96 \| 2.2 % \| 0.5 % \|
	\| Dynamic-SUPERB reverberation, small / medium / large room \| reverberation \| 0.994 / 1.000 / 0.996 \| 0.98 / 1.00 / 0.97 \| 0 % \| 0–1.5 % \|

	Ranking is near perfect; at the default threshold the model is conservative on
	mild noise (VoiceBank-DEMAND SNR 2.5–17.5 dB), so lower the threshold when
	filtering data.

	Held-out synthetic test sets (unseen speakers, noises and rooms; up to three
	overlapping degradations with random onsets):

	\| test set \| frame macro-AP (20 ms) \| 1-s segment macro-F1 \| clip macro-AUC \|
	\|---\|---\|---\|---\|
	\| English (LibriSpeech test-clean), 3 000 × 8 s \| 0.955 \| 0.905 \| 0.989 \|
	\| Russian (Russian LibriSpeech test), 3 000 × 8 s \| 0.960 \| 0.881 \| 0.987 \|
	\| English, real measured RIRs \| 0.954 \| 0.902 \| 0.991 \|
	\| English, MIT IR survey RIRs \| 0.946 \| 0.883 \| 0.990 \|

	Frame AP per class (English / Russian): pitch shift 0.999 / 0.993, time stretch 0.999 / 0.994, low-pass 0.996 / 0.994, clipping 0.993 / 0.997, packet loss 0.991 / 0.992, high-pass 0.991 / 1.000, music 0.985 / 0.990, reverberation 0.985 / 0.991, clicks 0.976 / 0.979, codec 0.925 / 0.961, background noise 0.898 / 0.799, mains hum 0.726 / 0.825.

	## Training

	* Speech: LibriSpeech train-clean-100 (English, 100 h) and Russian LibriSpeech
	train (93 h); validation and test speakers are disjoint.
	* Degradations, generated on the fly in 8-s windows, 0–3 events per window,
	whole-window or 0.5–4 s intervals with random onsets, fixed signal-chain order:
	MUSAN noise (−5…30 dB SNR) and music (0…25 dB), RIRS_NOISES room impulse responses,
	hard / soft clipping (2–30 % samples), Butterworth or resampling low-pass (1–5 kHz),
	high-pass (200–1000 Hz), MP3 8–41 kbit/s, Opus 10–32 kbit/s, Vorbis, GSM, µ-law,
	A-law, G.721, IMA-ADPCM, Gilbert–Elliott packet loss (5–30 %), 50/60 Hz hum,
	clicks, pitch shift (±2–8 semitones), time stretch (0.7–0.9, 1.15–1.5).
	Labels are per 20 ms; frames where a degradation is physically absent are masked.
	* Model: Whisper-small encoder, learnable weighted sum of all 13 hidden states,
	two 1-D convolutions and a linear head, 12 sigmoids per frame.
	* Recipe: AdamW, lr 3e-5 (encoder) / 1e-3 (head), 1 000 warm-up steps, cosine,
	bf16, batch 64 windows, 30 000 steps (≈ 4 300 h of generated audio), one H100,
	2.4 h. Best validation checkpoint at step 16 000 (frame macro-AP 0.969).

	## Files in this repository

	\| file \| model \| size \|
	\|---\|---\|---\|
	\| `aadd_v3_1_whisper_small_frames.pt` \| v3.1 (default `v3`): 12 realistic classes every 20 ms (this card) \| 177 MB \|
	\| `aadd_v3_whisper_small_frames.pt` \| v3.0: the first release of v3 \| 177 MB \|
	\| `aadd_v2_efficientnet_b0_mel.pt` \| v2: EfficientNet-B0 on log-mel, 7 synthetic classes per second \| 16 MB \|
	\| `aadd_v1_efficientnet_b0_png.pt` \| v1: EfficientNet-B0 on rendered mel images, 7 synthetic classes per second \| 16 MB \|
	\| `eval/` \| evaluation reports, training logs, speed benchmark \| — \|

	All are loaded the same way: `Detector("v1" \| "v2" \| "v3")`, or `Detector(path)`.

	### v1 and v2

	Earlier, smaller models for seven synthetic `audiomentations` degradations
	(Gaussian noise, pitch shift, time stretch, circular shift, clipping, low-pass
	150–750 Hz, high-pass 2–8 kHz) on 1-s chunks of English read speech; trained on
	LibriSpeech dev-clean with one random degradation per second (p = 0.5). Inputs:
	v1 renders a 128-band mel spectrogram to a 310×385 PNG with matplotlib (kept
	bit-exact in the package), v2 uses the log-mel tensor directly (1×128×63).
	CPU-friendly (4 M parameters).

	Regenerated LibriSpeech test-clean set, 108 936 one-second chunks, threshold 0.5:

	\| \| v1 \| v2 \|
	\|---\|---\|---\|
	\| F1 macro (8 outputs) \| 0.831 \| 0.919 \|
	\| F1 micro \| 0.908 \| 0.930 \|
	\| exact match per second \| 0.906 \| 0.927 \|
	\| file-level Spearman (true vs predicted number of degraded seconds) \| 0.932 \| 0.938 \|
	\| CPU, ms per second of audio (Apple M3 Pro) \| 133 \| 39 \|

	Per-class F1 (v1 / v2): Gaussian noise 1.000 / 0.999, pitch shift 0.934 / 0.912,
	time stretch 0.941 / 0.968, circular shift 0.000 / 0.726, clipping 0.864 / 0.818,
	low-pass 0.991 / 0.997, high-pass 0.992 / 0.992, none 0.926 / 0.942. These models
	only know their seven synthetic classes; for real recordings use v3.

	## Limitations

	* Degradations are synthetic. Real-world tests above cover noise and reverberation;
	the other classes are validated only on held-out synthetic data.
	* The "clean" training speech comes from LibriVox audiobooks, which themselves
	contain low-level noise, hum and MP3 history. The model therefore flags some of it
	(22 % of English and 44 % of Russian synthetic "clean" windows have a frame above
	0.5 with v3.1, which is more sensitive to noise than v3.0), which caps the measured AP for noise and hum. On studio-clean speech
	(VoiceBank clean) the false-alarm rate is 0.0 % (DNS clean: 0.7 %).
	* Read speech only; music, far-field meetings and TTS output were not evaluated.
	* Inputs are resampled to 16 kHz, so content above 8 kHz is not seen.

	## Licence and data

	Weights: MIT. Base model: Whisper-small (Apache 2.0). Training data: LibriSpeech
	(CC BY 4.0), Russian LibriSpeech (CC BY 4.0 card, public-domain source recordings),
	MUSAN (CC BY 4.0), RIRS_NOISES (Apache 2.0). Evaluation data: MIT IR survey,
	VoiceBank-DEMAND, DNS Challenge 2020, Dynamic-SUPERB.