File size: 3,641 Bytes
e187f2f
 
 
 
 
 
 
 
 
 
45294c1
 
e187f2f
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
---
license: cc-by-nc-3.0
library_name: pytorch
tags:
- audio
- query-by-example
- sound-event-detection
- next-lab
---

![echo](https://cdn-uploads.huggingface.co/production/uploads/67d46bc5fe6ad6f6511d6f44/MT-kCSyCuZW3B70EcsR1y.png)

# Next Echo

Next Echo finds **every moment a given example sound occurs** in a longer recording — a clap, a knock, a bark — with no
fixed class list. A small CNN embeds 1-second log-mel windows; the recording is scanned against the example.
It is compared with plain log-mel similarity; the final score fuses both after per-recording normalisation.

**Space:** https://huggingface.co/spaces/thelamapi/Next-Echo — record a 2-second example with your microphone (or pick one), choose a recording, and listen to each hit. Runs in the browser.

## Results on constructed 30 s test recordings (ESC-50 fold 5; 40 recordings per cell)

Event F1 (±0.5 s) / false alarms per minute — log-mel baseline vs Next Echo (better F1 in bold):

| Classes / condition | Log-mel similarity | Next Echo (learned) |
|---|---:|---:|
| seen/clean | **0.676** / 2.15 | 0.619 / 3.25 |
| seen/noisy | 0.606 / 1.45 | **0.630** / 2.30 |
| seen/overlap | **0.788** / 0.80 | 0.749 / 1.05 |
| seen/absent | – / 4.00 | – / 4.95 |
| unseen/clean | 0.617 / 2.10 | **0.658** / 3.30 |
| unseen/noisy | 0.667 / 1.10 | **0.717** / 2.05 |
| unseen/overlap | **0.790** / 0.85 | 0.767 / 1.70 |
| unseen/absent | – / 3.65 | – / 5.60 |
| seen/all-present | **0.690** / 1.47 | 0.662 / 2.20 |
| unseen/all-present | 0.691 / 1.35 | **0.712** / 2.35 |

*unseen* = the 10 classes never used in training (clapping, door_wood_knock, rooster, church_bells, glass_breaking, keyboard_typing, siren, cow, sneezing, can_opening). *absent* recordings contain no target, so
only false alarms count. Thresholds were chosen on validation recordings built from fold 4.

![Example timeline and benchmark](preview.png)

## Method
* Audio: 16 kHz mono, 512-point STFT (hop 256), 64 mel bands (50–8000 Hz), log.
* Embedding: 4 conv blocks + linear → 64-d, supervised contrastive loss (τ = 0.1) on random 1-s crops of 960 clips from 40 classes, with gain, mixing and noise augmentation, 5000 steps.
* Training windows mimic the test conditions: real waveform segments mixed with another sound and brown noise (0–30 dB SNR), alongside clean segments like the user's example.
* Query: the loudest 1-s windows of the example are embedded and averaged. Recording: windows every 0.256 s.
* Score: per-recording robust normalisation (median/MAD), then Next Echo = α·z(learned) + (1−α)·z(log-mel) with α = 0.75 chosen on validation recordings
  (options α ∈ {1, 0.75, 0.5}). Consecutive windows above the threshold form one detection.
* Baseline: cosine between per-band mean/std of log-mel windows, raw or normalised — whichever validated better (raw).
* The browser computes the same log-mel (own FFT) and CNN with `nlrt.js`; Node parity on 3 real snippets: max abs difference 7.105427357601002e-15.

## Usage
```bash
python inference.py --query clap.wav --recording scene.wav     # 16 kHz mono 16-bit WAV
python inference.py --selftest
```

## Limitations and license
* Recordings are mixtures of real clips, not naturally recorded scenes; real rooms add reverberation and far more varied backgrounds.
* Very short or very quiet examples give weak queries. Sounds that resemble each other (knock vs footsteps) cause false alarms.
* **Non-commercial:** ESC-50 is CC BY-NC 3.0, so this model, its demo audio and the Space are for non-commercial use only.
* Trained on NVIDIA L4; profile full. Failure cases are listed in `evaluation.json`.