laion/neural-audio-fingerprint

Neural audio fingerprint

Maps every 1-second window of 16 kHz mono audio to a 128-d L2-normalised embedding. A window and a transformed copy of the same recording score a high cosine similarity. The model is trained to see through re-encoding (MP3, AAC, Opus, Vorbis), resampling, trimming and offsets, gain changes, filtering, reverb, small speed changes and mixing with other audio, while staying low between different recordings.

Intended use: finding duplicate or leaked audio in large collections, for example screening a training corpus for copies of evaluation-set recordings, deduplicating datasets, or tracing re-uploads. Treat a hit as a candidate. When precision matters, confirm it with waveform cross-correlation.

Model

• Input: 1 s at 16 kHz → 64-band log-mel (25 ms window, 10 ms hop, 30 Hz–8 kHz), divided by the window's mean energy, so absolute level does not matter.
• Encoder (80.3M parameters, 3.9 GFLOP per window): patches of 4 frames (25 tokens), a 6-layer pre-LN transformer (width 1024, 16 heads), mean + max pooling and a 2-layer MLP to 128-d.
• Matching (embed.py): query windows at a 0.5 s hop, reference windows at a 0.25 s hop, and windows more than 60 dB below a file's loudest window are dropped. A match is two consecutive query windows with cosine ≥ τ against the same reference at a consistent offset (±0.25 s). For a reference of 1.5 s or less, one window is enough. τ = 0.90 is the default.

Files

file	contents
model.pt	weights and architecture config (torch.load(..., weights_only=True))
model.py	the architecture and load()
embed.py	file → window embeddings, plus a match test between two files

Usage

pip install torch torchaudio soundfile soxr
python embed.py query.mp3 reference.flac
# max cosine 0.986; 26/26 query windows >= 0.9; match: True at offset +3.25 s

from model import load
from embed import embed, read_16k

model = load("model.pt")
t, e = embed(model, read_16k("clip.flac"), hop=0.5)   # start times (s), (N, 128) embeddings

Training

• Data: 301,056 clips of web and public-dataset audio, one excerpt of up to 6 s each, sampled so that the 11 broad sound groups are balanced (speech 22 %, music 24 %, animals 14 %, designed sound effects 10 %, the other seven groups 3–5 % each). 4,704 clips are held out for validation.
• Codec bank: each clip was also rendered once with a random resample rate (8–48 kHz) and, with probability 0.85, a lossy codec: MP3, AAC, Opus or Vorbis at a log-uniform bitrate of 24–320 kbps. The rendering was then decoded and time-aligned to the clean version.
• Views: each positive pair is two 1-s views of the same clip. The anchor is mostly clean and lightly augmented. The query is mostly codec-rendered, shifted by up to ±250 ms and sped up or slowed by up to ±3 %. It is then augmented on the GPU with EQ, low- and high-pass filters, synthetic reverb, mixing with another clip at −5 to +20 dB SNR, white or pink noise at 10–40 dB SNR, gain of ±12 dB and int16 quantisation.
• Objective: NT-Xent (temperature 0.05), 16,384 pairs per step, with negatives gathered across 4 GPUs. AdamW (lr 3e-4, weight decay 0.05), 1,500 warm-up steps, then cosine decay over 35,858 steps: 3.3 h on 4 GH200 GPUs.

Evaluation

Held-out retrieval (4,704 validation clips). Clean windows of every clip, at a 0.25 s hop, form the database. 2,048 transformed 1-s queries are searched against it, and a query counts as correct when the top hit is the same clip within ±0.5 s. Top-1 accuracy is 99.1 % with codecs only, 92.3 % with the full training augmentation and 74.6 % when the query is mixed 1:1 (0 dB) with another clip.

Copy detection vs. GraFPrint. The reference set has 30,189 recordings from public audio benchmarks. 65,518 unrelated clips (425 h) are used to measure false positives. Positives are 6,491 real copies, known copies of benchmark recordings found in web and public-dataset collections by ID or identical waveform (including mono mixdowns and re-encodes), and 1,800 synthetic copies (18 transforms × 100). Hard negatives are 1,250 pairs of different recordings with the same label, word or genre. The query hop is 1 s. Each model's τ is set so that an unrelated 1-s window crosses τ against a given reference with probability 10⁻⁷: τ = 0.845 for this model and 0.975 for GraFPrint (public checkpoint, trained on music). Numbers are the % of pairs detected.

	this model	GraFPrint
real copies (6,491)	99.6	90.6
synthetic copies (1,800)	85.9	55.2
hard negatives flagged (1,250, lower is better)	4.4	3.7
— exact copy / MP3 64 / MP3 128 kbps	99 / 100 / 100	100 / 100 / 100
— AAC 96 / Opus 32 kbps	100 / 97	29 / 78
— resample 44.1 kHz / gain −6 dB / trim 0.5 s	100 / 99 / 100	100 / 97 / 100
— speed 1 % / reverb / low-pass 4 kHz	100 / 99 / 88	52 / 70 / 21
— 3-s excerpt / embedded in a 30-s clip	97 / 79	8 / 12
— mixed at 20 / 10 / 5 / 0 / −5 dB SNR	96 / 88 / 62 / 23 / 15	46 / 22 / 8 / 1 / 7

On this benchmark, at τ = 0.90, the false-positive probability per (unrelated window, reference) is 2.1 × 10⁻⁸.

Full-scale check. The reference set grows to 319,848 benchmark recordings (1,281 h), with a 0.5 s query hop and τ = 0.92, the lowest score stored in that run. Recall at τ = 0.90 is at least as high.

• The false-positive probability per (unrelated window, reference) is 4.5 × 10⁻⁸.
• Real copies: 99.5 % of 10,617 pairs detected. Synthetic copies: 84.5 % of 2,400.
• A separate set of 47,141 real copy pairs: 94.5 % detected.
• A separate set of 75,532 synthetic copies: 84.6 % detected, of which mixed at 20 / 10 / 5 / 0 / −5 dB: 96 / 83 / 56 / 26 / 16 %.
• 8,000 random unrelated clips: no detections.

Limitations

• Copies mixed under other audio at ≤ 5 dB SNR are mostly missed (see the table).
• Short tonal or percussive sounds of different recordings can look alike. 20 % of same-stroke mridangam pairs and 20 % of same-preset NSynth notes were flagged as matches.
• Speed changes beyond ±3 %, pitch shifts and time-stretching were not part of training.
• Quiet windows are dropped, so silence and near-silence never match.

Licence

Apache-2.0.