Hugging Face's logo

SpeechAntiSpoofingBenchmarks
/
SpoofSUPERB-XLSR

PyTorch
audio
anti-spoofing
deepfake-detection
speech-anti-spoofing-arena
Model card Files
xet
 Community
SpoofSUPERB-XLSR / implementation-notes.md
korallll's picture
korallll
Wrapper code (speech-spoof-bench)
1919b5b verified
preview code
|
Raw History Blame Contribute Delete
4.67 kB

SpoofSUPERB-XLSR — implementation notes

Plan: speech-spoof-bench/docs/plans/2026-11-07-eleven-model-batch-arena-submission.md (rows 9/10).

Source

  • Code: github.com/issflab/spoof_SUPERB @ 8b457584e4120ba525f720e714d0a4b2ed400d68 (Apache-2.0), cloned to benchmarks/_ckpt_dl/issflab__spoof_SUPERB/src. Ported: spoof_superb/models/linear_head.py (SSLModel, UtteranceLevel, FrameLevel, MeanPooling) + spoof_superb/scoring/backends.py (pad, head loader, logits[:,1]) → _net.py.
  • Head: HF issf/spoof-superb-models @ 5245a543… xls_r_300m.pth (1 MB, sha256 0b9c4fb48fcd9ad49d8dc99ba24241bad14c9c7f826791c5cbadd86434741703 = the repo's SHA256SUMS). Plain tensor dict → torch.load(weights_only=True); exactly the 5 trained keys (asserted).
  • Upstream: HF s3prl/converted_ckpts @ 8cad0b370e7e35f8d56951d95d2be036ea85510c xlsr2_300m.pt (sha256 9e998d92419b8b7d6c6768853e14a37ca1b159352801c8cd92afb02e2cef87ea) — the file s3prl 0.4.18's hub entry xls_r_300m downloads. Built with S3PRLUpstream("wav2vec2_local", path_or_url=<local>) (no network); s3prl's internal torch.load is forced to weights_only=True (_net._weights_only); s3prl loads the upstream state dict strict. Configs live inside the ckpt; vendored copy upstream_config.json (test asserts equality). Weight files here are hard links to _ckpt_dl/.

Semantics kept (checked in s3prl 0.4.18 source)

  • Waveform: the upstream's own per-utterance F.layer_norm(wav, wav.shape) (cfg normalize = True) inside the s3prl expert; S3PRLUpstream(normalize=False), Featurizer(normalize=False) as in the authors' code.
  • 25 hidden states = inputs of the 24 transformer layers + encoder output (s3prl hooks); S3PRLUpstream repeats the last frame to len(range(0, T, 320)) = 202 frames for T = 64600 (CNN gives 201); softmax-weighted sum; Linear(1024,256); mean over 202 frames; Linear(256,2). score = logits[:,1] (class 1 = bona fide).
  • Input: first 64600 samples, shorter clips tiled (backends.pad); empty clip → zeros (not in source). No resampling. FP32, cuDNN TF32 off.
  • Params: 317,653,531 (upstream 317,390,592 incl. the pre-training quantizer/project_q/final_proj tensors s3prl loads but never uses for hidden states; head 262,939).

Parity (source_parity.py, source_parity_float.py)

authors' code vs ours max|d| = 0 (32 ITW clips); forward_export vs forward 0; published ITW scores vs ours on the Arena PCM_16 parquet: max 0.661 / mean 0.0785 / corr 0.99877; on the original float wavs: max 0.0044 / mean 0.0010. The authors scored the float In-the-Wild originals; the Arena's InTheWild copy is FLAC PCM_16, and the upstream's waveform layer-norm amplifies the quantisation noise on quiet clips — hence the larger drift on the parquet only.

Batch sweep (sweep.py, 64600 samples, 2026-11-07; logs_sweep_g{2,3}.txt)

4070 Ti S 107.2 utt/s @24 (plateau 24-64, 108.8 @64); 5060 Ti 40.4 @4 (~30 at >= 8) → batch_size = 24, 4 on a 5060 Ti (set in load).

Sanity (PyTorch, RTX 5060 Ti idx 3)

  • Full J-SPAW_LA: EER 8.328% (2397 trials, 0 skipped). Not inverted (bona fide higher in the unit test).

TensorRT (trt_spoofsuperb_xlsr.py, sidecar trt_spoofsuperb-xlsr.json, TRT 10.16.1.11)

Adapted from Forensics-XLSR-Wild's script: _ExportNet calls net.forward_export (batch-vectorised restatement of the s3prl path, identical outputs) and returns (B,2); s3prl's pad_to_multiple added to the patch list. FP16 misses the 0.9999 build Spearman → FP32 on both cards. FP16 engines deleted.

card precision best batch utt/s (sweep) J-SPAW EER (Δ vs PT) J-SPAW Spearman / max Δ
RTX 4070 Ti SUPER (idx 2) FP32 (FP16 build Spearman 0.99963 < 0.9999) 32 173.9 8.2655 (Δ -0.063) 0.999999 / 0.026
RTX 5060 Ti (idx 1) FP32 (FP16 0.99982) 4 66.9 8.3281 (Δ 0) 0.999999 / 0.042

Both cards accepted (|ΔEER| ≤ 0.25, Spearman ≥ 0.999). The 5060 Ti engine was built/run on idx 1 (idx 3 was busy with the PyTorch runs; same card model, the sidecar is keyed by GPU name) → results_trt1/; 4070 → results_trt2/. J-SPAW_LA is submitted from PyTorch scores (results/J-SPAW_LA = copy of results_pt/J-SPAW_LA). Scripts: run_sweep.sh, run_pt.sh, run_trt_export.sh (written for idx 3/2), run_trt_jspaw.sh <gpu>.

meta.yaml

  • Paper arXiv 2603.01482. trained_on: none (ASVspoof 2019 LA train; the Arena's ASVspoof2019_LA is the eval split).
  • checkpoint currently points at the authors' HF head file @ 5245a543; re-host (Apache-2.0) and repoint. The upstream is s3prl's public converted ckpt (referenced in notes, not re-hosted).