File size: 4,671 Bytes
1919b5b | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 | # SpoofSUPERB-XLSR — implementation notes
Plan: `speech-spoof-bench/docs/plans/2026-11-07-eleven-model-batch-arena-submission.md` (rows 9/10).
## Source
- Code: github.com/issflab/spoof_SUPERB @ `8b457584e4120ba525f720e714d0a4b2ed400d68` (Apache-2.0), cloned to
`benchmarks/_ckpt_dl/issflab__spoof_SUPERB/src`. Ported: `spoof_superb/models/linear_head.py` (SSLModel,
UtteranceLevel, FrameLevel, MeanPooling) + `spoof_superb/scoring/backends.py` (pad, head loader, `logits[:,1]`) → `_net.py`.
- Head: HF issf/spoof-superb-models @ `5245a543…` `xls_r_300m.pth` (1 MB, sha256 `0b9c4fb48fcd9ad49d8dc99ba24241bad14c9c7f826791c5cbadd86434741703` = the repo's
SHA256SUMS). Plain tensor dict → `torch.load(weights_only=True)`; exactly the 5 trained keys (asserted).
- Upstream: HF s3prl/converted_ckpts @ `8cad0b370e7e35f8d56951d95d2be036ea85510c` `xlsr2_300m.pt` (sha256 `9e998d92419b8b7d6c6768853e14a37ca1b159352801c8cd92afb02e2cef87ea`) —
the file s3prl 0.4.18's hub entry `xls_r_300m` downloads. Built with `S3PRLUpstream("wav2vec2_local", path_or_url=<local>)`
(no network); s3prl's internal `torch.load` is forced to `weights_only=True` (`_net._weights_only`); s3prl loads the
upstream state dict strict. Configs live inside the ckpt; vendored copy `upstream_config.json` (test asserts equality).
Weight files here are hard links to `_ckpt_dl/`.
## Semantics kept (checked in s3prl 0.4.18 source)
- Waveform: the upstream's own per-utterance `F.layer_norm(wav, wav.shape)` (cfg normalize = True) inside the s3prl expert;
S3PRLUpstream(normalize=False), Featurizer(normalize=False) as in the authors' code.
- 25 hidden states = inputs of the 24 transformer layers + encoder output (s3prl hooks); S3PRLUpstream repeats the last
frame to `len(range(0, T, 320))` = 202 frames for T = 64600 (CNN gives 201); softmax-weighted sum; Linear(1024,256);
mean over 202 frames; Linear(256,2). score = logits[:,1] (class 1 = bona fide).
- Input: first 64600 samples, shorter clips tiled (`backends.pad`); empty clip → zeros (not in source). No resampling. FP32,
cuDNN TF32 off.
- Params: 317,653,531 (upstream 317,390,592 incl. the pre-training quantizer/project_q/final_proj tensors s3prl loads but never uses for hidden states; head 262,939).
## Parity (`source_parity.py`, `source_parity_float.py`)
authors' code vs ours max|d| = 0 (32 ITW clips); forward_export vs forward 0; published ITW scores vs ours on the Arena PCM_16 parquet: max 0.661 / mean 0.0785 / corr 0.99877; on the original float wavs: max 0.0044 / mean 0.0010.
The authors scored the float In-the-Wild originals; the Arena's InTheWild copy is FLAC PCM_16, and the upstream's waveform
layer-norm amplifies the quantisation noise on quiet clips — hence the larger drift on the parquet only.
## Batch sweep (`sweep.py`, 64600 samples, 2026-11-07; `logs_sweep_g{2,3}.txt`)
4070 Ti S 107.2 utt/s @24 (plateau 24-64, 108.8 @64); 5060 Ti 40.4 @4 (~30 at >= 8) → `batch_size = 24`, 4 on a 5060 Ti (set in `load`).
## Sanity (PyTorch, RTX 5060 Ti idx 3)
- Full J-SPAW_LA: EER **8.328%** (2397 trials, 0 skipped). Not inverted (bona fide higher in the unit test).
## TensorRT (`trt_spoofsuperb_xlsr.py`, sidecar `trt_spoofsuperb-xlsr.json`, TRT 10.16.1.11)
Adapted from Forensics-XLSR-Wild's script: `_ExportNet` calls `net.forward_export` (batch-vectorised restatement of the
s3prl path, identical outputs) and returns (B,2); s3prl's `pad_to_multiple` added to the patch list. FP16 misses the
0.9999 build Spearman → FP32 on both cards. FP16 engines deleted.
| card | precision | best batch | utt/s (sweep) | J-SPAW EER (Δ vs PT) | J-SPAW Spearman / max Δ |
|---|---|---|---|---|---|
| RTX 4070 Ti SUPER (idx 2) | FP32 (FP16 build Spearman 0.99963 < 0.9999) | 32 | 173.9 | 8.2655 (Δ -0.063) | 0.999999 / 0.026 |
| RTX 5060 Ti (idx 1) | FP32 (FP16 0.99982) | 4 | 66.9 | 8.3281 (Δ 0) | 0.999999 / 0.042 |
Both cards accepted (|ΔEER| ≤ 0.25, Spearman ≥ 0.999). The 5060 Ti engine was built/run on idx 1 (idx 3 was busy with the
PyTorch runs; same card model, the sidecar is keyed by GPU name) → `results_trt1/`; 4070 → `results_trt2/`.
J-SPAW_LA is submitted from PyTorch scores (`results/J-SPAW_LA` = copy of `results_pt/J-SPAW_LA`).
Scripts: `run_sweep.sh`, `run_pt.sh`, `run_trt_export.sh` (written for idx 3/2), `run_trt_jspaw.sh <gpu>`.
## meta.yaml
- Paper arXiv 2603.01482. trained_on: none (ASVspoof 2019 LA *train*; the Arena's ASVspoof2019_LA is the eval split).
- `checkpoint` currently points at the authors' HF head file @ 5245a543; re-host (Apache-2.0) and repoint. The upstream is
s3prl's public converted ckpt (referenced in notes, not re-hosted).
|