Nes2Net-X-CtrSVDD / implementation-notes.md
korallll's picture
Wrapper code (speech-spoof-bench)
c970736 verified
|
Raw History Blame Contribute Delete
4.39 kB

Nes2Net-X-CtrSVDD β€” implementation notes

Plan: speech-spoof-bench/docs/plans/2026-11-07-eleven-model-batch-arena-submission.md (model #11). Singing-voice deepfake detector (CtrSVDD / SVDD 2024); the speech Nes2Net is the separate benchmarks/Nes2Net/ row.

Source

  • Code: github Liu-Tianchi/Nes2Net @ 3fed08a603991cf8b422978e0c942a976deb9399 (Apache-2.0), models/WavLM_Nes2Net_X.py class WavLM_Nes2Net_noRes_w_allT; eval.sh: --agg SEA --pool_func mean --dilation 1 --Nes_ratio 8 8 --SE_ratio 1; eval.py (full utterance, bs 1, model.eval(), filtered non-strict load).
  • Weights: HF nielsr/nes2net-checkpoints @ 06818cabb70c44d3008fdeb9285f141f198aa92a, WavLM_Nes2Net_X_e75_seed420_valid0.03192785031473534.pt (1,264,329,467 B, sha256 f1c2942635c68b584a742b10dbc5630759fc6cd55409ff8dc4d71f9c81ccdf5f) β€” an HF mirror (by nielsr) of the authors' Google-Drive release (README row WavLM_Nes2Net_X seed 420 epoch 75). No license β†’ not re-hosted, linked; local symlink into benchmarks/_ckpt_dl/nielsr__nes2net-checkpoints/.
  • Front-end config: cfg of s3prl/converted_ckpts @ 8cad0b370e7e35f8d56951d95d2be036ea85510c wavlm_large.pt (sha256 6fb4b3c3…100f, read with weights_only=True) β†’ wavlm_large_cfg.json. normalize: true.

Port (_net.py)

  • s3prl 0.4.18 WavLM/WavLMConfig (installed package) built from the vendored cfg; no base weights or hub download. _UpstreamExpert stands in for s3prl's UpstreamExpert (keeps ssl_model.model.model.* keys): per-clip F.layer_norm(wav, (L,)), extract_features(mask=False), 25 hidden states via forward pre-hooks on the 24 encoder layers (= s3prl hook input[0].transpose(0,1)) + forward hook on the encoder (output[0], after the final LayerNorm). padding_mask=None (s3prl passes all-False for equal lengths; identical, see parity). int(shape) in layer_norm keeps the ONNX trace static.
  • SEA merge (AdaptiveAvgPool2d β†’ Linear 25β†’8 β†’ ReLU β†’ Linear 8β†’25 β†’ Sigmoid β†’ weighted sum) and the Nes2Net-X back-end verbatim; fc Linear(1024,1). Training-only branches (AttM/WeightedSum/ASTP, SSL_freeze) dropped.
  • Checkpoint: plain OrderedDict, 1050 tensors (488 WavLM), torch.load(weights_only=True), strict=True 0 missing / 0 unexpected (the authors' filtered load also drops/misses nothing).
  • Score = the logit (focal-BCE training, label 1 = bonafide); higher = bona fide.
  • Params 315,964,534 (WavLM-Large 315,453,120 + SEA 400 + back-end 511,014).

Input (maintainer: 4 s cap)

Authors evaluate the full utterance (their 4s mode = first 64000, repeat-padded). Here first 64600 samples (Arena rule), tile-repeat if shorter, empty β†’ zeros. FP32, cuDNN TF32 off.

Source parity (source_parity.py, log source_parity.log)

Authors' WavLM_Nes2Net_X.py with hub.wavlm_large patched to s3prl wavlm_local(<local wavlm_large.pt>) (real UpstreamExpert with hooks + all-False padding mask), authors' filtered load, vs our wrapper on 24 In-the-Wild clips (same window): max |Ξ”| = 0 single-clip, 3.1e-6 batched.

Batch sweep (sweep.py, 2026-11-07, logs logs_sweep_g{2,1}.txt)

card best notes
RTX 4070 Ti SUPER (idx 2) 114.6 utt/s @24 plateau 16–64 β†’ batch_size = 24
RTX 5060 Ti (idx 1) 43.7 utt/s @4 ~31 at bs β‰₯ 8 β†’ 4 on 5060

Sanity (PyTorch)

Full J-SPAW_LA (results_pt/, RTX 5060 Ti idx 3): EER 1.440 % (2397 trials, 0 skipped) β€” surprisingly strong for a singing model; not inverted. Copied to results/J-SPAW_LA. Unit tests test_nes2net_x_ctrsvdd.py: 8 passed.

TensorRT (trt_nes2net_x_ctrsvdd.py, sidecar trt_nes2net-x-ctrsvdd.json, TRT 10.16.1.11)

FP16 produces NaN (WavLM overflow; build Spearman 0.40) β†’ FP32 on both cards; FP16 engines deleted.

card precision best batch utt/s (sweep) build parity Spearman / max Ξ” J-SPAW EER J-SPAW Spearman vs PT / max Ξ”
RTX 4070 Ti SUPER (idx 2) FP32 32 167.1 1.00000 / 0.0027 1.4402 (=PT) 1.000000 / 0.0053
RTX 5060 Ti (idx 3) FP32 32 113.3 1.00000 / 0.022 1.4402 (=PT) 0.999999 / 0.013

Scripts: run_sweep.sh, run_pt.sh [gpu], run_trt_export.sh ["3 2"], run_trt_jspaw.sh <gpu>, cmp_scores.py.

meta.yaml

trained_on omitted (CtrSVDD only). Self-reported CtrSVDD test EER 2.48 % (2.22 % w/ ACESinger bona fide).