File size: 4,391 Bytes
c970736
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
# Nes2Net-X-CtrSVDD β€” implementation notes

Plan: `speech-spoof-bench/docs/plans/2026-11-07-eleven-model-batch-arena-submission.md` (model #11).
**Singing-voice** deepfake detector (CtrSVDD / SVDD 2024); the speech Nes2Net is the separate `benchmarks/Nes2Net/` row.

## Source
- Code: github Liu-Tianchi/Nes2Net @ `3fed08a603991cf8b422978e0c942a976deb9399` (Apache-2.0),
  `models/WavLM_Nes2Net_X.py` class `WavLM_Nes2Net_noRes_w_allT`; `eval.sh`: `--agg SEA --pool_func mean --dilation 1
  --Nes_ratio 8 8 --SE_ratio 1`; `eval.py` (full utterance, bs 1, `model.eval()`, filtered non-strict load).
- Weights: HF nielsr/nes2net-checkpoints @ `06818cabb70c44d3008fdeb9285f141f198aa92a`,
  `WavLM_Nes2Net_X_e75_seed420_valid0.03192785031473534.pt` (1,264,329,467 B, sha256
  `f1c2942635c68b584a742b10dbc5630759fc6cd55409ff8dc4d71f9c81ccdf5f`) β€” an HF mirror (by nielsr) of the authors'
  Google-Drive release (README row WavLM_Nes2Net_X seed 420 epoch 75). No license β†’ **not re-hosted**, linked; local
  symlink into `benchmarks/_ckpt_dl/nielsr__nes2net-checkpoints/`.
- Front-end config: `cfg` of s3prl/converted_ckpts @ `8cad0b370e7e35f8d56951d95d2be036ea85510c` `wavlm_large.pt`
  (sha256 `6fb4b3c3…100f`, read with weights_only=True) β†’ `wavlm_large_cfg.json`. `normalize: true`.

## Port (`_net.py`)
- s3prl 0.4.18 `WavLM`/`WavLMConfig` (installed package) built from the vendored cfg; no base weights or hub
  download. `_UpstreamExpert` stands in for s3prl's UpstreamExpert (keeps `ssl_model.model.model.*` keys):
  per-clip `F.layer_norm(wav, (L,))`, `extract_features(mask=False)`, 25 hidden states via forward pre-hooks on the
  24 encoder layers (= s3prl hook `input[0].transpose(0,1)`) + forward hook on the encoder (`output[0]`, after the
  final LayerNorm). padding_mask=None (s3prl passes all-False for equal lengths; identical, see parity).
  `int(shape)` in layer_norm keeps the ONNX trace static.
- SEA merge (AdaptiveAvgPool2d β†’ Linear 25β†’8 β†’ ReLU β†’ Linear 8β†’25 β†’ Sigmoid β†’ weighted sum) and the Nes2Net-X
  back-end verbatim; fc Linear(1024,1). Training-only branches (AttM/WeightedSum/ASTP, SSL_freeze) dropped.
- Checkpoint: plain OrderedDict, 1050 tensors (488 WavLM), `torch.load(weights_only=True)`, `strict=True`
  0 missing / 0 unexpected (the authors' filtered load also drops/misses nothing).
- Score = the logit (focal-BCE training, label 1 = bonafide); higher = bona fide.
- Params 315,964,534 (WavLM-Large 315,453,120 + SEA 400 + back-end 511,014).

## Input (maintainer: 4 s cap)
Authors evaluate the full utterance (their `4s` mode = first 64000, repeat-padded). Here first **64600** samples
(Arena rule), tile-repeat if shorter, empty β†’ zeros. FP32, cuDNN TF32 off.

## Source parity (`source_parity.py`, log `source_parity.log`)
Authors' `WavLM_Nes2Net_X.py` with `hub.wavlm_large` patched to s3prl `wavlm_local(<local wavlm_large.pt>)` (real
UpstreamExpert with hooks + all-False padding mask), authors' filtered load, vs our wrapper on 24 In-the-Wild clips
(same window): max |Ξ”| = 0 single-clip, 3.1e-6 batched.

## Batch sweep (`sweep.py`, 2026-11-07, logs `logs_sweep_g{2,1}.txt`)
| card | best | notes |
|---|---|---|
| RTX 4070 Ti SUPER (idx 2) | 114.6 utt/s @24 | plateau 16–64 β†’ `batch_size = 24` |
| RTX 5060 Ti (idx 1) | 43.7 utt/s @4 | ~31 at bs β‰₯ 8 β†’ 4 on 5060 |

## Sanity (PyTorch)
Full J-SPAW_LA (`results_pt/`, RTX 5060 Ti idx 3): EER **1.440 %** (2397 trials, 0 skipped) β€” surprisingly strong
for a singing model; not inverted. Copied to `results/J-SPAW_LA`. Unit tests `test_nes2net_x_ctrsvdd.py`: 8 passed.

## TensorRT (`trt_nes2net_x_ctrsvdd.py`, sidecar `trt_nes2net-x-ctrsvdd.json`, TRT 10.16.1.11)
FP16 produces NaN (WavLM overflow; build Spearman 0.40) β†’ FP32 on both cards; FP16 engines deleted.
| card | precision | best batch | utt/s (sweep) | build parity Spearman / max Ξ” | J-SPAW EER | J-SPAW Spearman vs PT / max Ξ” |
|---|---|---|---|---|---|---|
| RTX 4070 Ti SUPER (idx 2) | FP32 | 32 | 167.1 | 1.00000 / 0.0027 | 1.4402 (=PT) | 1.000000 / 0.0053 |
| RTX 5060 Ti (idx 3) | FP32 | 32 | 113.3 | 1.00000 / 0.022 | 1.4402 (=PT) | 0.999999 / 0.013 |

Scripts: `run_sweep.sh`, `run_pt.sh [gpu]`, `run_trt_export.sh ["3 2"]`, `run_trt_jspaw.sh <gpu>`, `cmp_scores.py`.

## meta.yaml
trained_on omitted (CtrSVDD only). Self-reported CtrSVDD test EER 2.48 % (2.22 % w/ ACESinger bona fide).