File size: 8,257 Bytes
1c51dd9
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
# Vocoded-VoxCeleb2-SSL-P1 / -P2 / -P3 β€” implementation notes

Plan: `speech-spoof-bench/docs/plans/2026-11-01-nine-model-batch-arena-submission.md` (models #5, #6, #10).
The three dirs share `_net.py` (byte-identical), the wrapper/test/sweep/TRT scripts differ only in the
model ID (and `_TWO_SSL = True` for P2). This file is copied into each dir.

## Source
- Code: github.com/nii-yamagishilab/project-NN-Pytorch-scripts @ `b529705b65588d62db17b56f98636ee734ff725e`
  (BSD-3-Clause, `LICENSE.source`), `project/10-asvspoof-vocoded-trn-ssl/model-ID-{P1,P2,P3}/model.py`,
  `sandbox/block_nn.py` (DropoutForMC), `sandbox/util_bayesian.py` (score), `core_scripts/nn_manager`
  (inference loop: `model.eval()`, `torch.no_grad()`, batch size 1, full utterance).
- Weights: Zenodo record 8337778, `project10-cm-ssl-model-ID-{P1,P2,P3}.tar` β†’
  `model-ID-*/trained-for-paper/01/trained_network.pt` (+ the authors' score logs). Tars listed before
  extraction (only the .pt and 8 score logs, no odd paths); extracted to
  `benchmarks/_ckpt_dl/zenodo__8337778/<ID>/`. The .pt is a plain `OrderedDict` of tensors: read with
  `torch.load(weights_only=True)`, strict-loaded, re-saved **unchanged** (same keys, bit-equal tensors) as
  `model.safetensors` by `convert_ckpt.py`. P1/P3: 443 tensors, 317,571,590 elements; P2: 872 tensors,
  634,962,182 elements (two SSLs). Keys: `m_front_end.ssl_model[2].model.upstream.model.*` (s3prl
  nesting, kept), `m_back_end.*`, and four unused scalars `input_mean/std, output_mean/std` (= 0/1/0/1).
- What the IDs are (paper Β§3.3): v.vox = XLSR-53 continually pre-trained on vocoded VoxCeleb2.
  P1 = single SSL from v.vox; P2 = two SSLs, back-end input = `ssl_model(x) βˆ’ ssl_model2(x)` with
  `ssl_model` = wav2vec2_custom (v.vox) and `ssl_model2` = xlsr_53 (verified in model-ID-P2/model.py);
  P3 = single student SSL initialised from xlsr_53 and distilled from the (xlsr, v.vox) difference. All
  fine-tuned end-to-end on voc.LA (vocoded ASVspoof 2019 LA train bona fide) with RawBoost; 19LA dev for
  early stopping. β†’ `trained_on`: none.

## Front end (s3prl β†’ fairseq, no hub downloads)
- XLSR-53 architecture config: read from the `cfg.model` of fairseq's `xlsr_53_56k.pt` by HTTP-range
  fetching only the first 2 MB (zip entry `data.pkl` comes first; unpickled with stubbed
  persistent_load, no tensors). The vocoded SSL `wav2vec_ft2_vox_vocoded.pt` (Zenodo 8336949, s3prl
  converted format) was checked the same way (first 4 MB): identical architecture (only the
  conv-layer string is spelled differently; quantizer_depth/factor = defaults). Both task cfgs have
  `normalize: true`. Not the XLS-R 300M cfg (`xlsr2_300m.pt`), though the encoder shapes coincide.
- s3prl wav2vec2 expert semantics reproduced: `F.layer_norm(wav, wav.shape)` per clip (numpy_wav_normalize
  False), `extract_features(mask=False)`, padding mask all-False (equal-length windows), `all_hs[-1]`
  = hook on `model.encoder` output = last block after the final encoder LayerNorm = fairseq
  `features_only=True` `'x'` (dim 1024, T = 201 frames for 64600 samples).

## Back end + score (kept as the authors')
- Linear 1024β†’128 (`m_feat_compress`), 3 Γ— [Linear 128β†’128, LeakyReLU, DropoutForMC(p=0.5)] (no residual
  connection despite the class name), mean over frames, Linear 128β†’2. Dropout is per element of the
  [frames Γ— 128] activations.
- In the authors' inference only these three DropoutForMC layers are stochastic (`F.dropout(training=True)`
  regardless of mode); everything else is in eval mode (SSL dropouts are 0 in the cfg anyway).
- Score = `compute_logit_from_prob(mean over 144 samples of softmax[:, 1])` (source
  `compute_llr_eps_ale`, verbatim incl. the 1e-7 floors β†’ finite, |score| ≀ ~16.1). Bona fide = class 1.
- Reproducibility: the source draws masks from the global RNG (depends on order/batch). We draw explicit
  keep-masks (Γ—2) per clip from a CUDA `torch.Generator` seeded with `blake2b(key="1", data=clip window
  float32 bytes)` (seed 1 = source default `--seed`). So masks depend only on the fixed seed and the clip β†’
  deterministic and batched == single (tested). Two identical clips get identical masks (harmless).
- The SSL runs once per clip; the 144 samples are expanded only in the 128-d back end (cheap).
- `< 3000 samples β†’ score 0` (source `__inference`): **not applied**. It only guards the full-utterance
  path against too-short inputs; with the fixed 64600-sample repeat-tiled window it can never trigger
  (the debug full-utterance script `parity_fullutt.py` does apply it). Empty clips β†’ zeros window.

## Window (maintainer decision 2026-11-01, Q2)
Fixed first 64600 samples, shorter clips repeat-tiled; the authors evaluate the entire utterance at batch 1.
FP32 with cuDNN/matmul TF32 off.

## Reproduction check vs the authors' released score logs (In-the-Wild)
`parity_fullutt.py <ID> <full|win> 400`: 400 random ITW clips (50 per parquet shard, rng seed 0); uid
mapping `Wild_%05d` ↔ `ITW_N` (verified: the released logs give EER P1 6.14 / P2 4.22 / P3 5.86 % with our
labels, = the paper/plan targets). Released-log EERs with our labels (19LA / 21LA / 21DF / ITW):
P1 2.76 / 17.10 / 6.14 / 6.14; P2 1.43 / 15.03 / 7.37 / 4.22; P3 1.73 / 14.88 / 6.02 / 5.86.

| model | mode | Pearson | Spearman | mean |Ξ”| | max |Ξ”| | EER ours / theirs (same 400) |
|---|---|---|---|---|---|---|
| P3 | full utterance (authors' protocol) | 0.99989 | 0.99945 | 0.048 | 0.42 | 6.35 / 5.20 |
| P3 | Arena window (4 s) | 0.9719 | 0.9698 | 0.64 | 5.99 | 4.16 / 5.20 |
| P2 | full utterance | 0.99993 | 0.99964 | 0.036 | 0.25 | 3.44 / 3.44 |
| P2 | Arena window | 0.9805 | 0.9715 | 0.59 | 5.33 | 2.50 / 3.44 |
| P1 | full utterance | 0.99992 | 0.99856 | 0.039 | 0.28 | 6.45 / 5.93 |
| P1 | Arena window | 0.9742 | 0.9678 | 0.51 | 9.05 | 8.01 / 5.93 |

Full-utterance agreement is at the MC-dropout noise level (different random masks), i.e. the port is
exact; the 400-clip EER difference is 1–2 near-threshold clips. The 4-s window changes scores materially
(expected). Raw: `parity_itw_{full,win}*.npz` / `.log` in each dir.

## Batch sweep (sweep.py, 64600-sample random input, GPUs shared with other agents, 2026-11-02)
| model | RTX 4070 Ti SUPER (idx 2) | RTX 5060 Ti (idx 3) |
|---|---|---|
| P1/P3 | 61.5 utt/s @24 (plateau 4–32) | 20.0 @4 (~14.5 at β‰₯8) |
| P2 | 32.7 @24 | (not swept; β†’ 4) |

## TensorRT (`trt_vocoded_voxceleb2_ssl_<id>.py`, sidecar `trt_vocoded-voxceleb2-ssl-<id>.json`)
Only the SSL(s) run on TensorRT (one engine per SSL: normalised wav [B, 64600] β†’ [B, 201, 1024]); the
waveform layer_norm, P2's difference, the MC back end with the identical per-clip masks, and the score stay
in PyTorch. FP16 kept iff 64-clip ITW Spearman β‰₯ 0.9999, else FP32.
- FP16 failed that bar everywhere (Spearman 0.9966–0.9996); a first FP32 build with TRT's default TF32 was
  also off (max |Ξ”| 0.2–0.75, Spearman 0.9993–0.9997): the logit(mean prob) score amplifies small feature
  errors. Final engines: **strict FP32, TF32 cleared** (`TRT_NO_TF32=1 TRT_FORCE_FP32=1 ... export`) β€”
  64-clip build parity max |Ξ”| ≀ 4.3e-4, Spearman 1.0 on both cards.
- P2: two engines; MAX_BATCH_CAP 16 (32 β†’ context-creation OOM with other agents on the card). The export
  first created a second copy of the engines for the sweep (OOM) β€” fixed (reuses the parity engines).
- J-SPAW_LA (2397 trials) TRT vs PyTorch (`results_pt` = PyTorch on RTX 4070 Ti SUPER; `cmp_scores.py`):

| model | PT EER | 4070 TRT EER / Spearman / max Ξ” | 5060 TRT EER / Spearman / max Ξ” | TRT utt/s 4070 / 5060 (best batch) |
|---|---|---|---|---|
| P1 | 19.125 | 19.125 / 1.000000 / 8e-4 βœ… | 19.286 / **0.998762** / 0.063 ❌ β†’ PyTorch on RTX_5060_Ti | 63.4 (16) / β€” |
| P2 | 7.451 | 7.451 / 1.000000 / 4e-4 βœ… | 7.625 (Ξ”0.17) / 0.999640 / 0.075 βœ… | 33.3 (16) / 20.5 (16) |
| P3 | 9.625 | 9.625 / 1.000000 / 2e-4 βœ… | 9.625 / 0.999948 / 0.051 βœ… | 64.1 (32) / 38.0 (32) |

  P1's RTX_5060_Ti key and engine were deleted (results_trt3 kept as evidence). The 5060 deviations are
  cross-card (5060 kernels vs the 4070 PyTorch reference); the 64-clip same-card build parity was exact.
  PyTorch utt/s: P1/P3 ~58–61 (4070) / 20 (5060, bs 4); P2 32.7 (4070).