Codecfake-W2V2-AASIST-CoTrain / implementation-notes.md

Wrapper code (speech-spoof-bench)

26da0bd verified 12 days ago

5.05 kB

# Codecfake-W2V2-AASIST-CoTrain — implementation notes

Arena wrapper for the co-trained **W2V2-AASIST** countermeasure of *The Codecfake Dataset and Countermeasures
for the Universally Detection of Deepfake Audio* (Xie et al., arXiv 2405.04880), built 2026-11-01 under
`speech-spoof-bench/docs/plans/2026-11-01-nine-model-batch-arena-submission.md` (model #2). Sibling model #8
(codec-trained) lives in `../Codecfake-W2V2-AASIST-Codec/` with an identical copy of `_net.py`.

## Source
- Code: https://github.com/xieyuankun/Codecfake @ `72c32e2840eef95fbeade349425fa9c5391fa11e` — **no LICENSE**
  (so no LICENSE.source; weights not re-hosted).
- Checkpoint: `pretrained_model/cotrain_w2v2aasist_CSAM/anti-spoofing_feat_model.pt` (1.8 MB, sha256
  `86876e2c…c644a`), trained on ASVspoof2019 LA train + Codecfake train with the CSAM sampler.
  It is `torch.save(model)` of the whole `W2VAASIST` module. Pickle globals inspected with pickletools
  first: only `model.{W2VAASIST,GraphAttentionLayer,HtrgGraphAttentionLayer,GraphPool,Residual_block}`,
  `torch.nn.modules.*`, torch storage rebuilders, OrderedDict, set.
- Frozen front-end: HF `facebook/wav2vec2-xls-r-300m` @ `1a640f32ac3e39899438a2931f9924c02f080a54` (main).

## Weight acquisition (`_net.head_state`)
First `load()`: download from the pinned raw URL
`https://raw.githubusercontent.com/xieyuankun/Codecfake/72c32e2…/pretrained_model/<sub>/anti-spoofing_feat_model.pt`
into `../_ckpt_dl/xieyuankun__Codecfake/<rev>/…`, check sha256, unpickle with an **allow-list unpickler** that maps
the source's `model.*` classes to a bare `nn.Module` stub (no upstream code imported or run), take its
`state_dict()`, save `head_<sub>.safetensors` in the model dir (gitignored). Later loads read the safetensors.
Pickled hyper-params checked against the port: GAT temperatures 2/2/100×4, pool ratios 0.5×6; module was
saved in eval mode. Strict load: 245 keys, 0 missing / 0 unexpected.

## Port (`_net.py`)
- `W2VAASIST` + layers ported verbatim (prints/unused imports removed). Residual_block: the source computes
  bn1→selu for non-first blocks then applies conv1 to `x` (result discarded) — dead code dropped, bn1 kept.
- Pipeline as `generate_score.py`: `pad_dataset` (first 64 600 samples, `int(cut/len)+1` tiles) →
  `Wav2Vec2FeatureExtractor` normalisation (do_normalize; reproduced per row in torch, max |Δ| 5e-7) →
  XLS-R `hidden_states[5]` → `unsqueeze(1).transpose(2,3)` → (B,1,1024,201) → head.
- XLS-R truncated to 5 transformer layers (`XLSRLayer5`: feature encoder → projection → pos-conv → layers 0–4,
  no final LayerNorm). `hidden_states[5]` of the stable-layer-norm encoder is the input of layer 5 = output of
  layer 4 before `encoder.layer_norm`; verified **bit-identical** (`torch.equal`, max |Δ| 0.0) against the full
  24-layer HF model (eager attention) on 3 random clips. Load asserts 0 missing keys and only layers 5–23 +
  pre-training heads unexpected.
- Attention: eager (as transformers 4.36 used by the authors).
- TF32 disabled in `load()` (cuDNN TF32 convs otherwise moved scores by ~2e-3 between batch compositions).

## Score
Class 0 = bona fide (source writes `softmax(out)[:, 0]`). Arena score = `logits[:,0] - logits[:,1]` = logit of
the authors' probability (monotone, not saturated). Higher = bona fide.

## Params
76,554,634 = XLS-R used part 76,107,392 + head 447,242 (`meta.yaml` 76.554634).

## Reproduction vs the authors' score files (`parity.py`; authors' `result/*.txt` = softmax[:,0])
| set (full) | authors EER | ours EER | Pearson (logit) | Spearman | median \|Δ logit\| |
|---|---|---|---|---|---|
| ASVspoof2019_LA eval (71 237) | 0.312 | 0.312 | 0.99994 | 0.999995 | 0.002 |
| InTheWild (31 779) | 4.681 | 4.562 | 0.9986 | 0.9995 | 0.046 |
Paper: ITW 4.68, 19LA 0.31 — matched. ITW residual explained (`parity_float_vs_flac.py`, 512 clips): scoring the
original float wavs (`release_in_the_wild/`) gives median |Δ| 0.0035 vs the authors (0.043 from the arena's
16-bit FLAC copy); i.e. the port is exact, the gap is the dataset packaging.
## Throughput / TRT (2026-11-01, GPUs shared with other agents' jobs)
PyTorch sweep: RTX 4070 Ti SUPER 288 utt/s @32, RTX 5060 Ti 104 utt/s @16 (< 300 on the 5060 → TRT built).
TRT (`trt_codecfake_w2v2_aasist_cotrain.py`, DPDA-GA-TCM pattern, sidecar `trt_codecfake-w2v2-aasist-cotrain.json`):
| card | precision | best batch | utt/s | build Spearman (64 ITW) | J-SPAW_LA EER PT → TRT | Spearman vs PT |
|---|---|---|---|---|---|---|
| RTX 4070 Ti SUPER | FP32 (FP16 0.9992 < 0.9999) | 32 | 396 | 0.99991 | 0.0626 → 0.0626 | 0.99999 |
| RTX 5060 Ti | FP32 (FP16 0.9991) | 16 | 306 | 0.99977 | 0.0626 → 0.0626 | 0.99999 |
Both accepted. Note J-SPAW_LA EER is ~0 for this model, so ΔEER is a weak check; Spearman carries it.
## Files
`run_sets.sh` (PyTorch full-set runs, both dirs), `trt_export_both.sh`, `trt_jspaw.sh` (both dirs), `parity.py`,
`parity_float_vs_flac.py`. results_pt/{J-SPAW_LA,InTheWild,ASVspoof2019_LA}, results_trt2/3/J-SPAW_LA.

	# Codecfake-W2V2-AASIST-CoTrain — implementation notes

	Arena wrapper for the co-trained W2V2-AASIST countermeasure of *The Codecfake Dataset and Countermeasures
	for the Universally Detection of Deepfake Audio* (Xie et al., arXiv 2405.04880), built 2026-11-01 under
	`speech-spoof-bench/docs/plans/2026-11-01-nine-model-batch-arena-submission.md` (model #2). Sibling model #8
	(codec-trained) lives in `../Codecfake-W2V2-AASIST-Codec/` with an identical copy of `_net.py`.

	## Source
	- Code: https://github.com/xieyuankun/Codecfake @ `72c32e2840eef95fbeade349425fa9c5391fa11e` — no LICENSE
	(so no LICENSE.source; weights not re-hosted).
	- Checkpoint: `pretrained_model/cotrain_w2v2aasist_CSAM/anti-spoofing_feat_model.pt` (1.8 MB, sha256
	`86876e2c…c644a`), trained on ASVspoof2019 LA train + Codecfake train with the CSAM sampler.
	It is `torch.save(model)` of the whole `W2VAASIST` module. Pickle globals inspected with pickletools
	first: only `model.{W2VAASIST,GraphAttentionLayer,HtrgGraphAttentionLayer,GraphPool,Residual_block}`,
	`torch.nn.modules.*`, torch storage rebuilders, OrderedDict, set.
	- Frozen front-end: HF `facebook/wav2vec2-xls-r-300m` @ `1a640f32ac3e39899438a2931f9924c02f080a54` (main).

	## Weight acquisition (`_net.head_state`)
	First `load()`: download from the pinned raw URL
	`https://raw.githubusercontent.com/xieyuankun/Codecfake/72c32e2…/pretrained_model/<sub>/anti-spoofing_feat_model.pt`
	into `../_ckpt_dl/xieyuankun__Codecfake/<rev>/…`, check sha256, unpickle with an allow-list unpickler that maps
	the source's `model.*` classes to a bare `nn.Module` stub (no upstream code imported or run), take its
	`state_dict()`, save `head_<sub>.safetensors` in the model dir (gitignored). Later loads read the safetensors.
	Pickled hyper-params checked against the port: GAT temperatures 2/2/100×4, pool ratios 0.5×6; module was
	saved in eval mode. Strict load: 245 keys, 0 missing / 0 unexpected.

	## Port (`_net.py`)
	- `W2VAASIST` + layers ported verbatim (prints/unused imports removed). Residual_block: the source computes
	bn1→selu for non-first blocks then applies conv1 to `x` (result discarded) — dead code dropped, bn1 kept.
	- Pipeline as `generate_score.py`: `pad_dataset` (first 64 600 samples, `int(cut/len)+1` tiles) →
	`Wav2Vec2FeatureExtractor` normalisation (do_normalize; reproduced per row in torch, max \|Δ\| 5e-7) →
	XLS-R `hidden_states[5]` → `unsqueeze(1).transpose(2,3)` → (B,1,1024,201) → head.
	- XLS-R truncated to 5 transformer layers (`XLSRLayer5`: feature encoder → projection → pos-conv → layers 0–4,
	no final LayerNorm). `hidden_states[5]` of the stable-layer-norm encoder is the input of layer 5 = output of
	layer 4 before `encoder.layer_norm`; verified bit-identical (`torch.equal`, max \|Δ\| 0.0) against the full
	24-layer HF model (eager attention) on 3 random clips. Load asserts 0 missing keys and only layers 5–23 +
	pre-training heads unexpected.
	- Attention: eager (as transformers 4.36 used by the authors).
	- TF32 disabled in `load()` (cuDNN TF32 convs otherwise moved scores by ~2e-3 between batch compositions).

	## Score
	Class 0 = bona fide (source writes `softmax(out)[:, 0]`). Arena score = `logits[:,0] - logits[:,1]` = logit of
	the authors' probability (monotone, not saturated). Higher = bona fide.

	## Params
	76,554,634 = XLS-R used part 76,107,392 + head 447,242 (`meta.yaml` 76.554634).

	## Reproduction vs the authors' score files (`parity.py`; authors' `result/*.txt` = softmax[:,0])
	\| set (full) \| authors EER \| ours EER \| Pearson (logit) \| Spearman \| median \\|Δ logit\\| \|
	\|---\|---\|---\|---\|---\|---\|
	\| ASVspoof2019_LA eval (71 237) \| 0.312 \| 0.312 \| 0.99994 \| 0.999995 \| 0.002 \|
	\| InTheWild (31 779) \| 4.681 \| 4.562 \| 0.9986 \| 0.9995 \| 0.046 \|

	Paper: ITW 4.68, 19LA 0.31 — matched. ITW residual explained (`parity_float_vs_flac.py`, 512 clips): scoring the
	original float wavs (`release_in_the_wild/`) gives median \|Δ\| 0.0035 vs the authors (0.043 from the arena's
	16-bit FLAC copy); i.e. the port is exact, the gap is the dataset packaging.

	## Throughput / TRT (2026-11-01, GPUs shared with other agents' jobs)
	PyTorch sweep: RTX 4070 Ti SUPER 288 utt/s @32, RTX 5060 Ti 104 utt/s @16 (< 300 on the 5060 → TRT built).
	TRT (`trt_codecfake_w2v2_aasist_cotrain.py`, DPDA-GA-TCM pattern, sidecar `trt_codecfake-w2v2-aasist-cotrain.json`):
	\| card \| precision \| best batch \| utt/s \| build Spearman (64 ITW) \| J-SPAW_LA EER PT → TRT \| Spearman vs PT \|
	\|---\|---\|---\|---\|---\|---\|---\|
	\| RTX 4070 Ti SUPER \| FP32 (FP16 0.9992 < 0.9999) \| 32 \| 396 \| 0.99991 \| 0.0626 → 0.0626 \| 0.99999 \|
	\| RTX 5060 Ti \| FP32 (FP16 0.9991) \| 16 \| 306 \| 0.99977 \| 0.0626 → 0.0626 \| 0.99999 \|
	Both accepted. Note J-SPAW_LA EER is ~0 for this model, so ΔEER is a weak check; Spearman carries it.

	## Files
	`run_sets.sh` (PyTorch full-set runs, both dirs), `trt_export_both.sh`, `trt_jspaw.sh` (both dirs), `parity.py`,
	`parity_float_vs_flac.py`. results_pt/{J-SPAW_LA,InTheWild,ASVspoof2019_LA}, results_trt2/3/J-SPAW_LA.