README.md

---
license: apache-2.0
library_name: pytorch
tags:
  - audio
  - speech-separation
  - target-speech-extraction
  - speaker-diarization
  - tf-gridnet
---

# dagger — Phase 2 DoD extractor (scratch-trained 3/4/5-speaker curriculum, LibriMix)

The checkpoint `dagger`'s **Phase 2 final numbers are reported from**: a TF-GridNet +
cross-attention extractor `G(x_O, ē_i)` conditioned on a TitaNet speaker embedding, which
extracts every speaker from the **untouched original mixture** rather than a running
residual ([`CLAUDE.md` §1](https://github.com/RohanBanerjee88/dagger/blob/main/CLAUDE.md#1-the-one-rule-you-must-never-break)).

Unlike the earlier Phase 1 / Phase 2 checkpoints, this one is trained **from random init by a
single command** — no warm-start chain — so every reported number is regenerable from one
config.

- **Repo / code:** https://github.com/RohanBanerjee88/dagger
- **Training config:** [`configs/phase2/dod/phase2_librimix_curriculum_3_4_5_train_scratch.yaml`](https://github.com/RohanBanerjee88/dagger/blob/main/configs/phase2/dod/phase2_librimix_curriculum_3_4_5_train_scratch.yaml)
  (`--system proposed`). Multi-depth curriculum: batches interleaved from **three loaders**
  (`n_src` 3, 4, 5; 800 scheduled-placement Libri3Mix train-360 scenes each, 2400 total),
  10 epochs, batch 4, lr 3e-4, `grad_clip 50`, one T4, ~6.5 h. "Curriculum" here means
  *several overlap depths in one run*, not easy-to-hard ordering.
- **Eval configs:** [`configs/phase2/dod/phase2_librimix_{3,4,5}spk_eval_scratch.yaml`](https://github.com/RohanBanerjee88/dagger/tree/main/configs/phase2/dod)
  — 150 test scenes each, oracle diarization, tag `scratch345clip50`.
- **Sample rate:** 8 kHz.
- **Speaker counts seen in training:** 3, 4, 5 (recorded in the checkpoint as `trained_n_src`).

## Results

Phase 2's claims are **relative**: every comparison holds scenes, mixture and checkpoint
fixed and varies only the reconstruction strategy.

- **Ordering, 9 of 9 slices:** `coarse_to_fine > gated_deflation > ungated_deflation` at every
  overlap depth 2–5 across all three eval sets. Holds on both scratch checkpoints (18/18) and
  on every warm-started checkpoint before them.
- **Accumulation** (`ungated_deflation`, SI-SDR vs. number of prior estimates subtracted into
  the residual; m=5 at depth 5, n=150 per level): `-4.97 → -5.89 → -6.31 → -7.00 → -6.78`,
  **-1.81 dB** total, monotone through the body of the chain. The terminal step is the
  one-and-rest endpoint and flattens — a benign case of Theorem 2's `‖E_m‖ ≤ m·ε` upper bound.
- **Absolute SI-SDR** (`no_recursion`, deepest depth per set): 3spk **-1.29**, 4spk **-3.47**,
  5spk **-4.87** dB. Negative at depths 4–5 for *every* system, including the one with no
  deflation logic — this is the extractor's operating point under a small training budget
  (~2,000 steps at `n_src=3` vs. Phase 1's 15,000), not the reconstruction strategy.

Full tables, spread, `n` and diagnostic counts:
[`results/phase2/dod_final/`](https://github.com/RohanBanerjee88/dagger/tree/main/results/phase2/dod_final).

## Caveats

- **The ordering is checkpoint-robust; the accumulation *magnitude* is not.** It measures
  ~1.8 dB on scratch-trained curriculum checkpoints (replicated twice, agreeing to 0.08 dB per
  level) and 5.30 dB on the earlier warm-started chain trained on the same curriculum recipe.
  Quote the magnitude as a range with the checkpoint named.
- **Oracle diarization only.** Real-diarization robustness is Phase 3 and not yet measured.
- **Synthetic 8 kHz LibriMix** (English read speech, hard boundaries, no reverb). Real-corpora
  numbers are Phase 4.
- **Set `refine.rounds: 0` unless you have contaminated enrollment.** Embedding refinement was
  measured net-harmful under clean enrollment (-0.07 to -0.41 dB across 9 slices, gate healthy).
- Research artifact for reproducing the Phase 2 experiment — not a production separation model.

## Files

- `proposed_librimix_curriculum_3_4_5_scratch_clip50.pt` — a `torch.save` dict with
  `state_dict`, `model_config`, `system`, `trained_n_src`. The architecture must match exactly
  or `load_state_dict` raises: `hidden_channels 64`, `n_blocks 6`, `n_fft 256`, `hop_length 64`,
  `n_heads 4`, `n_tokens 8`, `cross_attn_blocks 6`.

## License

Apache-2.0, matching the `dagger` repository.

**Dependency note (same as the Phase 1/2 checkpoints):** at inference `dagger` loads NVIDIA's
**TitaNet-Large** (`nvidia/speakerverification_en_titanet_large`) as the frozen speaker encoder
`φ`, via NeMo (Apache-2.0). TitaNet-Large is **CC-BY-4.0 by NVIDIA, a separate model, and is
NOT bundled here** — this repo's Apache-2.0 covers only the extractor weights uploaded here.
Canonical text: [`NOTICE`](https://github.com/RohanBanerjee88/dagger/blob/main/NOTICE).

## Usage

```python
from huggingface_hub import hf_hub_download
import torch

ckpt_path = hf_hub_download(
    repo_id="AdityaAA2004/dagger-phase2-curriculum-3-4-5-scratch",
    filename="proposed_librimix_curriculum_3_4_5_scratch_clip50.pt",
)
state = torch.load(ckpt_path, map_location="cpu")
state["state_dict"], state["model_config"], state["trained_n_src"]
# load into dagger.extract.tfgridnet_crossattn — see the repo for the full
# pipeline (diarization -> enrollment -> extraction -> reconstruction).