docs/training.md

Training

All released training routes use one command:

said train --config configs/training/<recipe>.yaml

Before running a recipe, set the paths it needs in configs/data.yaml. Each recipe is an independent entry point: it loads the published initialization listed in its YAML unless that path is changed. Running one recipe does not automatically change the initialization of another recipe.

Choose a route

Goal	Recipe	Local data required	Published initialization
Fine-tune SAID (PaSST) on DCASE	dcase_passt.yaml	DCASE development audio and labels	said_passt.ckpt
Fine-tune SAID (AudioMAE) on DCASE	dcase_audiomae.yaml	DCASE development audio and labels	said_audiomae.ckpt
Train SAID (PaSST) with SourceBank	sourcebank_passt.yaml	SourceBank manifest and local audio	audio2sph.ckpt and the PaSST Class Feature Encoder from said_passt.ckpt
Train SAID (AudioMAE) with SourceBank	sourcebank_audiomae.yaml	SourceBank manifest and local audio	audio2sph.ckpt and the AudioMAE Class Feature Encoder from said_audiomae.ckpt
Train Audio2Sph + Panoramic Decoder	audio2sph.yaml	Prepared VCTK v0.80	none

The quickest training check is DCASE fine-tuning. After setting dcase_recordings.root in configs/data.yaml, run:

said train --config configs/training/dcase_passt.yaml

Replace passt with audiomae to use SAID (AudioMAE).

Complete paper workflow

The paper training workflow is:

Audio2Sph pretraining
→ complete SAID training with SourceBank and Online Scene Generation
→ DCASE fine-tuning

1. Train Audio2Sph

Install the Online Scene Generation dependencies, set the VCTK input and prepared-output paths, prepare VCTK once, and then start training:

pip install -e '.[render]'

# Set simulated_scenes.vctk.source_root and prepared_root in configs/data.yaml.
said prepare --config configs/training/audio2sph.yaml
said train --config configs/training/audio2sph.yaml

The preparation step converts VCTK v0.80 into the validated two-second input used by the recipe. The training recipe runs 2.5 million optimizer steps with batch size one and writes Audio2Sph together with its training-time Panoramic Decoder.

2. Train complete SAID with SourceBank

Set simulated_scenes.sourcebank.manifest and simulated_scenes.sourcebank.source_audio_root in configs/data.yaml, then run one Class Feature Encoder route:

said train --config configs/training/sourcebank_passt.yaml
# or
said train --config configs/training/sourcebank_audiomae.yaml

This route keeps the Audio2Sph encoder, removes the Panoramic Decoder, attaches Sph2Imaging, and trains complete SAID on generated two-second scenes. It uses exact 21-frame training targets. Data defines the SourceBank manifest and permission contract.

3. Fine-tune complete SAID on DCASE

Set dcase_recordings.root in configs/data.yaml, then run the matching recipe:

said train --config configs/training/dcase_passt.yaml
# or
said train --config configs/training/dcase_audiomae.yaml

DCASE fine-tuning consumes 20 official half-open 10 Hz frames per two-second sample and discards only the detector endpoint during loss computation. It does not merge adjacent official label frames.

Pass a newly trained model to the next route

The published recipes start from published checkpoints. To continue a newly trained workflow, edit the next recipe before launching it:

Next recipe	Field to change	Example preceding output
sourcebank_passt.yaml or sourcebank_audiomae.yaml	load_audio2sph_ckpt	../../runs/audio2sph_pretraining/latest.ema.ckpt
dcase_passt.yaml	load_said_ckpt	../../runs/said_sourcebank_passt/latest.ema.ckpt
dcase_audiomae.yaml	load_said_ckpt	../../runs/said_sourcebank_audiomae/latest.ema.ckpt

The SourceBank recipes also use load_class_feature_encoder_ckpt to extract the selected frozen Class Feature Encoder. Its type must match class_feature_encoder: passt or class_feature_encoder: audiomae.

Canonical published paths under checkpoints/ are downloaded and verified on first use. Any other path is treated as an explicit local checkpoint.

Checkpoints and resume

Each run writes into its configured training.output_directory:

File	Purpose
latest.ema.ckpt	Latest model weights; use this or a retained EMA milestone to initialize another recipe
latest.training.pt	Model, optimizer, scheduler, random state, step, and validation state for exact resume
best.ema.ckpt, best.json	Lowest validation loss when validation is enabled
step-*.ema.ckpt, step-*.training.pt	Retained milestones at keep_every_steps
metrics.jsonl	Complete machine-readable training and validation records

Rerunning the same command resumes latest.training.pt automatically. Use --resume PATH only to select another .training.pt from the same training recipe and the same data configuration. The trainer verifies a fingerprint of both YAML files before restoring the optimizer and random state.

Use a flat .ema.ckpt through load_audio2sph_ckpt or load_said_ckpt when moving between routes. Validation-enabled Audio2Sph and SourceBank recipes record best.ema.ckpt, while the handoff examples above deliberately use the latest completed state. DCASE fine-tuning leaves validation disabled so the development-test recordings remain reserved for full-recording evaluation.

The terminal reports step, loss, learning rate, throughput, ETA, validation, and checkpoint events. Configuration defines every recipe field; Data defines the required dataset layouts.