SAID: Semantic Acoustic Imaging Detector for Sound Event Localization and Detection

SAID predicts a separate labeled acoustic map for every active sound source from four-channel spatial audio. Each 180 × 360 map jointly represents the source region, its acoustic energy, and its sound-event class over time.

SAID ranked first on the official DCASE2026 Task 3 Track A evaluation set, with a Macro mAP of 0.1080 and a Macro Pearson r of 0.3962.

Runbang Wang, Zining Liang, Yin Cao, and Qiuqiang Kong

Pretrained models and Model Card

Overview

Demos

The following results use the SAID (PaSST) paper checkpoint. Ground Truth is shown on the left and Prediction on the right with a shared timeline, class legend, colors, and display scale.

Demo 1 — TAU scene

Open Demo 1 as MP4

Demo 3 — Sony scene

Open Demo 3 as MP4

Class-agnostic Audio2Sph demo — TAU scene

Open the class-agnostic TAU demo as MP4

Class-agnostic Audio2Sph demo — Sony scene

Open the class-agnostic Sony demo as MP4

Results

DCASE2026 Task 3 Track A evaluation set

System	Rank	Macro mAP	Macro Pearson r
CUHK (SAID)	1	0.1080	0.3962

These are the official challenge results for the submitted SAID system. Both metrics use the official macro-averaged protocol.

Full-recording development test set

Paper checkpoint	Macro mAP	Macro Pearson r	Mask AP	Macro Class-F1
SAID (PaSST)	0.120150	0.426790	0.237972	0.388488
SAID (AudioMAE)	0.113441	0.430736	0.228684	0.395124

The development-test results cover all 78 full recordings. Macro mAP and Macro Pearson r use the official DCASE evaluator. Mask AP applies the same official AP calculation after merging the classes, measuring source-region detection. Macro Class-F1 measures classification for spatial matches within 20°.

Quick start

SAID supports Python 3.10 through 3.12. Install the project from the repository root:

pip install -e .

The release contains two complete SAID checkpoints and the paper's class-agnostic Audio2Sph checkpoint:

• said_passt.ckpt for SAID (PaSST), the default model;
• said_audiomae.ckpt for SAID (AudioMAE);
• audio2sph.ckpt for Audio2Sph + Panoramic Decoder.

Run the four demos

said demo --model said_passt
said demo --model said_audiomae
said demo --model audio2sph

All three commands write to said_demo_output/. Their files are named said_passt_demo_1.mp4 through said_passt_demo_4.mp4, said_audiomae_demo_1.mp4 through said_audiomae_demo_4.mp4, and audio2sph_demo_1.mp4 through audio2sph_demo_4.mp4. Each 20-second video presents Ground Truth and Prediction side by side over a synchronized 360-degree scene. The Audio2Sph videos show the merged acoustic field without class labels. The four licensed demo inputs are included in the release.

Run inference

said infer recording.wav --model said_passt

recording.wav can contain the complete 32-channel Eigenmike recording or the four Eigenmike capsules [6, 10, 26, 22] in that order. The input pipeline selects these capsules and resamples them to 48 kHz. SAID writes a lossless, chunked prediction archive containing the labeled acoustic maps, class IDs, confidence scores, timing information, and checkpoint identity.

Select SAID (AudioMAE) with:

said infer recording.wav --model said_audiomae

Run the published Audio2Sph and Panoramic Decoder without Sph2Imaging or a Class Feature Encoder with:

said infer recording.wav --model audio2sph

This route writes dense class-agnostic 180×360 maps at the native 100 Hz time base.

See Inference for recording-domain selection, visualization, output fields, and the Python API.

Reproduce the evaluation

said evaluate /path/to/dcase2026 --model said_passt

The command runs full-recording inference on the 78-file DCASE development-test set, writes standard DCASE predictions, and reports the four metrics shown above. On its first metric run, SAID obtains the official DCASE2026 Task 3 evaluator directly from the official baseline repository and records its source identity with the results.

See Evaluation for the dataset layout, frozen paper protocol, reusable predictions, and metric definitions.

Paper checkpoints

All three published checkpoints are fingerprinted and loaded with strict schema, shape, dtype, size, and SHA256 verification. Their identities and provenance are documented in the SAID (PaSST) checkpoint card and SAID (AudioMAE) checkpoint card, and the Audio2Sph checkpoint card. The CLI downloads a selected checkpoint from the official Hugging Face repository on first use and stores it in the repository-root checkpoints/ directory.

The checkpoints are released for non-commercial research use only under the SAID Model Weights Non-Commercial Research License 1.0, together with all applicable upstream terms. The SAID source code remains available under the MIT License for independently trained models. Commercial use of these checkpoints is not permitted, including modified or fine-tuned derivatives.

Training

The paper training workflow is:

Audio2Sph pretraining
-> complete SAID training with SourceBank and Online Scene Generation
-> DCASE fine-tuning

Install the rendering dependencies and set the local dataset paths in configs/data.yaml. Each public recipe is directly runnable from a published checkpoint in the repository-root checkpoints/ directory. The PaSST route is launched with:

pip install -e '.[render]'
said prepare --config configs/training/audio2sph.yaml
said train --config configs/training/audio2sph.yaml
said train --config configs/training/sourcebank_passt.yaml
said train --config configs/training/dcase_passt.yaml

The corresponding AudioMAE route uses configs/training/sourcebank_audiomae.yaml and configs/training/dcase_audiomae.yaml. All recipes use the same configuration schema and said train interface. Interrupted runs resume with said train --config CONFIG --resume CHECKPOINT.

See Training, Configuration, and Data preparation for the complete workflow and interfaces.

Data

Readers obtain the training and evaluation datasets from their rights holders and connect them through the included adapters:

• DCASE2026 Task 3 recordings and labels for fine-tuning and evaluation;
• VCTK v0.80 for Audio2Sph pretraining;
• a local SourceBank manifest for class-labeled Online Scene Generation.

SAID includes the renderer, Eigenmike geometry, DCASE adapters, VCTK preparation, SourceBank validation, and target construction. Dataset paths and permissions remain explicit in configs/data.yaml and the SourceBank manifest. See Data preparation for layouts, sources, and license-aware interfaces.

Prediction compression

Prediction compression is an optional post-processing utility for producing DCASE JSON files under the per-recording size limit:

said compress said_evaluation/inference_outputs --output said_compressed

See Prediction compression for the paper parameters, fallback policy, and evaluation procedure.

Documentation

• Inference and Python API
• DCASE evaluation
• Training
• Configuration reference
• Data preparation
• Model architecture
• Prediction compression

Command-specific options are also available through said COMMAND --help.

License

Original SAID software and documentation are released under the MIT License. The three published checkpoints are distributed for non-commercial research. Third-party implementations, pretrained components, datasets, demo media, and the official evaluator retain their corresponding terms. The complete attribution and license boundaries are recorded in Third-party notices and the model and asset license summary.

Commercial applications can train new models with the MIT-licensed SAID code and independently obtained components and data whose licenses permit the intended use.

Citation

If you use SAID, please cite:

@inproceedings{wang2026said,
  title     = {{SAID}: Semantic Acoustic Imaging Detector for Sound Event
               Localization and Detection},
  author    = {Wang, Runbang and Liang, Zining and Cao, Yin and Kong, Qiuqiang},
  booktitle = {Proceedings of the Detection and Classification of Acoustic
               Scenes and Events 2026 Workshop},
  year      = {2026}
}

Machine-readable citation metadata is available in CITATION.cff.

Acknowledgements

The Online Scene Generation components build on the NESD release_v1.0 renderer. The rigid-sphere implementation credits Yin Cao and Qiuqiang Kong; the renderer and shoebox image-source implementation credit Qiuqiang Kong and CUHK. Detailed source revisions and notices are provided in THIRD_PARTY_NOTICES.md.

This work was supported by the Innovation and Technology Fund (ITF), Hong Kong, under Project ITS/301/24.