Finalize evaluation versions and six-metric results

README.md

@@ -21,7 +21,10 @@ with a Macro mAP of **0.1080** and a Macro Pearson r of **0.3962**.
 
 Runbang Wang, Zining Liang, Yin Cao, and Qiuqiang Kong
 
-[Pretrained models and Model Card](https://huggingface.co/IN03X/SAID)
+[Official DCASE2026 results](https://dcase.community/challenge2026/task-semantic-acoustic-imaging-for-sound-event-localization-and-detection-from-spatial-audio-and-audiovisual-scenes-results)
+· [Models and checkpoints](https://huggingface.co/IN03X/SAID)
+· [Evaluation and reproduction](docs/evaluation.md)
+· [Training](docs/training.md)
 
 ## Overview
 
@@ -65,20 +68,27 @@ legend, colors, and display scale.
 |---|---:|---:|---:|
 | CUHK (SAID) | **1** | **0.1080** | **0.3962** |
 
-These are the official challenge results for the submitted SAID system. Both
-metrics use the official macro-averaged protocol.
+These are the
+[official challenge results](https://dcase.community/challenge2026/task-semantic-acoustic-imaging-for-sound-event-localization-and-detection-from-spatial-audio-and-audiovisual-scenes-results)
+for the submitted SAID system. Both metrics use the official macro-averaged
+protocol.
 
 ### Full-recording development test set
 
-| Paper checkpoint | Macro mAP | Macro Pearson r | Mask AP | Macro Class-F1 |
-|---|---:|---:|---:|---:|
-| SAID (PaSST) | **0.120150** | 0.426790 | **0.237972** | 0.388488 |
-| SAID (AudioMAE) | 0.113441 | **0.430736** | 0.228684 | **0.395124** |
+Metric labels: **(4.2)** is the official evaluator from **2026-04-02**
+([commit `84b2cd1`](https://github.com/iranroman/DCASE2026_Task3_SAISELD_baseline/commit/84b2cd155c9ee0c2176f386db54541ffd75312b2));
+**(6.30)** is the current official evaluator from **2026-06-30**
+([commit `d4df662`](https://github.com/iranroman/DCASE2026_Task3_SAISELD_baseline/commit/d4df66251f39e34bc0157be93858e5a68ec9d7c4)).
 
-The development-test results cover all 78 full recordings. Macro mAP and Macro
-Pearson r use the official DCASE evaluator. Mask AP applies the same official AP
-calculation after merging the classes, measuring source-region detection.
-Macro Class-F1 measures classification for spatial matches within 20°.
+| Paper checkpoint | Macro mAP (4.2) | Mask AP (4.2) | Macro mAP (6.30) | Class-agnostic AP (6.30) | Macro Pearson r (6.30) | Macro Class-F1 |
+|---|---:|---:|---:|---:|---:|---:|
+| SAID (PaSST) | **0.120150** | **0.237972** | 0.124514 | **0.197483** | 0.426790 | 0.388488 |
+| SAID (AudioMAE) | 0.113441 | 0.228684 | **0.124763** | 0.193225 | **0.430736** | **0.395145** |
+
+The paper reports the (4.2) AP values shown in the table; the (6.30) columns
+apply the current official definitions to the same predictions.
+[Evaluation](docs/evaluation.md#metric-versions) gives the exact definitions,
+upstream commits, and output fields.
 
 ## Quick start
 
@@ -117,46 +127,68 @@ class labels. The four licensed demo inputs are included in the release.
 
 ### Run inference
 
+The shortest command writes one readable JSON prediction:
+
 ```bash
 said infer recording.wav --model said_passt
 ```
 
 `recording.wav` can contain the complete 32-channel Eigenmike recording or the
 four Eigenmike capsules `[6, 10, 26, 22]` in that order. The input pipeline
-selects these capsules and resamples them to 48 kHz. SAID writes a lossless,
-chunked prediction archive containing the labeled acoustic maps, class IDs,
-confidence scores, timing information, and checkpoint identity.
+selects these capsules and resamples them to 48 kHz. Complete SAID models write
+standard DCASE JSON by default to
+`said_output/recording_inference.json`. The file can be read with any JSON
+library and passed directly to the DCASE evaluation and compression tools.
 
-Select SAID (AudioMAE) with:
+A longer form can retain lossless float32 maps and create a synchronized
+prediction video at the same time:
 
 ```bash
-said infer recording.wav --model said_audiomae
+said infer recording.wav \
+  --model said_passt \
+  --format npz \
+  --visual panorama.mp4 \
+  --output said_output
 ```
 
-Run the published Audio2Sph and Panoramic Decoder without Sph2Imaging or a
-Class Feature Encoder with:
+`--format npz` and `--visual panorama.mp4` are independent options: select
+either one or combine them as above. The same interface selects SAID
+(AudioMAE) or the class-agnostic Audio2Sph and Panoramic Decoder:
 
 ```bash
+said infer recording.wav --model said_audiomae
 said infer recording.wav --model audio2sph
 ```
 
-This route writes dense class-agnostic 180×360 maps at the native 100 Hz time
-base.
+Audio2Sph writes native 100 Hz class-agnostic spherical points to one
+`audio2sph-prediction-json-v1` file by default. Its optional NPZ retains the
+dense 180×360 float32 maps.
 
 See [Inference](docs/inference.md) for recording-domain selection,
 visualization, output fields, and the Python API.
 
 ### Reproduce the evaluation
 
+Download the official STAIRS26 development archives
+[`32ch_audio_dev.zip`](https://zenodo.org/api/records/18171005/files/32ch_audio_dev.zip/content)
+and
+[`labels_dev.zip`](https://zenodo.org/api/records/18171005/files/labels_dev.zip/content),
+then extract both into one dataset directory. For example, when the extracted
+directories are
+`$HOME/datasets/dcase2026_task3/eigen_dev/` and
+`$HOME/datasets/dcase2026_task3/labels_dev/`, run:
+
 ```bash
-said evaluate /path/to/dcase2026 --model said_passt
+said evaluate "$HOME/datasets/dcase2026_task3" \
+  --model said_passt \
+  --add-previous-metrics
 ```
 
-The command runs full-recording inference on the 78-file DCASE development-test
-set, writes standard DCASE predictions, and reports the four metrics shown
-above. On its first metric run, SAID obtains the official DCASE2026 Task 3
-evaluator directly from the official baseline repository and records its
-source identity with the results.
+The default output follows the current (6.30) official definition.
+`--add-previous-metrics` adds the two (4.2) COCO AP values used in the paper.
+Pearson r always uses (6.30).
+
+Metric code: [official DCASE2026 Task 3 evaluator](https://github.com/iranroman/DCASE2026_Task3_SAISELD_baseline).
 
 See [Evaluation](docs/evaluation.md) for the dataset layout, frozen paper
 protocol, reusable predictions, and metric definitions.
@@ -201,11 +233,18 @@ said train --config configs/training/sourcebank_passt.yaml
 said train --config configs/training/dcase_passt.yaml
 ```
 
+The supplied recipes use published checkpoint assets by default, so each
+command is directly runnable after its required data is configured. To carry
+newly trained weights from one part of the workflow into the next, update the
+next recipe's `load_audio2sph_ckpt` or `load_said_ckpt` as listed in
+[Training](docs/training.md#reference-recipes).
+
 The corresponding AudioMAE route uses
 `configs/training/sourcebank_audiomae.yaml` and
 `configs/training/dcase_audiomae.yaml`. All recipes use the same configuration
 schema and `said train` interface. Interrupted runs resume with
-`said train --config CONFIG --resume CHECKPOINT`.
+the same `said train --config CONFIG` command. The latest complete training
+state is detected automatically; `--resume CHECKPOINT` selects another state.
 
 See [Training](docs/training.md), [Configuration](docs/configuration.md), and
 [Data preparation](docs/data.md) for the complete workflow and interfaces.
@@ -219,6 +258,11 @@ and connect them through the included adapters:
 - VCTK v0.80 for Audio2Sph pretraining;
 - a local SourceBank manifest for class-labeled Online Scene Generation.
 
+These map to `dcase_recordings.root`, the two
+`simulated_scenes.vctk.*_root` fields, and the two
+`simulated_scenes.sourcebank` path fields in `configs/data.yaml`, respectively.
+Only the files selected by the current recipe are required.
+
 SAID includes the renderer, Eigenmike geometry, DCASE adapters, VCTK
 preparation, SourceBank validation, and target construction. Dataset paths and
 permissions remain explicit in `configs/data.yaml` and the SourceBank manifest.
@@ -234,6 +278,13 @@ DCASE JSON files under the per-recording size limit:
 said compress said_evaluation/inference_outputs --output said_compressed
 ```
 
+For a single recording produced by `said infer`, use its output directory
+directly:
+
+```bash
+said compress said_output --output said_compressed
+```
+
 See [Prediction compression](docs/compression.md) for the paper parameters,
 fallback policy, and evaluation procedure.
 

docs/checkpoints/said_audiomae.md

@@ -77,9 +77,12 @@ API consistency and does not apply a domain embedding in this variant.
 
 ## Reported development-test metrics
 
-| Macro mAP | Macro Pearson r | Mask AP | Macro Class-F1 |
-|---:|---:|---:|---:|
-| 0.113441 | 0.430736 | 0.228684 | 0.395124 |
+Here (4.2) means 2026-04-02, commit `84b2cd1`; (6.30) means
+2026-06-30, commit `d4df662`. See [Evaluation](../evaluation.md#metric-versions).
+
+| Macro mAP (4.2) | Mask AP (4.2) | Macro mAP (6.30) | Class-agnostic AP (6.30) | Macro Pearson r (6.30) | Macro Class-F1 |
+|---:|---:|---:|---:|---:|---:|
+| 0.113441 | 0.228684 | 0.124763 | 0.193225 | 0.430736 | 0.395145 |
 
 The metrics use all 78 full development-test recordings. Pearson correlation
 uses the official macro protocol.

docs/evaluation.md

@@ -1,59 +1,60 @@
 # DCASE development-test evaluation
 
-SAID evaluates the complete DCASE2026 Task 3 development-test split: 30 Sony
-recordings and 48 TAU recordings. The command discovers audio and labels under
-the dataset root, assigns the corresponding recording-domain embedding, runs
-full-recording inference, writes standard DCASE JSON, and runs the official
-DCASE evaluator.
+SAID evaluates all 78 full recordings in the DCASE2026 Task 3 development-test
+split: 30 Sony recordings and 48 TAU recordings.
 
-The complete command is:
+## Metric versions
 
-```bash
-said evaluate /path/to/dcase2026 --model said_passt
-```
+The official evaluator changed during the challenge. The short labels used
+below are defined here before their first use:
 
-On the first metric run, SAID downloads the current official `evaluate.py` from
-the DCASE2026 Task 3 baseline repository and stores it at
-`said/evaluation/metrics/official/evaluate.py`. Later runs reuse the local file.
-The official source is obtained from its original repository; it is not
-committed to SAID or included in the wheel.
+| Label | Official definition |
+|---|---|
+| **(4.2) COCO AP** — **2026-04-02**, [`84b2cd1`](https://github.com/iranroman/DCASE2026_Task3_SAISELD_baseline/commit/84b2cd155c9ee0c2176f386db54541ffd75312b2) | 180×360 spherical Gaussian masks, σ = 6°, 10% peak threshold, and COCO segmentation AP at IoU 0.50:0.95. |
+| **(6.30) Soft-IoU / Macro Pearson** — **2026-06-30**, [`d4df662`](https://github.com/iranroman/DCASE2026_Task3_SAISELD_baseline/commit/d4df66251f39e34bc0157be93858e5a68ec9d7c4) | 100×50 interpolated soft maps, soft-IoU AP at 0.25, 0.50, and 0.75, class-macro soft-map Pearson r, and correct zero AP when a class has ground truth but no detection. |
 
-SAID normalizes prediction filenames, runs the official evaluator for the
-class-aware metrics, and runs it again after temporary class collapsing for
-class-agnostic Mask AP. The independent 20-degree matched Class Macro-F1 is in
-`said/evaluation/metrics/class_f1.py`. Temporary class-collapsed JSON files
-are removed automatically.
+The paper uses the versions below:
 
-For local score reproduction, SAID disables only the official script's 20 MB
-submission-file guard; its metric functions and parameters are unchanged.
-Per-recording submission-size compliance is verified separately by
-`said compress`.
+| Paper column | Version |
+|---|---|
+| Macro mAP | **(4.2)**, averaged over 13 classes |
+| Mask AP | **(4.2)**, after merging all classes into `All classes` |
+| Macro Pearson r | **(6.30)** |
+| Macro Class-F1 | SAID diagnostic: class F1 after 20° spatial matching |
 
-Existing prediction directories, including output from `said compress`, can be
-scored directly:
+The (4.2) evaluator also contains an older Pearson calculation. The paper does
+not use it. Pearson r always comes from (6.30).
 
-```bash
-said evaluate /path/to/dcase2026 \
-  --predictions said_compressed \
-  --output said_compressed_evaluation
-```
+Complete full-recording development-test results:
+
+| Paper checkpoint | Macro mAP (4.2) | Mask AP (4.2) | Macro mAP (6.30) | Class-agnostic AP (6.30) | Macro Pearson r (6.30) | Macro Class-F1 |
+|---|---:|---:|---:|---:|---:|---:|
+| SAID (PaSST) | **0.120150** | **0.237972** | 0.124514 | **0.197483** | 0.426790 | 0.388488 |
+| SAID (AudioMAE) | 0.113441 | 0.228684 | **0.124763** | 0.193225 | **0.430736** | **0.395145** |
+
+The AP values differ because (4.2) and (6.30) are different official metric
+definitions applied to the same predictions.
 
-This path invokes the downloaded official evaluator and does not load a model or
-repeat inference.
+## Download the development data
 
-`--predictions-only` explicitly selects export and validation without metric
-computation:
+Download these two files from the official
+[STAIRS26 record](https://zenodo.org/records/18171005):
+
+- [`32ch_audio_dev.zip`](https://zenodo.org/api/records/18171005/files/32ch_audio_dev.zip/content)
+- [`labels_dev.zip`](https://zenodo.org/api/records/18171005/files/labels_dev.zip/content)
+
+Extract both archives into one directory:
 
 ```bash
-said evaluate /path/to/dcase2026 \
-  --model said_passt \
-  --predictions-only
+mkdir -p "$HOME/datasets/dcase2026_task3"
+unzip 32ch_audio_dev.zip -d "$HOME/datasets/dcase2026_task3"
+unzip labels_dev.zip -d "$HOME/datasets/dcase2026_task3"
 ```
 
-## Dataset layout
+Expected layout:
 
 ```text
-dcase2026/
+$HOME/datasets/dcase2026_task3/
 ├── eigen_dev/
 │   ├── dev-test-sony/*.wav
 │   └── dev-test-tau/*.wav
@@ -62,58 +63,116 @@ dcase2026/
     └── dev-test-tau/*_std.json
 ```
 
-Audio and label stems must match one-to-one. Missing files, duplicate label
-stems, and incomplete paper splits are rejected. The number of evaluated frames
-for each recording is `max(metadata_frame_index) + 1` from its label file; WAV
-duration does not extend the paper evaluation interval. The complete WAV is
-loaded before resampling, and the final two-second model window retains real
-audio beyond the final annotated frame when that context exists. Only emitted
-predictions are truncated to the label-defined frame count.
-
-## Frozen paper protocol
-
-- 48 kHz selected Eigenmike capsules `[6, 10, 26, 22]`;
-- two-second windows with the first 20 of 21 detector positions retained;
-- 10 fps, truncated to each label-defined recording length;
-- slot confidence at least 0.05;
-- at most four slots per frame, sorted by confidence;
-- map points at least 10% of the corresponding map peak;
-- no additional energy top-k limit;
-- score and energy rounded to six decimal places;
-- zero-based 13-class taxonomy;
-- `x_dcase = (179 - x_model) mod 360`.
+## Run the paper evaluation
 
-`evaluation_manifest.json` records the software and protocol revisions,
-checkpoint SHA256, model variant, Eigenmike capsule order, coordinate
-convention, Sony/TAU counts, label-defined frame counts, audio and label
-hashes, a complete dataset fingerprint, and each prediction JSON hash and
-size. Every field is revalidated before an existing inference is reused.
+```bash
+said evaluate "$HOME/datasets/dcase2026_task3" \
+  --model said_passt \
+  --add-previous-metrics
+```
 
-## Official evaluator source
+The example includes `--add-previous-metrics` because it reproduces the paper
+table.
 
-The official evaluator is downloaded from:
+| Command | Metrics produced |
+|---|---|
+| `said evaluate ...` | Current (6.30) Macro mAP, class-agnostic AP, Macro Pearson r, and Class-F1 |
+| `said evaluate ... --add-previous-metrics` | All default metrics, plus (4.2) Paper Macro mAP and Paper Mask AP |
 
-```text
-repository: https://github.com/iranroman/DCASE2026_Task3_SAISELD_baseline
-download:   https://raw.githubusercontent.com/iranroman/DCASE2026_Task3_SAISELD_baseline/main/evaluate.py
-```
+`--add-previous-metrics` adds the two (4.2) AP values. It does not change
+inference or Pearson r.
 
-The repository, source URL, downloaded-file SHA256, and local path are recorded
-in `evaluator.json` for result provenance. The SHA256 is descriptive rather
-than a version gate, so future updates published by the official repository can
-be downloaded without changing SAID. A failed download stops evaluation before
-an incomplete file is published or executed.
+The evaluator comes from the
+[official DCASE2026 Task 3 repository](https://github.com/iranroman/DCASE2026_Task3_SAISELD_baseline).
 
-The output `metrics.json` contains the four paper fields:
+## Output fields
+
+With `--add-previous-metrics`, `metrics.json` contains:
 
 ```json
 {
-  "official_macro_map": 0.120150225347623,
+  "official_macro_map": 0.124514075016962,
+  "official_class_agnostic_ap": 0.197482567736491,
   "official_macro_pearson_r": 0.426790037524333,
-  "class_agnostic_mask_ap": 0.237972219425958,
-  "matched_class_macro_f1": 0.388487929954242
+  "matched_class_macro_f1": 0.388487929954242,
+  "paper_coco_macro_map": 0.120150225347623,
+  "paper_coco_mask_ap": 0.237972219425958
 }
 ```
 
-The first two are official DCASE metrics. Class-agnostic Mask AP and 20-degree
-matched Class Macro-F1 are the paper's class and localization diagnostics.
+| JSON field | Meaning |
+|---|---|
+| `official_macro_map` | (6.30) current official Macro mAP |
+| `official_class_agnostic_ap` | (6.30) AP after merging all classes |
+| `official_macro_pearson_r` | (6.30) current official Macro Pearson r |
+| `matched_class_macro_f1` | SAID 20° matched Macro Class-F1 |
+| `paper_coco_macro_map` | (4.2) paper Macro mAP |
+| `paper_coco_mask_ap` | (4.2) paper Mask AP with one `All classes` category |
+
+Without `--add-previous-metrics`, the final two fields are absent.
+
+## Score existing predictions
+
+Inference does not need to be repeated:
+
+```bash
+said evaluate "$HOME/datasets/dcase2026_task3" \
+  --predictions said_evaluation/inference_outputs \
+  --output said_evaluation_rescored \
+  --add-previous-metrics
+```
+
+Compressed DCASE JSON can be evaluated in the same way by passing its output
+directory to `--predictions`.
+
+To export predictions without computing metrics:
+
+```bash
+said evaluate "$HOME/datasets/dcase2026_task3" \
+  --model said_passt \
+  --predictions-only
+```
+
+## Paper inference protocol
+
+- selected Eigenmike capsules `[6, 10, 26, 22]` at 48 kHz;
+- two-second windows, retaining the first 20 of 21 detector positions;
+- 10 predictions per second;
+- at most four slots per frame with confidence at least 0.05;
+- map points at least 10% of the corresponding map peak;
+- scores and energy rounded to six decimal places;
+- zero-based 13-class taxonomy;
+- `x_dcase = (179 - x_model) mod 360`.
+
+The evaluated frame count is `max(metadata_frame_index) + 1` from each label
+file. Predictions beyond that interval are not emitted.
+
+## Exact official revisions
+
+| Date | Commit | Change |
+|---|---|---|
+| **(4.2)** | [`84b2cd1`](https://github.com/iranroman/DCASE2026_Task3_SAISELD_baseline/commit/84b2cd155c9ee0c2176f386db54541ffd75312b2) | COCO Mask AP used by the paper |
+| 5.18 | [`2bbbe41`](https://github.com/iranroman/DCASE2026_Task3_SAISELD_baseline/commit/2bbbe41af0815c84122d1df5247ec25d483acbb1) | Replaced COCO masks with interpolated soft maps and introduced the later Pearson framework |
+| 5.28 | [`d47b751`](https://github.com/iranroman/DCASE2026_Task3_SAISELD_baseline/commit/d47b751548a303cad109a2affd5934a12ee2d65e) | Set soft-IoU thresholds to 0.25, 0.50, and 0.75 |
+| **(6.30)** | [`d4df662`](https://github.com/iranroman/DCASE2026_Task3_SAISELD_baseline/commit/d4df66251f39e34bc0157be93858e5a68ec9d7c4) | Corrected AP handling for ground truth with no detection |
+
+The default command downloads the official repository's current `main`, which
+is (6.30) at this release. The optional (4.2) AP source is fixed to `84b2cd1`.
+
+Sources:
+
+```text
+Repository:     https://github.com/iranroman/DCASE2026_Task3_SAISELD_baseline
+Current (6.30): https://raw.githubusercontent.com/iranroman/DCASE2026_Task3_SAISELD_baseline/main/evaluate.py
+Paper AP (4.2): https://raw.githubusercontent.com/iranroman/DCASE2026_Task3_SAISELD_baseline/84b2cd1/evaluate.py
+```
+
+The downloaded source URL and SHA256 are recorded in `evaluator.json`. Metric
+workers process recordings in parallel and merge them in recording order. The
+official metric formulas and parameters are unchanged. The evaluator's 20 MB
+submission guard is checked separately by `said compress` and is not applied
+when reproducing local scores.
+
+Evaluation progress is recorded after each completed recording. Repeating an
+interrupted command validates the checkpoint, dataset, protocol, and completed
+prediction hashes before continuing.