Create README.md

README.md

@@ -0,0 +1,328 @@
+---
+license: mit
+language:
+- en
+tags:
+- audio
+- sound_event_detection
+- bioacoustics
+---
+
+# Sound Event Detection
+Run using code available on [`Github`](https://github.com/earthspecies/sound-event-detection)
+
+Pretrained sound event detection models focused on bioacoustics. Supports three main functions:
+
+- Inference with pre-trained models: Within python, via a script, or via the large-scale inference (LSI) pipeline.
+- Evaluation of model performance on detection datasets.
+- Load pre-computed model detections for datasets like Xeno-Canto and iNaturalist.
+
+## Installation
+
+Requires [`uv`](https://docs.astral.sh/uv/). Installation may take several minutes. GPU is not required but will improve speed.
+
+Required packages are listed in `pyproject.toml`. To install them, run:
+
+```bash
+uv sync --group gpu   # omit --group gpu for CPU-only
+```
+
+All commands run through `uv run`. It may be necessary to include `--group gpu` if using a GPU. Evaluation and LSI also need the dataset storage referenced by `configs/data/*.yml`.
+
+Large-scale inference and using precomputed selection tables both require [`alp-data`](https://github.com/earthspecies/alp-data/), which is already included in `pyproject.toml`.
+
+## Quick start — BirdCODE over a folder of audio
+
+Run the pretrained BirdCODE detector (loaded from the Hub) over every audio file in a folder — any sample rate, resampled to 32 kHz as needed — and write a selection table next to each recording: `dir/x.wav` → `dir/BirdCODE_predictions/x.txt`. Currently supports wav, flac, ogg, and mp3.
+
+```bash
+uv run sed-folder --folder /path/to/audio
+```
+
+Two short demo recordings are provided. To run BirdCODE on them, do:
+
+```bash
+uv run sed-folder --folder tests/samples/demo/audio
+```
+
+This writes `tests/samples/demo/audio/BirdCODE_predictions/{20230730,20260623}.txt`, which should match the tables in `tests/samples/demo/output_expected/`. On CPU it takes roughly 1.5 minutes after the model weights (~1.1 GB) are downloaded.
+
+Postprocessing is applied: By default, per-frame detections are thresholded at 0.5, boxes with the same label are merged if separated by less than 1 second, and non-maximal suppression is applied with an IoU threshold of 0.8. Geography filtering is off by default; enable it with `--geo-filter`, a directory of `*.gpkg` range maps, and the recording site's coordinates (applied to every file):
+
+```bash
+uv run sed-folder --folder /path/to/audio \
+    --geo-filter --range-map-dir geography/range_maps \
+    --latitude 42.5 --longitude -72.2
+```
+
+## Official models
+
+| Model | Publication | Checkpoint | Summary |
+|---|---|---|---|
+| BirdCODE | TODO | [EarthSpeciesProject/sed-birdcode](https://huggingface.co/EarthSpeciesProject/sed-birdcode) | Bird Communication Detector |
+
+## CLI entry points
+
+| Command | Purpose | Assumes running |
+|---|---|---|
+| `sed-folder` | Run BirdCODE over a folder of audio → selection tables | — (loads the model in-process) |
+| `sed-server` | Serve a frame detector or sliding-window detector | backing classifier server (sliding-window only) |
+| `sed-denoising-server` | Serve the denoising detector | a detector server + a separator server |
+| `sed-eval` | Run an evaluation against a served model | a `sed-server` / `sed-denoising-server` server |
+| `sed-lsi` | Large-scale inference over a dataset | a `sed-server` (`preds`) or `sed-denoising-server` (`denoised`/`stems`) server |
+| `sed-lsi-postprocess` | Turn LSI predictions into selection tables | — (reads shards) |
+| `sed-lsi-features` | Add per-event acoustic features to selection tables | — (reads shards) |
+
+Every CLI has a `describe` subcommand that prints its config schema(s), e.g. `uv run sed-eval describe`.
+
+## Using BirdCODE in Python
+
+`FrameDetector` loads a trained detector in-process, either from the HuggingFace Hub by repo id or from a checkpoint directory (local, `gs://…`, or `r2://…`):
+
+```python
+from sound_event_detection.models import FrameDetector
+
+# From the HuggingFace Hub (downloads the snapshot, then rebuilds the model);
+birdcode = FrameDetector.from_hf_hub("EarthSpeciesProject/sed-birdcode").eval().to("cuda")
+
+# Or from a checkpoint directory: weights from best_model.pt, labels from
+# labels.txt, architecture from config.yaml.
+ckpt = "checkpoints/birdcode_esp_research"
+birdcode = FrameDetector.from_checkpoint_dir(ckpt, f"{ckpt}/config.yaml").to("cuda")
+
+out = birdcode.run(audio, overlap=0.5)   # audio: np.ndarray [batch, samples] at 32 kHz
+out.predictions                          # [batch, time, classes] probabilities in [0, 1]
+out.class_names                          # list[str] labels aligned to the classes axis
+```
+
+## Serving models
+
+For large-scale inference and evaluation, we serve the model over HTTP, then point a client CLI at it via an http-client config.
+
+A **model config** YAML tells the server what to load, dispatching on `type`. The unified server (`sed-server`) reads its path from the `SED_MODEL_CONFIG` environment variable.
+
+### Frame detectors — `type: frame`
+
+Trained detectors (BirdCODE and ablations) loaded either from the HuggingFace Hub or from a local checkpoint directory. All current checkpoints run at 32 kHz.
+
+Set `hf_repo_id` to download and serve a checkpoint from the Hub — this is how the example config loads BirdCODE. An optional `revision` pins a branch, tag, or commit (defaults to the repo's default branch):
+
+```yaml
+type: frame
+hf_repo_id: EarthSpeciesProject/sed-birdcode
+# revision: main   # optional
+```
+
+Alternatively, `model_folder` serves a local checkpoint directory (expects `config.yaml`, `best_model.pt`, and `labels.txt`).
+
+Serve either config the same way:
+
+```bash
+SED_MODEL_CONFIG=configs/birdcode/models/birdcode_esp_research.yml \
+    uv run sed-server --host 0.0.0.0 --port 8100
+```
+
+`sed-server` accepts `--host` (default `localhost`), `--port` (default `8100`), `--workers`, `--reload`, and `--log-level`. `SED_DEVICE=cpu|cuda` selects the device (default: cuda if available).
+
+Ablation checkpoints use the same `type: frame` shape:
+`configs/birdcode/models/ablations/`.
+
+### Sliding-window detectors — `type: perch2 | audioprotopnet | beats_sl_all`
+
+Clip classifiers wrapped in a `SlidingWindowDetector` to produce frame-level predictions. Each needs a **backing classifier server** already running, discovered through `addr_file` (a text file containing `host:port`):
+
+```yaml
+type: audioprotopnet
+addr_file: ~/audioprotopnet-server/server.addr
+window_size: 5.0        # seconds
+hop_size: 2.0           # seconds
+analysis_window: 2.0    # optional; defaults to window_size
+```
+
+| Type | Backing server | Sample rate |
+|---|---|---|
+| `perch2` | [earthspecies/perch2-server](https://github.com/earthspecies/perch2-server) | 32 kHz |
+| `audioprotopnet` | [earthspecies/audioprotopnet-server](https://github.com/earthspecies/audioprotopnet-server) | 32 kHz |
+| `beats_sl_all` | in-repo (below) | 16 kHz |
+
+The external servers write their own `server.addr`; point the config's `addr_file` at it. Serve the wrapper the same way as a frame detector:
+
+```bash
+SED_MODEL_CONFIG=configs/birdcode/models/baselines/audioprotopnet_2s.yml \
+    uv run sed-server --port 8100
+```
+
+`beats_sl_all` runs at 16 kHz — evaluate it with `frame_eval_16k.yml` (frame detection) or `birdset_clip_eval_16k.yml` (clip classification). Its backing classifier is served in-repo:
+
+```bash
+# 1. backing classifier (16 kHz), then record its host:port
+SED_DEVICE=cuda uv run uvicorn \
+    sound_event_detection.serving.sl_beats_all_server:app --host 0.0.0.0 --port 8200
+echo "HOST:8200" > .server_addrs/beats_sl_all.addr   # path the config's addr_file points at
+
+# 2. the sliding-window wrapper
+SED_MODEL_CONFIG=configs/birdcode/models/baselines/beats_sl_all_2s.yml \
+    uv run sed-server --port 8100
+```
+
+### Denoising detector — `type: denoising_detector`
+
+NOTE: This requires a separator server to be running. Separator server code will be provided at a later date.
+
+Wraps a detector client and a source-separator client, adding `POST /separate_and_detect` (used by LSI) to the standard contract. Both backing servers must be up when it starts. Its model config names them as pure http-client configs:
+
+```yaml
+type: denoising_detector
+detector:  {url: http://localhost:8100, timeout: 300}   # a sed-server detector server
+separator: {url: http://localhost:8200, timeout: 300}   # a separator server
+threshold: 0.5
+resampling_method: torchaudio_kaiser_fast
+```
+
+```bash
+# with a detector server and a separator server already running:
+SED_MODEL_CONFIG=configs/birdcode/models/denoising_detector.yml \
+    uv run sed-denoising-server --host 0.0.0.0 --port 8110
+```
+
+`sed-denoising-server` takes the same options as `sed-server` (default port `8110`).
+
+### HTTP contract
+
+- `GET /` — model metadata: `{labels, sample_rate, frame_rate, window_duration}`
+- `GET /health` — `{status: "ok"}` once the model is loaded
+- `GET /labels` — ordered label list
+- `POST /run` — frame-level inference; response `{predictions, shape [batch, time, classes], frame_rate}`
+- `POST /run_as_classifier` — clip-level pooled inference; response shape `[batch, classes]`
+- `POST /separate_and_detect` — denoising server only; per-stem audio + predictions
+
+## Evaluation — `sed-eval`
+
+Serve a model, then run `sed-eval` against it with an **eval config** (*what* to evaluate) and an **http-client config** (*how* to reach the model — a `url` plus optional `timeout`/`retries`/`auth`; the client kind is auto-detected from the server).
+
+```bash
+# write an http-client config pointing at the running server, e.g.:
+#   url: http://HOST:8100
+uv run sed-eval --eval-config configs/birdcode/frame_eval.yml \
+    --httpclient-config configs/birdcode/httpclient.yml \
+    [--checkpoint-dir <dir>] [--output-dir <dir>]
+```
+
+- `--checkpoint-dir` — resumable checkpoint directory (auto-generated under `checkpoints/sed/` if omitted).
+- `--output-dir` — override the eval config's `output_dir`.
+- `sed-eval --resume <checkpoint-dir>` — resume a run; configs are reloaded from the checkpoint.
+
+### Eval configs
+
+| Config | Pathway | Datasets | Sample rate |
+|---|---|---|---|
+| `configs/birdcode/frame_eval.yml` | frame (detection) | 68 WABAD sites + Powdermill + XC-AJ | 32 kHz |
+| `configs/birdcode/birdset_clip_eval.yml` | clip (classification) | 8 BirdSet test splits | 32 kHz |
+
+An eval config selects the pathway through its dataset lists: `frame_datasets` (strong labels, with `species_column`) go through detection; `clip_datasets` (weak labels) through classification.
+
+### Metrics
+
+- **Frame pathway**: frame mAP, event mAP per IoU threshold, thresholded precision/recall/F1.
+- **Clip pathway**: cmAP (headline), cmAP5, mAP, pcmAP, MultilabelAUROC, top-1/top-3 accuracy, per-class AP, and `gt_coverage`.
+
+### Results
+
+Each eval run writes `<output_dir>/results.yaml`, updated after every dataset:
+
+- `model` — the served model's metadata (`GET /` response)
+- `frame_eval` — the scoring parameters used
+- `frame_datasets.<name>` — per-dataset detection metrics
+- `clip_datasets.<name>` — per-dataset classification metrics
+
+## Large-scale inference (LSI)
+
+Run a served detector over a dataset, persist per-recording results as compressed `.npz` shards, then postprocess (and optionally enrich) them into selection tables. Three stages: **run → postprocess → features**. Each stage takes `--job-index N --num-jobs M` to split the work across an array of parallel jobs, and writes a `lineage.yaml` chaining back to the stage that produced its input.
+
+The LSI configs (`configs/inference/lsi_birdcode_*.yml`) run the BirdCODE frame detector over the full Xeno-Canto and iNaturalist training splits; they read their datasets from `configs/data/inference/`.
+
+### Run — `sed-lsi`
+
+Builds a dataset from a **run config** (*what* to run) and a detector client from an **http-client config** (*how* to reach the model), then runs the sharded engine over this job's slice.
+
+```bash
+# with the appropriate server running (see below):
+uv run sed-lsi --run-config configs/inference/lsi_birdcode_xc.yml \
+    --httpclient-config <httpclient.yml> [--job-index N --num-jobs M] [--output-dir DIR]
+```
+
+The run config's `output.detail` selects what is stored per recording — and which server the `url` must reach:
+
+| `detail` | Stored | Server |
+|---|---|---|
+| `preds` | combined framewise predictions | a `sed-server` detector server |
+| `denoised` | predictions + a threshold-gated denoised waveform | a `sed-denoising-server` server |
+| `stems` | the above + every separated stem (audio + preds) | a `sed-denoising-server` server |
+
+### Postprocess — `sed-lsi-postprocess`
+
+Reads the combined predictions in each shard and writes a per-recording selection table (1:1 with the input shards). Re-postprocessing is a cheap re-run into a sibling directory.
+
+```bash
+uv run sed-lsi-postprocess --config configs/inference/lsi_birdcode_xc_postprocess.yml \
+    --run-dir <run_dir> [--job-index N --num-jobs M]
+```
+
+`--run-dir` overrides the config's `input.run_dir` (postprocess several runs
+with one config).
+
+#### Geography filtering
+
+Setting `postprocessing.geo_filter: true` drops detections for species whose range maps exclude a recording's location (using the latitude/longitude stored in each shard). It requires `postprocessing.range_map_dir` — a directory (local path or cloud URI) of `*.gpkg` range-map files, globbed at startup and checked to exist before any shards are processed:
+
+```yaml
+postprocessing:
+  geo_filter: true
+  range_map_dir: geography/range_maps   # dir of *.gpkg range maps
+```
+
+To use geography filtering, download the open range-map dataset from iNaturalist (<https://www.inaturalist.org/pages/range_maps>) into `range_map_dir`. Each range map's species `name` is resolved to a GBIF canonical name to match the detector's labels. The filter fails open: a detection is dropped only on positive out-of-range evidence (valid coordinates **and** a range map that excludes the point); recordings without coordinates, or species without a range map, are left untouched.
+
+### Features — `sed-lsi-features`
+
+Enriches a postprocessed selection table with per-event `v0minimal` acoustic features. Writes enriched selection tables 1:1 with the postprocess shards.
+
+```bash
+uv run sed-lsi-features --config configs/inference/lsi_birdcode_xc_features.yml \
+    --run-dir <run_dir> --postprocessing postprocessed_thr0.50_merge1.00_nms0.80_geo \
+    [--job-index N --num-jobs M]
+```
+
+## Loading a dataset with attached selection tables (Python)
+
+Public GCS buckets hold BirdCODE detections as selection tables for a subset of **Xeno-Canto** and **iNaturalist** recordings. Two data configs load each corpus with those tables attached via the `attach_lsi_selection_tables` transform:
+
+- `configs/data/inference/xeno_canto_selection_tables.yml`
+- `configs/data/inference/inaturalist_selection_tables.yml`
+
+Load either with `alp_data.dataset_from_config`, importing the transforms module first so the custom transform is registered:
+
+```python
+import io
+import pandas as pd
+from alp_data import dataset_from_config
+import sound_event_detection.data.transforms  # noqa: F401 — registers attach_lsi_selection_tables
+
+dataset, meta = dataset_from_config("configs/data/inference/xeno_canto_selection_tables.yml")
+print(meta["attach_lsi_selection_tables"])  # {'matched': ..., 'unmatched': ...}
+
+# The attached `selection_table` column lives on the metadata backend
+# (`dataset._data`), so you can read it without decoding audio. It is a TSV
+# string (empty for unmatched rows); parse it into a DataFrame of events:
+for row in dataset._data:
+    if row["selection_table"]:
+        events = pd.read_csv(io.StringIO(row["selection_table"]), sep="\t")
+        break
+```
+
+Each row of a parsed `selection_table` is one detection event, with columns:
+
+- `Begin Time (s)`, `End Time (s)` — the event's span within the recording
+- `Species` — predicted class label
+- `Score` — mean BirdCODE probability over the event
+- 13 `v0minimal` acoustic-feature columns (see `sound_event_detection.inference.features_v0minimal.FEATURE_COLS`)