Lui123/tane-mahuta-model

Model card: Tāne Mahuta bird-sound classifier

Status: pending: real model. The architecture, data handling and evaluation method below are fixed; the metrics section is empty on purpose until eval/evaluate.py has been run with TANE_MODEL=real on the held-out test split. Nothing in this card is a measured number unless it names the eval/REPORT.md run it came from.

Model details

• Developer: Lui Hellesoe, for KiwiData (Indigenous-governed AI and data sovereignty venture).
• Task: 10-way classification of 3 s audio windows into ten New Zealand bird species, plus an explicit "unknown" outcome.
• Architecture: ImageNet-pretrained MobileNetV3-Small (torchvision) with its first convolution collapsed to one input channel and a 10-way head, fine-tuned on log-mel spectrograms (DECISIONS.md 19; details in docs/decisions/agent-b.md). A small-cnn-logmel fallback exists behind --arch and is used only if the card says so.
• Input: mono audio resampled to 32 kHz; 3 s windows (96,000 samples); 128-band log-mel spectrogram, n_fft 1024, hop 320, 150 to 15,000 Hz, dB relative to the window peak, floored at -80 dB, per-window standardised. The exact values are read from models/metadata.json at inference so training and serving cannot drift.
• Output: softmax over the ten species_id values in SPECIES.json order (models/labels.json must equal that order or the model refuses to load).
• Version: models/metadata.json: model_version. Weights are CC BY-NC-SA 4.0 (see Licence); the code is MIT.
• Interface: tane/model.py: BirdModel, contract in MODEL_CONTRACT.md.

Species

Ten manu, Māori name first, in model label order (SPECIES.json):

species_id	Māori	English	Scientific
tui	tūī	Tūī	Prosthemadera novaeseelandiae
kereru	kererū	New Zealand pigeon	Hemiphaga novaeseelandiae
ruru	ruru	Morepork	Ninox novaeseelandiae
piwakawaka	pīwakawaka	New Zealand fantail	Rhipidura fuliginosa
korimako	korimako	Bellbird	Anthornis melanura
riroriro	riroriro	Grey warbler	Gerygone igata
kaka	kākā	New Zealand kākā	Nestor meridionalis
pukeko	pūkeko	Pūkeko	Porphyrio melanotus
tauhou	tauhou	Silvereye	Zosterops lateralis
kotare	kōtare	Sacred kingfisher	Todiramphus sanctus

Intended use

• Helping a person in Aotearoa put a name to a bird they can hear, from a phone recording or an uploaded clip, with the top three guesses and their confidence, a spectrogram, and a plain "not confident, possibly none of these" when the model is unsure.
• Demonstrating, in code that can be read and tested, how a small audio model can be built on openly licensed recordings with attribution, consent-based contribution and deletion.
• Education and non-commercial research.

Out-of-scope use

• Any commercial use. The training pool includes CC BY-NC and BY-NC-SA recordings, so the weights, manifest, windows and any derived dataset are non-commercial (CC BY-NC-SA 4.0). This is a licence condition, not a preference.
• Species other than the ten above. The model has no "other bird" class; a tūī-like call from a bird it has never heard will be forced onto one of the ten or returned as unknown.
• Survey, monitoring, compliance or conservation decisions. The evaluation is on a few hundred recordings at most, from one website, with no field validation.
• Identifying people, places or anything other than bird sound. The model sees 3 s log-mel windows and nothing else.
• Anything outside Aotearoa New Zealand: see Limitations.

Training data

• Source: xeno-canto recordings of the ten species, country New Zealand, quality better than C where enough exist (relaxed per species by pipeline/xc.py when under 40 usable recordings; the manifest records what was kept).
• Licences: CC BY, BY-SA, BY-NC, BY-NC-SA 3.0 and 4.0 only; ND, CC0 and unknown licences refused. Redacted (restricted species) records skipped. No recording without a recordist name.
• Manifest: data/manifest.csv, schema DATA_MANIFEST_SCHEMA.json. SHA-256 of the manifest used for training: models/metadata.json: manifest_sha256 (pending: real model).
• Split: 70/15/15 per species by a seeded hash of recording_id (seed 20261017), so no recording's windows appear in two splits. Recordings with another target species audible in the background are forced to train and never scored.
• Windows: non-overlapping 3 s chunks for training, silent windows (RMS under -50 dBFS) dropped. Contributions through the app: none in the pool yet.
• Attribution: every recording is listed in ATTRIBUTION.md, generated by governance/attribution.py.
• Governance: DATA_GOVERNANCE.md.

How a prediction is made

1. The clip is decoded with ffmpeg to mono 32 kHz float, truncated to 60 s.
2. It is cut into 3 s windows with a 1.5 s hop (50 % overlap), the last partial window zero-padded if it holds at least 1 s of audio.
3. Windows with RMS under -50 dBFS are marked silent and skipped.
4. Each remaining window is scored; softmax gives a probability per species.
5. The aggregate per species is the mean of the per-window probabilities over non-silent windows; the top three aggregate values are returned, each with the full species card.
6. unknown is true with reason all_silent if no window was scored, or low_confidence if the top aggregate value is below UNKNOWN_THRESHOLD (tane/constants.py, currently 0.50; pending: value tuned on the val split by eval/evaluate.py --tune-threshold, recorded in DECISIONS.md).

The API (API_CONTRACT.md) also returns every window's top three so the interface can show where in the clip each guess came from.

Evaluation method

eval/evaluate.py (Agent E) scores the held-out test split at two levels:

• window level: every 3 s test window on its own through BirdModel.score_windows, the same feature and forward path the API uses;
• recording level: the windows of each test recording combined with tane.model.aggregate_windows, the same rule as the API, so the recording score is what a person uploading that whole recording would see.

Reported: per-species precision, recall and F1, macro-F1, accuracy with unknown counted as wrong, a confusion matrix with an unknown column, the unknown rate, and counts of windows and recordings. The unknown rate is also reported on an out-of-distribution set (eval/fetch_ood.py: xeno-canto recordings of Eurasian blackbird and Australian magpie from New Zealand, plus five synthetic noise clips) where unknown should be high. The threshold is tuned on val only; the harness refuses to tune on test.

Metrics

Pending: real model. This section is filled by the orchestrator from eval/REPORT.md after python eval/evaluate.py --data data --models models --split test --ood eval/ood has run with TANE_MODEL=real. Until then the eval/REPORT.md in the repository is a run of Agent B's smoke model (version 0.1.0, 8 epochs on the 33-recording data/sample split, val macro-F1 0.07 by its own metadata) on the sample's 8 test recordings, and eval/REPORT.stub.md is the stub-model run, labelled as such in its first line. Neither says anything about how a fully trained model performs and neither may be copied here.

To be filled, with the manifest SHA-256 and model version from eval/results.json: meta:

Metric	Window level	Recording level
Accuracy (unknown = wrong)	pending	pending
Macro-F1	pending	pending
Unknown rate on test	pending	pending
Per-species precision / recall / F1	see eval/REPORT.md	see eval/REPORT.md
OOD unknown rate (blackbird, magpie, noise)	n/a	pending
Chosen UNKNOWN_THRESHOLD and the sweep that chose it	n/a	pending
Test recordings / windows	pending	pending

Limitations and bias

• Recording-quality bias. The pool prefers xeno-canto quality A and B, which are mostly close, clean recordings made with directional microphones. A phone held up in the wind at a distance is a different distribution; expect lower confidence and more unknowns on real phone clips than the test split suggests.
• Recordist and place bias. xeno-canto's New Zealand uploads come from a small number of recordists and the places they visit, so sanctuaries and accessible bush are over-represented; a species' regional dialects (bellbird and tūī vary a lot by place) may be missing.
• Small data. Tens to a few hundred recordings per species. Per-species metrics on the test split rest on a handful of recordings for the rarer species and will swing with a single recording.
• Ten species only, NZ recordings only. Common introduced birds (blackbird, song thrush, starling, magpie) are not in the label set. The OOD check measures how often they are correctly returned as unknown; it does not make the model able to name them.
• Label noise. Species labels come from the recordist. Background birds the recordist did not list can be present; those listed as another target species are kept out of val and test but not out of train.
• Window independence. Aggregation is a plain mean over windows; a long clip dominated by silence or by one bird will drown out a second species.
• The unknown threshold is a single number on the aggregate top-1 and is tuned on in-domain val data plus a tiny OOD set; it is not calibrated.
• No consultation with iwi or hapū has taken place about the use of these species' calls; see DATA_GOVERNANCE.md section 6.

Licence

Model weights, data/manifest.csv, derived windows and any dataset built from them: CC BY-NC-SA 4.0, because the training pool contains CC BY-NC and BY-NC-SA recordings. Code: MIT. Each training recording keeps its own licence, listed in ATTRIBUTION.md.

Contact and changes

Governance questions and removal requests: see DATA_GOVERNANCE.md section 5. Changes to any constant in this card go through tane/constants.py and DECISIONS.md.