OceanGuard AI: BioDex species pack

Files that the OceanGuard AI Android app downloads on demand to identify marine species on the device, without a connection. The photo is encoded with BioCLIP 2 and compared with reference embeddings of 232 species, mostly Mediterranean plus 30 tropical species often seen by divers: prototypes of real photos plus the embedding of each species name. Below the calibrated confidence thresholds the app answers "not identified" or shows the closest candidates as "possible"; since pack v4 an unidentified photo can still get its group (for example "sharks and rays"). Only confident identifications are saved.

Packs

Pack Manifest Used by Encoder Species
v5 (current, manifest version 7) species_pack_v5.json app builds after 1.2.0-rc6 v4 pack plus species sprites (version 6 redraws 14 sprites whose source photo cut part of the animal; version 7 redraws 2 more) 232, all with a sprite
v4 species_pack_v4.json app 1.2.0-rc5 and later same encoder file as v3 232 (227 with reference photos, 5 by name only)
v3 species_pack_v3.json app 1.2.0-rc3 / rc4 BioCLIP 2 ViT-L/14, fp16 weights with fp32 compute (609 MB) 202 (197 with reference photos, 5 by name only)
v2 species_pack.json app 1.2.0-rc1 / rc2 BioCLIP ViT-B/16 fp32 (345 MB) 198
v1 (none) app up to 1.1.1 OpenCLIP ViT-B/32 (352 MB) 203

v4 files (the encoder is the v3 file, unchanged)

File Description
species_pack_v4.json Encoder contract (224 px, CLIP mean/std, 768 dimensions), scoring (image prototypes + 4 x species-name embedding), softmax temperature and thresholds calibrated on a validation split, session optimisation level, input crop mode, group threshold (groupFallback), SHA-256 of every file and a numeric self-test (the app refuses the encoder if the device result has cosine < 0.99).
biodex_bioclip2_v3.onnx BioCLIP 2 image encoder. Input pixel_values [1, 3, 224, 224], output image_features [1, 768].
species_index_v4.bin Reference index: 6 prototypes per species (k-means) from 23,668 photos, plus one text row per species (SPEX format).
species_catalog_v3.json Catalogue: verified common names in 6 languages with their source, AphiaID checked against WoRMS, group, short descriptions built only from WoRMS, IUCN and distribution data, MEOW ecoregions.
reference_provenance_v4.jsonl Observation, photo, licence and attribution of every reference photo.

meow_raster_v1.bin and ecoregion_hierarchy_v1.json (Marine Ecoregions of the World) are shared by all packs for the geographic prior.

v5 files (everything else is the v4 pack, unchanged)

File Description
species_pack_v5.json The v4 manifest plus sprites and spriteCredits, with the SHA-256 of each new file. Updating from v4 downloads only these files (about 0.8 MB).
sprites/<speciesKey>.webp Pixel-art sprite of each of the 232 species (256 px, lossless WebP), shown in the BioDex.
sprite_credits.json Source image of each sprite: attribution, licence and page.

Each sprite is drawn from one image of the live, whole animal: 217 iNaturalist photos (199 CC BY, 18 CC0) and 15 Wikimedia Commons images (9 public domain, mostly 19th-century naturalist plates, and 6 CC BY). The organism is cut out, redrawn with Stable Diffusion XL (img2img with an edge ControlNet and a style reference) and reduced to a 128-pixel grid.

Evaluation (pack v4)

Through the app's own chain on photos that are not in the reference index. Thresholds chosen on validation for precision >= 0.95 of the confident answers.

Test set Top-1 Top-5 Confident answers: share / precision Out-of-catalogue photos accepted as confident
iNaturalist, observers disjoint from the references (3,550 photos; 318 out-of-catalogue) 0.893 0.980 63 % / 98.2 % 8.8 %
Other sources: GBIF datasets other than iNaturalist and Wikimedia Commons (4,912 photos; 186 out-of-catalogue) 0.814 0.937 57 % / 97.7 % 3.8 %

The out-of-catalogue photos of v4 include look-alike relatives of catalogue species, so its confident share is lower than v3 (69 % / 62 % on the v3 sets) at the same precision. The 30 new species: top-1 0.89 on both sets, confident answers 99-100 % right. When no species is confident, the group answer covers 59 % of those photos and is right 93 % of the time (threshold chosen on validation for 95 %). "Possible" answers (shown, not saved) are right about 73 % of the time. The labels of the second set have not been checked by an expert. On a PC CPU (4 threads) one photo takes about 0.5 s with about 0.8 GB of memory; phone figures are not measured yet.

Licences

  • Reference index and provenance (species_index_*.bin, reference_provenance*.jsonl): CC BY 4.0. Derived from iNaturalist research-grade photos licensed CC0 or CC BY only; the photos are not redistributed. Attribution of each photo is in the provenance file.
  • Sprites (sprites/, sprite_credits.json): CC BY 4.0. Each one is derived from a CC0, CC BY or public-domain image; its author, licence and page are in sprite_credits.json and the app shows them on the species page.
  • Encoders: BioCLIP 2 (imageomics/bioclip-2), BioCLIP (imageomics/bioclip) and OpenCLIP ViT-B/32: MIT licence.
  • Catalogue: names and classification from WoRMS (CC BY 4.0), FAO ASFIS and GBIF; species ranges from OBIS and GBIF (CC BY 4.0) and AquaMaps (CC BY-NC; check its terms before any commercial use).
  • MEOW (Spalding et al., 2007, The Nature Conservancy): free use with attribution.

Citation

@inproceedings{gu2025bioclip2,
  title     = {{BioCLIP 2}: Emergent Properties from Scaling Hierarchical Contrastive Learning},
  author    = {Gu, Jianyang and Stevens, Samuel and Campolongo, Elizabeth G. and Thompson, Matthew J. and Zhang, Net and Wu, Jiaman and Kopanev, Andrei and Mai, Zheda and White, Alexander E. and Balhoff, James and Dahdul, Wasila and Rubenstein, Daniel and Lapp, Hilmar and Berger-Wolf, Tanya and Chao, Wei-Lun and Su, Yu},
  booktitle = {NeurIPS},
  year      = {2025}
}

@inproceedings{stevens2024bioclip,
  title     = {{BioCLIP}: A Vision Foundation Model for the Tree of Life},
  author    = {Stevens, Samuel and Wu, Jiaman and Thompson, Matthew J. and Campolongo, Elizabeth G. and Song, Chan Hee and Carlyn, David Edward and Dong, Li and Dahdul, Wasila M. and Stewart, Charles and Berger-Wolf, Tanya and Chao, Wei-Lun and Su, Yu},
  booktitle = {CVPR},
  year      = {2024}
}

Code to rebuild and evaluate the packs: tools/species/ and docs/SPECIES_ID.md in asferrer/oceanguard-app.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support