v0.4.0

README.md

@@ -0,0 +1,151 @@
+---
+license: apache-2.0
+library_name: onnx
+pipeline_tag: image-classification
+tags:
+  - multi-label
+  - image-attributes
+  - onnx
+  - efficientnetv2
+---
+
+# WhatDoWeSee-EfficientNetV2-B0
+
+Tags **what is visible in an image** with any number of 20 attributes, such as
+people, text, chart, diagram, table, map or aerial view. A small, fast EfficientNetV2-B0 model (24 MB ONNX, about 14 ms per image on one CPU thread)
+for sorting and describing large image collections.
+
+This is **v0.4.0** (2026-10-16): trained on 28,495 images labelled by three open vision-language models (Clef-omni, Gemma 4 26B, Qwen3-VL 32B), learning from how many of them agree.
+Try it in your browser: [WhatDoWeSee Space](https://huggingface.co/spaces/StephanMeijer/WhatDoWeSee).
+
+## Attributes
+
+An image gets every attribute that is clearly recognisable, also when blurry or in the background;
+tiny or unrecognisable details don't count. Depicted things count (a drawn dog is Animals).
+
+- **People** (`people`): one or more humans, or clearly human faces, hands or bodies
+- **Animals** (`animals`): animals, birds, fish or insects
+- **Plants** (`nature`): living green things: trees, plants, flowers, crops; not grass or a lawn as ground, not sea, sky or rocks (those are Landscape)
+- **Landscape** (`landscape`): an outdoor view of land, sea, sky, mountains, rocks, rivers or a city as a whole
+- **Buildings / structures** (`buildings`): buildings, interiors where the room or its architecture is clearly visible, and other built structures: bridges, towers, dams, monuments, ruins
+- **Vehicles** (`vehicles`): cars, bikes, trains, ships, planes or other vehicles
+- **Food** (`food`): food or drinks
+- **Text** (`text`): running text, sentences, lists or a page that make up a substantial part of the image; labels, captions, titles and signs don't count
+- **Handwriting** (`handwriting`): handwritten text, notes or signatures
+- **Chart** (`chart`): a data plot: values on axes or as proportions (bar, line, pie, scatter, area chart or similar)
+- **Diagram** (`diagram`): structure or relationships without measured values: a flowchart, schematic, process or relationship diagram; also labelled drawings that show how something is built (anatomy, parts, cross-sections)
+- **Table** (`table`): text or numbers laid out in rows and columns, with or without grid lines; not the furniture
+- **Map** (`map`): a geographic map, floor plan or route map
+- **Logo / icon** (`logo-icon`): a logo, icon, emblem, badge or pictogram
+- **Formula** (`formula`): mathematical, chemical or other notation
+- **Screen / UI** (`screen`): an app, website or device interface
+- **Artwork** (`artwork`): made as art: a painting, illustration, cartoon or sculpture, as the image itself or shown in it; not drawings made to explain (those are Diagram)
+- **Instruments** (`instruments`): musical instruments: guitar, piano, violin, drums, wind instruments and the like
+- **Furniture** (`furniture`): chairs, tables, beds, sofas, desks, cabinets, shelves and other furniture (a data table is Table)
+- **Aerial** (`aerial`): the ground seen from high above: aerial or satellite photos, drone shots, bird's-eye views, also through a plane window, straight down or at an angle; clouds without ground don't count; a drawn map is Map
+
+## Usage
+
+Download a fixed version (recommended), so a later release never changes your results:
+
+```python
+from huggingface_hub import hf_hub_download
+
+model_path = hf_hub_download("StephanMeijer/WhatDoWeSee-EfficientNetV2-B0", "model.onnx", revision="v0.4.0")
+config_path = hf_hub_download("StephanMeijer/WhatDoWeSee-EfficientNetV2-B0", "config.json", revision="v0.4.0")
+```
+
+Then:
+
+```python
+import json
+import numpy as np
+import onnxruntime as ort
+from PIL import Image
+
+config = json.load(open(config_path))
+session = ort.InferenceSession(model_path)
+size = config["input"]["shape"][-1]
+mean = np.array(config["input"]["mean"], dtype=np.float32)
+std = np.array(config["input"]["std"], dtype=np.float32)
+
+image = Image.open("photo.jpg").convert("RGB").resize((size, size), Image.BILINEAR)
+x = ((np.asarray(image, dtype=np.float32) / 255 - mean) / std).transpose(2, 0, 1)[None]
+probabilities = 1 / (1 + np.exp(-session.run(None, {"pixel_values": x})[0][0]))
+present = [label for label, p in zip(config["labels"], probabilities)
+           if p >= config["thresholds"][label]]
+```
+
+Preprocessing: RGB, resized to 224×224 (antialiased bilinear, no crop, aspect ratio not kept),
+scaled to [0, 1], normalised with the ImageNet mean and standard deviation, NCHW float32. Each label has
+its own threshold in `config.json`, tuned on a validation split.
+
+## Evaluation
+
+On 3435 held-out teacher-labelled images: mean average precision **0.91**,
+macro F1 **0.85** at the tuned thresholds.
+
+| label | name | positives | AP | F1 |
+|---|---|---:|---:|---:|
+| `people` | People | 626 | 0.95 | 0.89 |
+| `animals` | Animals | 268 | 0.87 | 0.82 |
+| `nature` | Plants | 382 | 0.89 | 0.82 |
+| `landscape` | Landscape | 438 | 0.93 | 0.84 |
+| `buildings` | Buildings / structures | 422 | 0.91 | 0.84 |
+| `vehicles` | Vehicles | 185 | 0.82 | 0.74 |
+| `food` | Food | 106 | 0.76 | 0.72 |
+| `text` | Text | 1035 | 0.95 | 0.90 |
+| `handwriting` | Handwriting | 243 | 0.86 | 0.80 |
+| `chart` | Chart | 260 | 0.98 | 0.95 |
+| `diagram` | Diagram | 235 | 0.94 | 0.93 |
+| `table` | Table | 275 | 0.97 | 0.91 |
+| `map` | Map | 169 | 0.95 | 0.92 |
+| `logo-icon` | Logo / icon | 551 | 0.84 | 0.77 |
+| `formula` | Formula | 30 | 0.90 | 0.81 |
+| `screen` | Screen / UI | 377 | 0.97 | 0.92 |
+| `artwork` | Artwork | 474 | 0.94 | 0.88 |
+| `instruments` | Instruments | 64 | 0.87 | 0.85 |
+| `furniture` | Furniture | 183 | 0.81 | 0.73 |
+| `aerial` | Aerial | 161 | 0.97 | 0.93 |
+
+### How it compares
+
+Against its teachers on the 3,435 test images: mean AP 0.91, macro F1 0.85 at 0.5 (MobileNetV4
+v0.4.0-rc.1 on the same images: 0.88 and 0.83). On the 5,158 validation images macro F1 at 0.5
+is 0.857, about the same as v0.3.0's 0.858, but v0.3.0 trained on most of those images; on
+the part it never saw, v0.3.0 reaches 0.74. About 14 ms per image on one CPU thread.
+Against v0.3.0 on 60 independent hand-labelled images: slightly worse (mean AP 0.81 against
+0.87, macro F1 at 0.5 0.76 against 0.77 to 0.78). Better on Animals, Screen / UI, Map and
+Artwork; weaker on People, Landscape, Handwriting and Logo / icon. Hand labels and teacher
+labels differ in what they count, and per-label scores on the 60 rest on 1 to 24 images:
+indicative only. A larger EfficientViT-B2 model scored higher (confirm macro F1 0.86) at four
+times the CPU time; it was not released.
+
+## Training
+
+- Backbone: `tf_efficientnetv2_b0.in1k` (ImageNet-pretrained, via timm), with a new
+  20-way sigmoid head; 30 epochs, AdamW, binary cross-entropy per label.
+- Data: The same teachers and images as WhatDoWeSee-MobileNetV4 v0.4.0-rc.1: Cloudflare Clef-omni, Gemma 4 26B-A4B and Qwen3-VL 32B, run on EuroHPC (Discoverer+), answered the attribute question for all 44,361 images of the full-v1 corpus. No hosted API and no hand labels are involved; the 681 hand-labelled images are left out entirely, and near-duplicates (pHash) stay in one split. New in v0.4.0: instead of a yes/no target, the model learns from the vote count (all three agree 0.95, two 0.35, one 0.10, none 0.02). Trained on 28,495 images, 30 epochs from ImageNet weights, best epoch picked on 1,441 held-out training images. The backbone (tf_efficientnetv2_b0, 224 px) won a champion-challenger selection over 35 backbones and several training variants, scored on 5,158 validation images where all three teachers agree; thresholds are tuned on those. Test: 3,435 teacher-labelled images it never trained, tuned or was selected on.
+- The images come from public collections: photo datasets, document and form scans, charts,
+  diagrams, screenshots, maps, artwork and icons.
+
+## Limitations
+
+- Small text-related details (a small table inside a document, a signature) are harder than
+  whole-image subjects.
+- The training images come from many public collections, some under research-only or
+  non-commercial terms. Check whether that matters for your use.
+
+## Licence
+
+The model weights and `config.json` are released under the Apache License 2.0, like the
+EfficientNetV2-B0 backbone they start from. The licence covers the model, not the images it was trained on.
+
+## Versions
+
+Each version on test images it never trained on (mean AP and macro F1). The test sets
+differ between some versions, so compare versions through the notes of each version:
+
+| version | date | test images | mean AP | macro F1 | |
+|---|---|---:|---:|---:|---|
+| **v0.4.0** | 2026-10-16 | 3435 | 0.91 | 0.85 | trained on 28,495 images labelled by three open vision-language models (Clef-omni, Gemma 4 26B, Qwen3-VL 32B), learning from how many of them agree |

config.json

@@ -0,0 +1,118 @@
+{
+  "version": "v0.4.0",
+  "task": "multi-label image attributes",
+  "group": "content",
+  "labels": [
+    "people",
+    "animals",
+    "nature",
+    "landscape",
+    "buildings",
+    "vehicles",
+    "food",
+    "text",
+    "handwriting",
+    "chart",
+    "diagram",
+    "table",
+    "map",
+    "logo-icon",
+    "formula",
+    "screen",
+    "artwork",
+    "instruments",
+    "furniture",
+    "aerial"
+  ],
+  "names": [
+    "People",
+    "Animals",
+    "Plants",
+    "Landscape",
+    "Buildings / structures",
+    "Vehicles",
+    "Food",
+    "Text",
+    "Handwriting",
+    "Chart",
+    "Diagram",
+    "Table",
+    "Map",
+    "Logo / icon",
+    "Formula",
+    "Screen / UI",
+    "Artwork",
+    "Instruments",
+    "Furniture",
+    "Aerial"
+  ],
+  "definitions": {
+    "people": "one or more humans, or clearly human faces, hands or bodies",
+    "animals": "animals, birds, fish or insects",
+    "nature": "living green things: trees, plants, flowers, crops; not grass or a lawn as ground, not sea, sky or rocks (those are Landscape)",
+    "landscape": "an outdoor view of land, sea, sky, mountains, rocks, rivers or a city as a whole",
+    "buildings": "buildings, interiors where the room or its architecture is clearly visible, and other built structures: bridges, towers, dams, monuments, ruins",
+    "vehicles": "cars, bikes, trains, ships, planes or other vehicles",
+    "food": "food or drinks",
+    "text": "running text, sentences, lists or a page that make up a substantial part of the image; labels, captions, titles and signs don't count",
+    "handwriting": "handwritten text, notes or signatures",
+    "chart": "a data plot: values on axes or as proportions (bar, line, pie, scatter, area chart or similar)",
+    "diagram": "structure or relationships without measured values: a flowchart, schematic, process or relationship diagram; also labelled drawings that show how something is built (anatomy, parts, cross-sections)",
+    "table": "text or numbers laid out in rows and columns, with or without grid lines; not the furniture",
+    "map": "a geographic map, floor plan or route map",
+    "logo-icon": "a logo, icon, emblem, badge or pictogram",
+    "formula": "mathematical, chemical or other notation",
+    "screen": "an app, website or device interface",
+    "artwork": "made as art: a painting, illustration, cartoon or sculpture, as the image itself or shown in it; not drawings made to explain (those are Diagram)",
+    "instruments": "musical instruments: guitar, piano, violin, drums, wind instruments and the like",
+    "furniture": "chairs, tables, beds, sofas, desks, cabinets, shelves and other furniture (a data table is Table)",
+    "aerial": "the ground seen from high above: aerial or satellite photos, drone shots, bird's-eye views, also through a plane window, straight down or at an angle; clouds without ground don't count; a drawn map is Map"
+  },
+  "thresholds": {
+    "people": 0.3858,
+    "animals": 0.3895,
+    "nature": 0.5188,
+    "landscape": 0.5707,
+    "buildings": 0.4672,
+    "vehicles": 0.3705,
+    "food": 0.2924,
+    "text": 0.6282,
+    "handwriting": 0.4616,
+    "chart": 0.6102,
+    "diagram": 0.5,
+    "table": 0.5683,
+    "map": 0.4185,
+    "logo-icon": 0.5563,
+    "formula": 0.5,
+    "screen": 0.6147,
+    "artwork": 0.5216,
+    "instruments": 0.2603,
+    "furniture": 0.6538,
+    "aerial": 0.5653
+  },
+  "input": {
+    "name": "pixel_values",
+    "shape": [
+      "batch",
+      3,
+      224,
+      224
+    ],
+    "mean": [
+      0.485,
+      0.456,
+      0.406
+    ],
+    "std": [
+      0.229,
+      0.224,
+      0.225
+    ],
+    "preprocessing": "RGB; resize to img_size x img_size (bilinear, antialiased, no crop, aspect not preserved); scale to [0,1]; (x - mean) / std; NCHW float32"
+  },
+  "output": {
+    "name": "logits",
+    "activation": "sigmoid per label; present when >= its threshold"
+  },
+  "backbone": "tf_efficientnetv2_b0.in1k"
+}

model.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:61d5ce2d6cd72cff357b7aba6d0b790d47c4453dcfebcce736c48375d1914bd5
+size 24058507