v0.3.0: trained on unanimous Albert teacher labels

README.md

@@ -15,7 +15,7 @@ Tags **what is visible in an image** with any number of 20 attributes, such as
 people, text, chart, diagram, table, map or aerial view. A small, fast MobileNetV4 model (34 MB ONNX, about 18 ms per image on one CPU thread)
 for sorting and describing large image collections.
 
-This is **v0.2** (2026-10-14): trained on 20,551 images labelled by a SigLIP 2 probe plus the 444 hand labels.
+This is **v0.3.0** (2026-10-15): trained on 38,386 images where three Albert vision-language models agree (same data as WhatDoWeSee-EfficientNet-B0 v0.3.0).
 Try it in your browser: [WhatDoWeSee Space](https://huggingface.co/spaces/StephanMeijer/WhatDoWeSee).
 
 ## Attributes
@@ -51,8 +51,8 @@ Download a fixed version (recommended), so a later release never changes your re
 ```python
 from huggingface_hub import hf_hub_download
 
-model_path = hf_hub_download("StephanMeijer/WhatDoWeSee-MobileNetV4", "model.onnx", revision="v0.2")
-config_path = hf_hub_download("StephanMeijer/WhatDoWeSee-MobileNetV4", "config.json", revision="v0.2")
+model_path = hf_hub_download("StephanMeijer/WhatDoWeSee-MobileNetV4", "model.onnx", revision="v0.3.0")
+config_path = hf_hub_download("StephanMeijer/WhatDoWeSee-MobileNetV4", "config.json", revision="v0.3.0")
 ```
 
 Then:
@@ -82,49 +82,49 @@ its own threshold in `config.json`, tuned on a validation split.
 
 ## Evaluation
 
-On 95 held-out hand-labelled images: mean average precision **0.82**,
-macro F1 **0.74** at the tuned thresholds. Every version is scored on the same
-test images. The set is small: per-label scores are indicative only.
+On 39 held-out hand-labelled images: mean average precision **0.90**,
+macro F1 **0.83** at the tuned thresholds. The set is small: per-label scores
+are indicative only.
 
 | label | name | positives | AP | F1 |
 |---|---|---:|---:|---:|
-| `people` | People | 11 | 0.78 | 0.74 |
-| `animals` | Animals | 5 | 0.82 | 0.89 |
-| `nature` | Plants | 4 | 0.26 | 0.00 |
-| `landscape` | Landscape | 3 | 0.59 | 0.50 |
-| `buildings` | Buildings / structures | 9 | 0.58 | 0.56 |
-| `vehicles` | Vehicles | 4 | 1.00 | 1.00 |
-| `food` | Food | 5 | 1.00 | 1.00 |
-| `text` | Text | 22 | 0.99 | 0.96 |
-| `handwriting` | Handwriting | 5 | 1.00 | 0.75 |
-| `chart` | Chart | 7 | 0.98 | 0.92 |
-| `diagram` | Diagram | 8 | 0.82 | 0.80 |
-| `table` | Table | 6 | 0.78 | 0.53 |
-| `map` | Map | 5 | 1.00 | 0.89 |
-| `logo-icon` | Logo / icon | 6 | 1.00 | 0.86 |
-| `formula` | Formula | 6 | 0.93 | 0.91 |
-| `screen` | Screen / UI | 6 | 0.82 | 0.67 |
-| `artwork` | Artwork | 7 | 0.76 | 0.73 |
-| `instruments` | Instruments | 7 | 0.76 | 0.67 |
-| `furniture` | Furniture | 5 | 0.90 | 0.75 |
-| `aerial` | Aerial | 11 | 0.67 | 0.70 |
+| `people` | People | 3 | 1.00 | 0.86 |
+| `animals` | Animals | 4 | 1.00 | 1.00 |
+| `nature` | Plants | 0 | - | 0.00 |
+| `landscape` | Landscape | 1 | 1.00 | 0.67 |
+| `buildings` | Buildings / structures | 0 | - | 0.00 |
+| `vehicles` | Vehicles | 1 | 1.00 | 1.00 |
+| `food` | Food | 1 | 1.00 | 1.00 |
+| `text` | Text | 17 | 1.00 | 0.87 |
+| `handwriting` | Handwriting | 5 | 0.94 | 0.83 |
+| `chart` | Chart | 5 | 1.00 | 0.89 |
+| `diagram` | Diagram | 3 | 1.00 | 0.80 |
+| `table` | Table | 8 | 0.94 | 0.86 |
+| `map` | Map | 2 | 1.00 | 1.00 |
+| `logo-icon` | Logo / icon | 3 | 0.51 | 0.44 |
+| `formula` | Formula | 0 | - | 0.00 |
+| `screen` | Screen / UI | 2 | 0.83 | 0.67 |
+| `artwork` | Artwork | 2 | 0.35 | 0.50 |
+| `instruments` | Instruments | 0 | - | 0.00 |
+| `furniture` | Furniture | 0 | - | 0.00 |
+| `aerial` | Aerial | 1 | 1.00 | 1.00 |
 
 ### How it compares
 
-Against WhatDoWeSee-EfficientNet-B0 v0.2 (same data, same test images): best yes/no decisions
-(macro F1 0.74 at the tuned thresholds, 0.75 at 0.5, against 0.68 and 0.71), mean AP 0.82
-against 0.84. Better on Landscape (AP 0.59 against 0.42), Food (1.00 / 0.83), Vehicles
-(1.00 / 0.88), Logo / icon (1.00 / 0.87), Buildings / structures (0.58 / 0.50), Map and
-Handwriting (1.00). Weaker on Plants (0.26 against 0.69), Aerial (0.67 / 0.86), Diagram
-(0.82 / 1.00), Table (0.78 / 0.90) and People (0.78 / 0.87). Twice as large and about twice as
-slow (34 MB, about 18 ms per image on one CPU thread). Per-label scores rest on 3 to 22 test
-images: indicative only.
+Against v0.2 on the same 39 independent hand-labelled images: better yes/no decisions (macro
+F1 0.83 against 0.73 at each version's thresholds) and better ranking (mean AP 0.91 against
+0.79). Clearly better on People (AP 1.00 against 0.67), Handwriting (0.94 / 0.69), Table
+(0.94 / 0.58), Diagram (1.00 / 0.71) and Landscape. Weaker on Logo / icon (0.51 against
+0.92). Thresholds are now tuned on 8,104 validation images instead of 95. On an earlier set
+of 95 hand-labelled images it was the best of all v0.3 candidates (mean AP 0.86). Plants,
+Buildings, Formula, Instruments and Furniture have no test image in the 39; per-label scores
+rest on 1 to 17 images: indicative only.
 
 ## Training
 
 - Backbone: `mobilenetv4_conv_medium.e500_r256_in1k` (ImageNet-pretrained, via timm), with a new
   20-way sigmoid head; 30 epochs, AdamW, binary cross-entropy per label.
-- Data: The same data as WhatDoWeSee-EfficientNet-B0 v0.2: a linear probe on SigLIP 2 (ViT-B/16) image features, trained on the 444 hand-labelled images, labelled about 44,000 images at per-attribute thresholds tuned on the hand-labelled validation images; at most 1,000 of those per rarest attribute were kept (20,107), plus the 444 hand labels. Validation and test are 95 + 95 hand-labelled images, never trained on. Backbone mobilenetv4_conv_medium at 256 px.
+- Data: Three vision-language models on the Albert API (Gemma 4 31B, Ministral 3 8B, Mistral Small 3.2 24B) answered the attribute question for about 81,000 images; an attribute counts only when all three picked it. Each image counts under its rarest attribute, at most 2,000 per attribute were kept for training (38,386 images); the 645 hand-labelled images are in the pool with their teacher labels like any other. Thresholds are tuned on the 8,104 teacher-labelled validation images. Test: 39 hand-labelled images that neither this version nor v0.2 trained on or tuned thresholds on. Same weights and thresholds as v0.3.0-rc.3.
 - The images come from public collections: photo datasets, document and form scans, charts,
   diagrams, screenshots, maps, artwork and icons.
 
@@ -142,8 +142,13 @@ MobileNetV4 backbone they start from. The licence covers the model, not the imag
 
 ## Versions
 
-Same test images for every version (mean AP and macro F1):
+Each version on hand-labelled test images it never trained on (mean AP and macro F1). The test sets
+differ between some versions, so compare versions through the notes of each version:
 
-| version | date | mean AP | macro F1 | |
-|---|---|---:|---:|---|
-| **v0.2** | 2026-10-14 | 0.82 | 0.74 | trained on 20,551 images labelled by a SigLIP 2 probe plus the 444 hand labels |
+| version | date | test images | mean AP | macro F1 | |
+|---|---|---:|---:|---:|---|
+| **v0.3.0** | 2026-10-15 | 39 | 0.90 | 0.83 | trained on 38,386 images where three Albert vision-language models agree (same data as WhatDoWeSee-EfficientNet-B0 v0.3.0) |
+| **v0.3.0-rc.3** | 2026-10-15 | 95 | 0.86 | 0.71 | release candidate, trained on 38,386 images where all three Albert vision-language models agree (same data as WhatDoWeSee-EfficientNet-B0 v0.3.0-rc.3) |
+| **v0.3.0-rc.2** | 2026-10-15 | 38 | 0.81 | 0.74 | release candidate, v0.3.0-rc.1 with stable thresholds (same data as WhatDoWeSee-EfficientNet-B0 v0.3.0-rc.2) |
+| **v0.3.0-rc.1** | 2026-10-15 | 38 | 0.81 | 0.76 | release candidate (same data as WhatDoWeSee-EfficientNet-B0 v0.3.0-rc.1), trained on 38,710 images labelled by three Albert vision-language models |
+| **v0.2** | 2026-10-14 | 95 | 0.82 | 0.74 | trained on 20,551 images labelled by a SigLIP 2 probe plus the 444 hand labels |

config.json

@@ -1,5 +1,5 @@
 {
-  "version": "v0.2",
+  "version": "v0.3.0",
   "task": "multi-label image attributes",
   "group": "content",
   "labels": [
@@ -69,26 +69,26 @@
     "aerial": "the ground seen from high above: aerial or satellite photos, drone shots, bird's-eye views, also through a plane window, straight down or at an angle; clouds without ground don't count; a drawn map is Map"
   },
   "thresholds": {
-    "people": 0.4117,
-    "animals": 0.5,
-    "nature": 0.9781,
-    "landscape": 0.9545,
-    "buildings": 0.5,
-    "vehicles": 0.05,
-    "food": 0.0368,
-    "text": 0.039,
-    "handwriting": 0.1818,
+    "people": 0.4865,
+    "animals": 0.3594,
+    "nature": 0.5881,
+    "landscape": 0.5811,
+    "buildings": 0.7084,
+    "vehicles": 0.4425,
+    "food": 0.7232,
+    "text": 0.7025,
+    "handwriting": 0.7569,
     "chart": 0.5,
-    "diagram": 0.5,
-    "table": 0.5,
+    "diagram": 0.8988,
+    "table": 0.5682,
     "map": 0.5,
-    "logo-icon": 0.5,
-    "formula": 0.5,
-    "screen": 0.5,
-    "artwork": 0.5,
-    "instruments": 0.5,
-    "furniture": 0.998,
-    "aerial": 0.9233
+    "logo-icon": 0.6388,
+    "formula": 0.8698,
+    "screen": 0.789,
+    "artwork": 0.1739,
+    "instruments": 0.9685,
+    "furniture": 0.8187,
+    "aerial": 0.2008
   },
   "input": {
     "name": "pixel_values",

model.onnx

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:3cd6b6cdff0d289c73e6b60f27e080ad6142d0e45841c19a8976d9db5871e227
+oid sha256:40e58d43ac69b88788e5aa511a31193d15af529ab29e53f4702ce5f8d766607f
 size 34102699