| --- |
| license: apache-2.0 |
| library_name: onnx |
| pipeline_tag: image-classification |
| tags: |
| - multi-label |
| - image-attributes |
| - onnx |
| - mobilenetv4 |
| --- |
| |
| # WhatDoWeSee-MobileNetV4 |
|
|
| Tags **what is visible in an image** with any number of 20 attributes, such as |
| people, text, chart, diagram, table, map or aerial view. A small, fast MobileNetV4 model (34 MB ONNX, about 18 ms per image on one CPU thread) |
| for sorting and describing large image collections. |
|
|
| This is **v0.3.0** (2026-10-15): trained on 38,386 images where three Albert vision-language models agree (same data as WhatDoWeSee-EfficientNet-B0 v0.3.0). |
| Try it in your browser: [WhatDoWeSee Space](https://huggingface.co/spaces/StephanMeijer/WhatDoWeSee). |
|
|
| ## Attributes |
|
|
| An image gets every attribute that is clearly recognisable, also when blurry or in the background; |
| tiny or unrecognisable details don't count. Depicted things count (a drawn dog is Animals). |
|
|
| - **People** (`people`): one or more humans, or clearly human faces, hands or bodies |
| - **Animals** (`animals`): animals, birds, fish or insects |
| - **Plants** (`nature`): living green things: trees, plants, flowers, crops; not grass or a lawn as ground, not sea, sky or rocks (those are Landscape) |
| - **Landscape** (`landscape`): an outdoor view of land, sea, sky, mountains, rocks, rivers or a city as a whole |
| - **Buildings / structures** (`buildings`): buildings, interiors where the room or its architecture is clearly visible, and other built structures: bridges, towers, dams, monuments, ruins |
| - **Vehicles** (`vehicles`): cars, bikes, trains, ships, planes or other vehicles |
| - **Food** (`food`): food or drinks |
| - **Text** (`text`): running text, sentences, lists or a page that make up a substantial part of the image; labels, captions, titles and signs don't count |
| - **Handwriting** (`handwriting`): handwritten text, notes or signatures |
| - **Chart** (`chart`): a data plot: values on axes or as proportions (bar, line, pie, scatter, area chart or similar) |
| - **Diagram** (`diagram`): structure or relationships without measured values: a flowchart, schematic, process or relationship diagram; also labelled drawings that show how something is built (anatomy, parts, cross-sections) |
| - **Table** (`table`): text or numbers laid out in rows and columns, with or without grid lines; not the furniture |
| - **Map** (`map`): a geographic map, floor plan or route map |
| - **Logo / icon** (`logo-icon`): a logo, icon, emblem, badge or pictogram |
| - **Formula** (`formula`): mathematical, chemical or other notation |
| - **Screen / UI** (`screen`): an app, website or device interface |
| - **Artwork** (`artwork`): made as art: a painting, illustration, cartoon or sculpture, as the image itself or shown in it; not drawings made to explain (those are Diagram) |
| - **Instruments** (`instruments`): musical instruments: guitar, piano, violin, drums, wind instruments and the like |
| - **Furniture** (`furniture`): chairs, tables, beds, sofas, desks, cabinets, shelves and other furniture (a data table is Table) |
| - **Aerial** (`aerial`): the ground seen from high above: aerial or satellite photos, drone shots, bird's-eye views, also through a plane window, straight down or at an angle; clouds without ground don't count; a drawn map is Map |
|
|
| ## Usage |
|
|
| Download a fixed version (recommended), so a later release never changes your results: |
|
|
| ```python |
| from huggingface_hub import hf_hub_download |
| |
| model_path = hf_hub_download("StephanMeijer/WhatDoWeSee-MobileNetV4", "model.onnx", revision="v0.3.0") |
| config_path = hf_hub_download("StephanMeijer/WhatDoWeSee-MobileNetV4", "config.json", revision="v0.3.0") |
| ``` |
|
|
| Then: |
|
|
| ```python |
| import json |
| import numpy as np |
| import onnxruntime as ort |
| from PIL import Image |
| |
| config = json.load(open(config_path)) |
| session = ort.InferenceSession(model_path) |
| size = config["input"]["shape"][-1] |
| mean = np.array(config["input"]["mean"], dtype=np.float32) |
| std = np.array(config["input"]["std"], dtype=np.float32) |
| |
| image = Image.open("photo.jpg").convert("RGB").resize((size, size), Image.BILINEAR) |
| x = ((np.asarray(image, dtype=np.float32) / 255 - mean) / std).transpose(2, 0, 1)[None] |
| probabilities = 1 / (1 + np.exp(-session.run(None, {"pixel_values": x})[0][0])) |
| present = [label for label, p in zip(config["labels"], probabilities) |
| if p >= config["thresholds"][label]] |
| ``` |
|
|
| Preprocessing: RGB, resized to 256×256 (antialiased bilinear, no crop, aspect ratio not kept), |
| scaled to [0, 1], normalised with the ImageNet mean and standard deviation, NCHW float32. Each label has |
| its own threshold in `config.json`, tuned on a validation split. |
|
|
| ## Evaluation |
|
|
| On 39 held-out hand-labelled images: mean average precision **0.90**, |
| macro F1 **0.83** at the tuned thresholds. The set is small: per-label scores |
| are indicative only. |
|
|
| | label | name | positives | AP | F1 | |
| |---|---|---:|---:|---:| |
| | `people` | People | 3 | 1.00 | 0.86 | |
| | `animals` | Animals | 4 | 1.00 | 1.00 | |
| | `nature` | Plants | 0 | - | 0.00 | |
| | `landscape` | Landscape | 1 | 1.00 | 0.67 | |
| | `buildings` | Buildings / structures | 0 | - | 0.00 | |
| | `vehicles` | Vehicles | 1 | 1.00 | 1.00 | |
| | `food` | Food | 1 | 1.00 | 1.00 | |
| | `text` | Text | 17 | 1.00 | 0.87 | |
| | `handwriting` | Handwriting | 5 | 0.94 | 0.83 | |
| | `chart` | Chart | 5 | 1.00 | 0.89 | |
| | `diagram` | Diagram | 3 | 1.00 | 0.80 | |
| | `table` | Table | 8 | 0.94 | 0.86 | |
| | `map` | Map | 2 | 1.00 | 1.00 | |
| | `logo-icon` | Logo / icon | 3 | 0.51 | 0.44 | |
| | `formula` | Formula | 0 | - | 0.00 | |
| | `screen` | Screen / UI | 2 | 0.83 | 0.67 | |
| | `artwork` | Artwork | 2 | 0.35 | 0.50 | |
| | `instruments` | Instruments | 0 | - | 0.00 | |
| | `furniture` | Furniture | 0 | - | 0.00 | |
| | `aerial` | Aerial | 1 | 1.00 | 1.00 | |
|
|
| ### How it compares |
|
|
| Against v0.2 on the same 39 independent hand-labelled images: better yes/no decisions (macro |
| F1 0.83 against 0.73 at each version's thresholds) and better ranking (mean AP 0.91 against |
| 0.79). Clearly better on People (AP 1.00 against 0.67), Handwriting (0.94 / 0.69), Table |
| (0.94 / 0.58), Diagram (1.00 / 0.71) and Landscape. Weaker on Logo / icon (0.51 against |
| 0.92). Thresholds are now tuned on 8,104 validation images instead of 95. On an earlier set |
| of 95 hand-labelled images it was the best of all v0.3 candidates (mean AP 0.86). Plants, |
| Buildings, Formula, Instruments and Furniture have no test image in the 39; per-label scores |
| rest on 1 to 17 images: indicative only. |
|
|
| ## Training |
|
|
| - Backbone: `mobilenetv4_conv_medium.e500_r256_in1k` (ImageNet-pretrained, via timm), with a new |
| 20-way sigmoid head; 30 epochs, AdamW, binary cross-entropy per label. |
| - Data: Three vision-language models on the Albert API (Gemma 4 31B, Ministral 3 8B, Mistral Small 3.2 24B) answered the attribute question for about 81,000 images; an attribute counts only when all three picked it. Each image counts under its rarest attribute, at most 2,000 per attribute were kept for training (38,386 images); the 645 hand-labelled images are in the pool with their teacher labels like any other. Thresholds are tuned on the 8,104 teacher-labelled validation images. Test: 39 hand-labelled images that neither this version nor v0.2 trained on or tuned thresholds on. Same weights and thresholds as v0.3.0-rc.3. |
| - The images come from public collections: photo datasets, document and form scans, charts, |
| diagrams, screenshots, maps, artwork and icons. |
|
|
| ## Limitations |
|
|
| - Small text-related details (a small table inside a document, a signature) are harder than |
| whole-image subjects. |
| - The training images come from many public collections, some under research-only or |
| non-commercial terms. Check whether that matters for your use. |
|
|
| ## Licence |
|
|
| The model weights and `config.json` are released under the Apache License 2.0, like the |
| MobileNetV4 backbone they start from. The licence covers the model, not the images it was trained on. |
|
|
| ## Versions |
|
|
| Each version on hand-labelled test images it never trained on (mean AP and macro F1). The test sets |
| differ between some versions, so compare versions through the notes of each version: |
|
|
| | version | date | test images | mean AP | macro F1 | | |
| |---|---|---:|---:|---:|---| |
| | **v0.3.0** | 2026-10-15 | 39 | 0.90 | 0.83 | trained on 38,386 images where three Albert vision-language models agree (same data as WhatDoWeSee-EfficientNet-B0 v0.3.0) | |
| | **v0.3.0-rc.3** | 2026-10-15 | 95 | 0.86 | 0.71 | release candidate, trained on 38,386 images where all three Albert vision-language models agree (same data as WhatDoWeSee-EfficientNet-B0 v0.3.0-rc.3) | |
| | **v0.3.0-rc.2** | 2026-10-15 | 38 | 0.81 | 0.74 | release candidate, v0.3.0-rc.1 with stable thresholds (same data as WhatDoWeSee-EfficientNet-B0 v0.3.0-rc.2) | |
| | **v0.3.0-rc.1** | 2026-10-15 | 38 | 0.81 | 0.76 | release candidate (same data as WhatDoWeSee-EfficientNet-B0 v0.3.0-rc.1), trained on 38,710 images labelled by three Albert vision-language models | |
| | **v0.2** | 2026-10-14 | 95 | 0.82 | 0.74 | trained on 20,551 images labelled by a SigLIP 2 probe plus the 444 hand labels | |
|
|
