---
license: apache-2.0
library_name: onnx
pipeline_tag: image-classification
tags:
  - multi-label
  - image-attributes
  - onnx
  - mobilenetv4
---

# WhatDoWeSee-MobileNetV4

Tags **what is visible in an image** with any number of 20 attributes, such as
people, text, chart, diagram, table, map or aerial view. A small, fast MobileNetV4 model (34 MB ONNX, about 18 ms per image on one CPU thread)
for sorting and describing large image collections.

This is **v0.3.0** (2026-10-15): trained on 38,386 images where three Albert vision-language models agree (same data as WhatDoWeSee-EfficientNet-B0 v0.3.0).
Try it in your browser: [WhatDoWeSee Space](https://huggingface.co/spaces/StephanMeijer/WhatDoWeSee).

## Attributes

An image gets every attribute that is clearly recognisable, also when blurry or in the background;
tiny or unrecognisable details don't count. Depicted things count (a drawn dog is Animals).

- **People** (`people`): one or more humans, or clearly human faces, hands or bodies
- **Animals** (`animals`): animals, birds, fish or insects
- **Plants** (`nature`): living green things: trees, plants, flowers, crops; not grass or a lawn as ground, not sea, sky or rocks (those are Landscape)
- **Landscape** (`landscape`): an outdoor view of land, sea, sky, mountains, rocks, rivers or a city as a whole
- **Buildings / structures** (`buildings`): buildings, interiors where the room or its architecture is clearly visible, and other built structures: bridges, towers, dams, monuments, ruins
- **Vehicles** (`vehicles`): cars, bikes, trains, ships, planes or other vehicles
- **Food** (`food`): food or drinks
- **Text** (`text`): running text, sentences, lists or a page that make up a substantial part of the image; labels, captions, titles and signs don't count
- **Handwriting** (`handwriting`): handwritten text, notes or signatures
- **Chart** (`chart`): a data plot: values on axes or as proportions (bar, line, pie, scatter, area chart or similar)
- **Diagram** (`diagram`): structure or relationships without measured values: a flowchart, schematic, process or relationship diagram; also labelled drawings that show how something is built (anatomy, parts, cross-sections)
- **Table** (`table`): text or numbers laid out in rows and columns, with or without grid lines; not the furniture
- **Map** (`map`): a geographic map, floor plan or route map
- **Logo / icon** (`logo-icon`): a logo, icon, emblem, badge or pictogram
- **Formula** (`formula`): mathematical, chemical or other notation
- **Screen / UI** (`screen`): an app, website or device interface
- **Artwork** (`artwork`): made as art: a painting, illustration, cartoon or sculpture, as the image itself or shown in it; not drawings made to explain (those are Diagram)
- **Instruments** (`instruments`): musical instruments: guitar, piano, violin, drums, wind instruments and the like
- **Furniture** (`furniture`): chairs, tables, beds, sofas, desks, cabinets, shelves and other furniture (a data table is Table)
- **Aerial** (`aerial`): the ground seen from high above: aerial or satellite photos, drone shots, bird's-eye views, also through a plane window, straight down or at an angle; clouds without ground don't count; a drawn map is Map

## Usage

Download a fixed version (recommended), so a later release never changes your results:

```python
from huggingface_hub import hf_hub_download

model_path = hf_hub_download("StephanMeijer/WhatDoWeSee-MobileNetV4", "model.onnx", revision="v0.3.0")
config_path = hf_hub_download("StephanMeijer/WhatDoWeSee-MobileNetV4", "config.json", revision="v0.3.0")
```

Then:

```python
import json
import numpy as np
import onnxruntime as ort
from PIL import Image

config = json.load(open(config_path))
session = ort.InferenceSession(model_path)
size = config["input"]["shape"][-1]
mean = np.array(config["input"]["mean"], dtype=np.float32)
std = np.array(config["input"]["std"], dtype=np.float32)

image = Image.open("photo.jpg").convert("RGB").resize((size, size), Image.BILINEAR)
x = ((np.asarray(image, dtype=np.float32) / 255 - mean) / std).transpose(2, 0, 1)[None]
probabilities = 1 / (1 + np.exp(-session.run(None, {"pixel_values": x})[0][0]))
present = [label for label, p in zip(config["labels"], probabilities)
           if p >= config["thresholds"][label]]
```

Preprocessing: RGB, resized to 256×256 (antialiased bilinear, no crop, aspect ratio not kept),
scaled to [0, 1], normalised with the ImageNet mean and standard deviation, NCHW float32. Each label has
its own threshold in `config.json`, tuned on a validation split.

## Evaluation

On 39 held-out hand-labelled images: mean average precision **0.90**,
macro F1 **0.83** at the tuned thresholds. The set is small: per-label scores
are indicative only.

| label | name | positives | AP | F1 |
|---|---|---:|---:|---:|
| `people` | People | 3 | 1.00 | 0.86 |
| `animals` | Animals | 4 | 1.00 | 1.00 |
| `nature` | Plants | 0 | - | 0.00 |
| `landscape` | Landscape | 1 | 1.00 | 0.67 |
| `buildings` | Buildings / structures | 0 | - | 0.00 |
| `vehicles` | Vehicles | 1 | 1.00 | 1.00 |
| `food` | Food | 1 | 1.00 | 1.00 |
| `text` | Text | 17 | 1.00 | 0.87 |
| `handwriting` | Handwriting | 5 | 0.94 | 0.83 |
| `chart` | Chart | 5 | 1.00 | 0.89 |
| `diagram` | Diagram | 3 | 1.00 | 0.80 |
| `table` | Table | 8 | 0.94 | 0.86 |
| `map` | Map | 2 | 1.00 | 1.00 |
| `logo-icon` | Logo / icon | 3 | 0.51 | 0.44 |
| `formula` | Formula | 0 | - | 0.00 |
| `screen` | Screen / UI | 2 | 0.83 | 0.67 |
| `artwork` | Artwork | 2 | 0.35 | 0.50 |
| `instruments` | Instruments | 0 | - | 0.00 |
| `furniture` | Furniture | 0 | - | 0.00 |
| `aerial` | Aerial | 1 | 1.00 | 1.00 |

### How it compares

Against v0.2 on the same 39 independent hand-labelled images: better yes/no decisions (macro
F1 0.83 against 0.73 at each version's thresholds) and better ranking (mean AP 0.91 against
0.79). Clearly better on People (AP 1.00 against 0.67), Handwriting (0.94 / 0.69), Table
(0.94 / 0.58), Diagram (1.00 / 0.71) and Landscape. Weaker on Logo / icon (0.51 against
0.92). Thresholds are now tuned on 8,104 validation images instead of 95. On an earlier set
of 95 hand-labelled images it was the best of all v0.3 candidates (mean AP 0.86). Plants,
Buildings, Formula, Instruments and Furniture have no test image in the 39; per-label scores
rest on 1 to 17 images: indicative only.

## Training

- Backbone: `mobilenetv4_conv_medium.e500_r256_in1k` (ImageNet-pretrained, via timm), with a new
  20-way sigmoid head; 30 epochs, AdamW, binary cross-entropy per label.
- Data: Three vision-language models on the Albert API (Gemma 4 31B, Ministral 3 8B, Mistral Small 3.2 24B) answered the attribute question for about 81,000 images; an attribute counts only when all three picked it. Each image counts under its rarest attribute, at most 2,000 per attribute were kept for training (38,386 images); the 645 hand-labelled images are in the pool with their teacher labels like any other. Thresholds are tuned on the 8,104 teacher-labelled validation images. Test: 39 hand-labelled images that neither this version nor v0.2 trained on or tuned thresholds on. Same weights and thresholds as v0.3.0-rc.3.
- The images come from public collections: photo datasets, document and form scans, charts,
  diagrams, screenshots, maps, artwork and icons.

## Limitations

- Small text-related details (a small table inside a document, a signature) are harder than
  whole-image subjects.
- The training images come from many public collections, some under research-only or
  non-commercial terms. Check whether that matters for your use.

## Licence

The model weights and `config.json` are released under the Apache License 2.0, like the
MobileNetV4 backbone they start from. The licence covers the model, not the images it was trained on.

## Versions

Each version on hand-labelled test images it never trained on (mean AP and macro F1). The test sets
differ between some versions, so compare versions through the notes of each version:

| version | date | test images | mean AP | macro F1 | |
|---|---|---:|---:|---:|---|
| **v0.3.0** | 2026-10-15 | 39 | 0.90 | 0.83 | trained on 38,386 images where three Albert vision-language models agree (same data as WhatDoWeSee-EfficientNet-B0 v0.3.0) |
| **v0.3.0-rc.3** | 2026-10-15 | 95 | 0.86 | 0.71 | release candidate, trained on 38,386 images where all three Albert vision-language models agree (same data as WhatDoWeSee-EfficientNet-B0 v0.3.0-rc.3) |
| **v0.3.0-rc.2** | 2026-10-15 | 38 | 0.81 | 0.74 | release candidate, v0.3.0-rc.1 with stable thresholds (same data as WhatDoWeSee-EfficientNet-B0 v0.3.0-rc.2) |
| **v0.3.0-rc.1** | 2026-10-15 | 38 | 0.81 | 0.76 | release candidate (same data as WhatDoWeSee-EfficientNet-B0 v0.3.0-rc.1), trained on 38,710 images labelled by three Albert vision-language models |
| **v0.2** | 2026-10-14 | 95 | 0.82 | 0.74 | trained on 20,551 images labelled by a SigLIP 2 probe plus the 444 hand labels |