| --- |
| license: other |
| license_name: per-model |
| license_link: LICENSE.md |
| library_name: onnx |
| pipeline_tag: image-feature-extraction |
| tags: |
| - onnx |
| - onnxruntime-web |
| - webgpu |
| - timm |
| - explainability |
| --- |
| |
| # Demo models |
|
|
| Vision models converted to ONNX for the in-browser demos on [karay.me](https://karay.me): |
| the [LaFAM demo](https://karay.me/examples/lafam/) and the |
| [region similarity demo](https://karay.me/examples/similarity/). The demos run them on the visitor's |
| device with [ONNX Runtime Web](https://onnxruntime.ai/) (WebGPU, or WebAssembly on the CPU); no image |
| leaves the browser. |
|
|
| Each file is the largest model of its family that stays under 100 MB, except where a specific model is the |
| point: CLIP RN50x4 (the model of the typographic attacks of Goh et al., 2021) and the pathology models |
| Hibou-B and QuiltNet-B-32. |
|
|
| ## Interface |
|
|
| Every model takes one input and returns the last feature map before pooling: |
|
|
| | | Shape | Meaning | |
| |---|---|---| |
| | input `image` | (1, 3, S, S) | RGB in [0, 1]; S = 224 (288 for CLIP RN50x4); the model's own normalization is part of the graph | |
| | output `features` | (1, C, H, W) | the last feature map; for ViTs, the patch tokens after the final norm, as a grid | |
| | output `cell_logits` | (1, K, H, W) | classifiers only: the classification head applied to every cell (K = 1000 for ImageNet, 5 tissue classes, 2 for the tumour detector) | |
| | output `logits` | (1, K) | only CLIP RN50x4 and QuiltNet-B-32: the model's own zero-shot logits (attention pooling or class token), which are not the mean of the cells | |
|
|
| For the classifiers that pool their features (ResNet, ConvNeXt, Swin, MaxViT), the mean of |
| `cell_logits` over the cells equals the model's logits, so the per-cell readout is exact |
| ([Karjauv, 2026](https://arxiv.org/abs/2606.14555)). For the two CLIP models with a zero-shot head, `cell_logits` |
| classifies each cell on its own: the cell's value embedding through the attention pooling's output |
| projection (RN50x4, as in MaskCLIP, [Zhou et al., 2022](https://arxiv.org/abs/2112.01071)), or the patch |
| token through the final projection (QuiltNet). Weights are stored as float16 and cast to float32 |
| in the graph; the files match the PyTorch models to a correlation of at least 0.9999 on the demo images. |
|
|
| ## Files |
|
|
| | File | Model | Source checkpoint (timm) | Grid | License | |
| |---|---|---|---|---| |
| | `resnet50.onnx` | ResNet-50, ImageNet classifier | [`resnet50.tv2_in1k`](https://huggingface.co/timm/resnet50.tv2_in1k) | 7×7 | BSD-3-Clause | |
| | `convnext_small.onnx` | ConvNeXt-S, ImageNet classifier | [`convnext_small.fb_in1k`](https://huggingface.co/timm/convnext_small.fb_in1k) | 7×7 | Apache-2.0 | |
| | `swin_small.onnx` | Swin-S, ImageNet classifier | [`swin_small_patch4_window7_224.ms_in1k`](https://huggingface.co/timm/swin_small_patch4_window7_224.ms_in1k) | 7×7 | MIT | |
| | `maxvit_tiny.onnx` | MaxViT-T, ImageNet classifier | [`maxvit_tiny_tf_224.in1k`](https://huggingface.co/timm/maxvit_tiny_tf_224.in1k) | 7×7 | Apache-2.0 | |
| | `dino_resnet50.onnx` | DINO ResNet-50, self-supervised | [DINO](https://github.com/facebookresearch/dino) `dino_resnet50_pretrain.pth` (torchvision) | 7×7 | Apache-2.0 | |
| | `dino_vit_small.onnx` | DINO ViT-S/16, self-supervised | [`vit_small_patch16_224.dino`](https://huggingface.co/timm/vit_small_patch16_224.dino) | 14×14 | Apache-2.0 | |
| | `dinov2_vit_small.onnx` | DINOv2 ViT-S/14, self-supervised | [`vit_small_patch14_dinov2.lvd142m`](https://huggingface.co/timm/vit_small_patch14_dinov2.lvd142m) | 16×16 | Apache-2.0 | |
| | `dinov2_reg_vit_small.onnx` | DINOv2 ViT-S/14 with registers, self-supervised | [`vit_small_patch14_reg4_dinov2.lvd142m`](https://huggingface.co/timm/vit_small_patch14_reg4_dinov2.lvd142m) | 16×16 | Apache-2.0 | |
| | `dinov3_vit_small_plus.onnx` | DINOv3 ViT-S+/16, self-supervised | [`vit_small_plus_patch16_dinov3.lvd1689m`](https://huggingface.co/timm/vit_small_plus_patch16_dinov3.lvd1689m) | 14×14 | [DINOv3 License](DINOV3_LICENSE.md) | |
| | `dinov3_convnext_small.onnx` | DINOv3 ConvNeXt-S, self-supervised | [`convnext_small.dinov3_lvd1689m`](https://huggingface.co/timm/convnext_small.dinov3_lvd1689m) | 7×7 | [DINOv3 License](DINOV3_LICENSE.md) | |
| | `fcmae_convnextv2_tiny.onnx` | ConvNeXt V2-T, masked autoencoder (FCMAE) | [`convnextv2_tiny.fcmae`](https://huggingface.co/timm/convnextv2_tiny.fcmae) | 7×7 | CC-BY-NC-4.0 (non-commercial use only) | |
| | `clip_resnet101.onnx` | CLIP ResNet-101 (OpenAI), image encoder trunk | [`resnet101_clip_gap.openai`](https://huggingface.co/timm/resnet101_clip_gap.openai) | 7×7 | MIT | |
| | `tinyclip_vit_medium.onnx` | TinyCLIP ViT-M/16, image encoder | [`vit_medium_patch16_clip_224.tinyclip_yfcc15m`](https://huggingface.co/timm/vit_medium_patch16_clip_224.tinyclip_yfcc15m) | 14×14 | MIT | |
| | `clip_rn50x4.onnx` | CLIP RN50x4 (OpenAI) with its zero-shot ImageNet classifier (80 prompts) | open_clip `RN50x4`, `openai` | 9×9 | MIT | |
| | `hibou_b.onnx` | Hibou-B (pathology, DINOv2 ViT-B/14 with registers) + a linear tissue head (tumour, stroma, lymphocytes, necrosis, fat) | [`histai/hibou-b`](https://huggingface.co/histai/hibou-b) via `hibou_port.py`; head `hibou_b_tissue_head.pt` | 16×16 | Apache-2.0 | |
| | `quiltnet_b32.onnx` | QuiltNet-B-32 (pathology CLIP) with a zero-shot head over the same tissue classes | open_clip [`wisdomik/QuiltNet-B-32`](https://huggingface.co/wisdomik/QuiltNet-B-32) | 7×7 | MIT | |
| | `tumour_detector.onnx` | breast tumour detector, ResNet-34 (other, tumour) | [`kaczmarj/breast-tumor-resnet34.tcga-brca`](https://huggingface.co/kaczmarj/breast-tumor-resnet34.tcga-brca) (torchvision) | 7×7 | CC-BY-4.0 | |
| | `kaiko_vit_small.onnx` | Kaiko ViT-S/16 (pathology, self-supervised) | torch.hub [`kaiko-ai/towards_large_pathology_fms`](https://github.com/kaiko-ai/towards_large_pathology_fms) `vits16` | 14×14 | Kaiko Non-Commercial | |
| | `ctranspath.onnx` | CTransPath (pathology, Swin-T with a convolutional stem) | [`1aurent/swin_tiny_patch4_window7_224.CTransPath`](https://huggingface.co/1aurent/swin_tiny_patch4_window7_224.CTransPath) | 7×7 | GPL-3.0 | |
|
|
| The pathology models expect H&E tiles at about 0.5 µm per pixel (224 pixels = 112 µm); the tumour detector |
| was trained on 0.25 µm per pixel tiles of 350 pixels resized to 224 (about 0.39 µm per pixel). The tissue |
| head of Hibou-B is logistic regression on the mean patch token, trained with one label per 112 µm tile |
| (the majority tissue of the pathologist's annotation) on 6,692 tiles of 132 BCSS cases, none lobular |
| (`hibou_head.py` in the author's project); as it is linear on the mean, the mean of its `cell_logits` is |
| its prediction. |
|
|
| Each file is a derivative of its source checkpoint and is distributed under that checkpoint's license |
| (see [LICENSE.md](LICENSE.md)). The DINOv3 models are distributed under the DINOv3 License, a copy of |
| which is included; ConvNeXt V2 (FCMAE) and Kaiko ViT-S/16 are for non-commercial use only; CTransPath is |
| GPL-3.0, with its conversion source in this repository. |
|
|
| ## Reproducing |
|
|
| [`export_models.py`](export_models.py) exports every file from its source checkpoint and checks it |
| against PyTorch on a folder of .jpg images (any will do): |
|
|
| SAMPLES=path/to/images uv run --no-project --with torch --with torchvision --with timm --with onnx --with onnxruntime \ |
| --with pillow --with open_clip_torch --with safetensors --with huggingface_hub python export_models.py NAME |
| |
