Request access to DermaLit-CLIP (research use only)
This repository is publicly accessible, but you have to accept the conditions to access its files and content.
These weights were trained on copyrighted dermatology images under the German research exception for text and data mining (§60d UrhG). Access is granted for non-commercial scientific research and teaching only. Requests are reviewed by hand. Please use an account that shows your name and institution. The licence text is the file LICENSE.md in this repository: https://huggingface.co/laion/DermaLit-CLIP-ViT-B-16/blob/main/LICENSE.md
Log in or Sign Up to review the conditions and access this model content.
DermaLit-CLIP ViT-B/16
CLIP ViT-B/16 fine-tuned on dermatology image–text pairs from the DermaLit-140k corpus (journal figures with their captions, web atlases, textbooks). The model maps skin images and text to a shared embedding space, for zero-shot classification, retrieval and linear probing in dermatology research.
Research use only. Not for clinical use. The model is not a medical device and was not validated on patients. See LICENSE.md.
Models in this repository
| Path | Training pairs | Steps | Use |
|---|---|---|---|
| repository root (default) | 54,716 pairs of DermaLit-140k, without sources whose terms restrict AI use | 3,000 | recommended |
reported/ours-3k |
69,536 pairs of DermaLit-140k | 3,000 | reproduces the project report and preprint |
reported/union-3k |
139,072 pairs: DermaLit-140k and Derm1M | 3,000 | reproduces the report and preprint; diagnosis model of the synthetic-image quality gate |
reported/ours-1k-seed0, -seed1, -seed2 |
69,536 pairs of DermaLit-140k | 1,000 | the three seeds of the equal-compute comparison with Derm1M |
Why two sets of weights. The checkpoints under reported/ are the ones behind the published numbers. 14,820 of their 69,536 DermaLit training pairs (21%) come from three sources whose terms of use restrict AI use (DermNet NZ, DermIS and via medici). reported/union-3k was also trained on Derm1M, which is gated and licensed CC BY-NC 4.0. We publish these checkpoints so that the reported results can be verified. The default model was retrained with the same recipe without the three restricted sources and without Derm1M. Use the default model unless you need to reproduce a reported number.
The Derm1M-only baselines of the report are not included. They can be retrained with the public code.
Results
Derm1M benchmark protocol: zero-shot accuracy on PAD-UFES-20, HAM10000, Fitzpatrick17k (114 classes, top-1) and Daffodil; linear probe (mean accuracy over the four benchmarks); image–text retrieval on SkinCAP (recall@10). All values in %.
| Model | PAD | HAM | F17K | Daffodil | Zero-shot mean | Linear probe mean | SkinCAP image→text R@10 | SkinCAP text→image R@10 |
|---|---|---|---|---|---|---|---|---|
| default (repository root) | 55.3 | 67.7 | 25.2 | 74.3 | 55.6 | 78.5 | 20.2 | 13.8 |
reported/ours-3k |
57.3 | 69.5 | 28.2 | 82.3 | 59.3 | 78.8 | 22.4 | 13.6 |
reported/union-3k |
56.2 | 70.8 | 26.8 | 84.3 | 59.5 | 78.5 | 17.3 | 13.9 |
reported/ours-1k-seed0 |
51.4 | 62.1 | 21.8 | 78.6 | 53.5 | 78.0 | 20.4 | 12.8 |
reported/ours-1k-seed1 |
52.3 | 65.1 | 22.3 | 80.4 | 55.0 | 78.3 | 19.3 | 12.4 |
reported/ours-1k-seed2 |
51.4 | 44.8 | 22.2 | 78.7 | 49.3 | 78.3 | 20.0 | 13.1 |
| OpenAI CLIP ViT-B/16 (starting point) | 45.8 | 20.6 | 6.6 | 45.8 | 29.7 | 74.9 | 9.1 | 6.0 |
The default model is a single run (seed 0). Against reported/ours-3k it is lower on Daffodil (74.3 vs 82.3) and Fitzpatrick17k (25.2 vs 28.2), about 2 points lower on PAD-UFES-20, HAM10000 and image→text retrieval, and equal on linear probe and text→image retrieval. Part of this gap may be seed noise; part is the 21% smaller training set.
Notes on these numbers:
- Images that are near-duplicates of any benchmark image were removed from all training sets (CLIP cosine ≥ 0.95).
- Fitzpatrick17k was built from two atlases that are also sources of DermaLit-140k (DermaAmin, Atlas Dermatológico). Exact and near-duplicate images were removed, but shared photographic style probably still favours these models on Fitzpatrick17k.
- Zero-shot accuracy on HAM10000 varies strongly between seeds (44.8–65.1% at 1,000 steps). Single-run differences of a few points are not meaningful.
- Fitzpatrick17k zero-shot accuracy by skin-type group (I–II / III–IV / V–VI): default model 24.4 / 26.8 / 24.1;
reported/ours-3k28.3 / 28.4 / 27.0; OpenAI CLIP 6.9 / 6.6 / 5.9. This does not prove fairness, because the dark-skin images of Fitzpatrick17k cover a different mix of diagnoses.
Usage
Access is gated. Request access on this page, then log in with hf auth login.
import open_clip, torch
from PIL import Image
repo = "hf-hub:laion/DermaLit-CLIP-ViT-B-16"
model, _, preprocess = open_clip.create_model_and_transforms(repo)
tokenizer = open_clip.get_tokenizer(repo)
model.eval()
image = preprocess(Image.open("lesion.jpg")).unsqueeze(0)
labels = ["psoriasis", "atopic dermatitis", "basal cell carcinoma", "melanoma"]
text = tokenizer([f"a clinical photograph of {c}" for c in labels])
with torch.no_grad():
i = torch.nn.functional.normalize(model.encode_image(image), dim=-1)
t = torch.nn.functional.normalize(model.encode_text(text), dim=-1)
probs = (100 * i @ t.T).softmax(dim=-1)
print(dict(zip(labels, probs[0].tolist())))
A reported checkpoint:
from huggingface_hub import hf_hub_download
import open_clip
path = hf_hub_download("laion/DermaLit-CLIP-ViT-B-16", "reported/ours-3k/open_clip_model.safetensors")
model, _, preprocess = open_clip.create_model_and_transforms("ViT-B-16-quickgelu", pretrained=path)
tokenizer = open_clip.get_tokenizer("ViT-B-16-quickgelu")
Training data
All training pairs are dermatology images (clinical photographs, dermoscopy or histopathology, selected by an image-type classifier) from the training split of DermaLit-140k, one image per near-duplicate cluster, with an original caption or a caption built from the source's structured fields.
Default model, 54,716 pairs:
- Journals (25,188 pairs): JAAD, Die Dermatologie, JDDG, American Journal of Clinical Dermatology, JAMA Dermatology, Journal of Investigative Dermatology, Dermatology and Therapy.
- Web atlases (27,951 pairs): eleven atlases whose terms do not restrict AI use.
- Textbooks (1,577 pairs): two German dermatology textbooks.
- Languages: English 46,134; German 7,558; French 770; Spanish 254.
Most of this material is under copyright. It was used under the German exception for text and data mining for scientific research (§60d UrhG). The training data is not part of this repository. The freely licensed part of the corpus and metadata for the rest are at https://huggingface.co/datasets/laion/DermaLit-140k.
Training recipe
OpenAI CLIP ViT-B/16 (open_clip ViT-B-16-quickgelu, openai weights) as starting point; contrastive loss; batch size 128; AdamW (learning rate 1e-5, weight decay 0.1, betas 0.9 / 0.98); cosine schedule with 5% warm-up; random resized crop (scale 0.4–1); bf16 autocast; seed 0 unless stated. Code: https://github.com/LAION-AI/derm-datasets (scripts/61_train_clip.py, evaluation scripts/60_eval_clip.py; training pairs from scripts/63_make_pairs.py, for the default model with the sources DermNet NZ, DermIS and via medici removed). The evaluation output of every checkpoint and of the OpenAI starting point is in eval/.
Limitations
- Skin type. Only about 6% of the clinical photographs in the corpus are estimated as Fitzpatrick V–VI (11% after bias correction). Thirteen frequent diagnoses have no such image. Expect weaker and less tested behaviour on dark skin.
- Disease mix. The data reflects what dermatologists photograph and publish. Melanoma and other tumours are strongly over-represented; common conditions such as scabies, fungal infections and atopic dermatitis are under-represented.
- Small fine-tuning budget. The models were trained for 1,000 or 3,000 steps for a dataset comparison at equal compute. They are not tuned for best performance.
- Text. About 43% of the captions of the default model are templates built from structured atlas fields (diagnosis, body site), not free text.
- Sensitive content. The model was trained on medical photographs, including intimate body sites.
- Memorisation. Like any fine-tuned model it may encode features of individual training images. Attempts to reconstruct training images are not permitted by the licence.
Licence
DermaLit-CLIP Research Licence, see LICENSE.md: non-commercial scientific research and teaching only, no clinical use, derivatives under the same licence. The original OpenAI CLIP weights are under the MIT licence.
Removal requests
If you hold rights in training material and object to this release, open a discussion on this repository or contact the LAION organisation. We will review the request and can withdraw the weights.
Funding and citation
Built in the project "The future of medical datasets: large-scale analysis and training of medical multimodal models through an iterative ML approach", funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation), Walter Benjamin Programme, project number 526052741.
@misc{kaczmarczyk2026dermalitclip,
author = {Kaczmarczyk, Robert},
title = {DermaLit-CLIP ViT-B/16: CLIP fine-tuned on a provenance-tracked dermatology image--text corpus},
year = {2026},
publisher = {Hugging Face},
howpublished = {\url{https://huggingface.co/laion/DermaLit-CLIP-ViT-B-16}}
}
- Downloads last month
- -
Model tree for laion/DermaLit-CLIP-ViT-B-16
Base model
openai/clip-vit-base-patch16