SkinMap-Factory / README.md
FabianGroeger's picture
Add the model card
f83e83e verified
|
Raw History Blame Contribute Delete
3.2 kB
metadata
license: cc-by-nc-4.0
library_name: pytorch
tags:
  - dermatology
  - medical-imaging
  - image-generation
  - diffusion
  - rectified-flow
pipeline_tag: image-to-image
extra_gated_prompt: >-
  This model generates synthetic skin images. It is for research only and must
  not be used for diagnosis or patient care. Access is approved by the Digital
  Dermatology maintainers.
extra_gated_fields:
  Name: text
  Affiliation: text
  Intended use: text
  I will use this model for non-commercial research only: checkbox

SkinMap Factory

SkinMap Factory renders skin images from SkinMap embeddings and clinical metadata. Code, training and editing methods: Digital-Dermatology/skinmap-factory.

File skinmap_factory_step2195000.pt (EMA weights, codec and config)
SHA-256 3559b8f401159f02ddf021e5dad43ce411d116d8fef81529c34667360a4f86a3
Model PixelDiT, 729M parameters, pixel space, 16 x 16 patches
Objective Rectified flow, 2,195,000 steps, REPA alignment to DINOv2
Output 256 x 256 RGB
Conditions 1024-d SkinMap embedding, plus modality, body_region, fitzpatrick, gender, age, continent, icd_category
Sampling defaults 100 steps, guidance 1.5

Access

The repo is gated. Request access above, wait for approval, then log in with hf auth login (or set HF_TOKEN) on each machine that downloads the model.

Usage

pip install git+https://github.com/Digital-Dermatology/skinmap-factory.git
from transformers import AutoModel
from torchvision.utils import save_image
from skinmap_factory import generate, load_decoder

encoder = AutoModel.from_pretrained("Digital-Dermatology/SkinMap", trust_remote_code=True, device="cuda")
z = encoder.encode_image("lesion.jpg")                      # (1024,) SkinMap embedding

decoder, codec, config = load_decoder(device="cuda")        # downloads this repo once
images = generate(decoder, codec, config, [z] * 4,
                  metadata={"modality": "dermoscopy", "body_region": "trunk_back"},
                  seed=0)                                    # (4, 3, 256, 256) in [0, 1]
save_image(images, "samples.png")                          # four samples of one embedding

The embedding sets most of the image, and metadata is optional: fields left out count as unknown. codec.vocab lists the allowed values of each field. A batch of six images peaks at about 6 GB of GPU memory. From the command line:

skinmap-factory-sample --z-npy embeddings.npy --output samples/

The repo README documents the metadata values and options.

Limits

The model learned from public dermatology datasets, and it inherits their gaps: few images of darker skin (Fitzpatrick V and VI), few children, and few images of rare diseases. Generated images are synthetic. Do not use them as clinical evidence or present them as real patients.

License

Weights: CC BY-NC 4.0, because several training datasets allow only non-commercial use. Code: Apache-2.0.