SkinMap-Factory / README.md
FabianGroeger's picture
Add the model card
f83e83e verified
|
Raw History Blame Contribute Delete
3.2 kB
---
license: cc-by-nc-4.0
library_name: pytorch
tags:
- dermatology
- medical-imaging
- image-generation
- diffusion
- rectified-flow
pipeline_tag: image-to-image
extra_gated_prompt: >-
This model generates synthetic skin images. It is for research only and must not be
used for diagnosis or patient care. Access is approved by the Digital Dermatology
maintainers.
extra_gated_fields:
Name: text
Affiliation: text
Intended use: text
I will use this model for non-commercial research only: checkbox
---
# SkinMap Factory
SkinMap Factory renders skin images from [SkinMap](https://huggingface.co/Digital-Dermatology/SkinMap) embeddings and clinical metadata. Code, training and editing methods: [Digital-Dermatology/skinmap-factory](https://github.com/Digital-Dermatology/skinmap-factory).
| | |
|---|---|
| File | `skinmap_factory_step2195000.pt` (EMA weights, codec and config) |
| SHA-256 | `3559b8f401159f02ddf021e5dad43ce411d116d8fef81529c34667360a4f86a3` |
| Model | PixelDiT, 729M parameters, pixel space, 16 x 16 patches |
| Objective | Rectified flow, 2,195,000 steps, REPA alignment to DINOv2 |
| Output | 256 x 256 RGB |
| Conditions | 1024-d SkinMap embedding, plus `modality`, `body_region`, `fitzpatrick`, `gender`, `age`, `continent`, `icd_category` |
| Sampling defaults | 100 steps, guidance 1.5 |
## Access
The repo is gated. Request access above, wait for approval, then log in with `hf auth login` (or set `HF_TOKEN`) on each machine that downloads the model.
## Usage
```sh
pip install git+https://github.com/Digital-Dermatology/skinmap-factory.git
```
```python
from transformers import AutoModel
from torchvision.utils import save_image
from skinmap_factory import generate, load_decoder
encoder = AutoModel.from_pretrained("Digital-Dermatology/SkinMap", trust_remote_code=True, device="cuda")
z = encoder.encode_image("lesion.jpg") # (1024,) SkinMap embedding
decoder, codec, config = load_decoder(device="cuda") # downloads this repo once
images = generate(decoder, codec, config, [z] * 4,
metadata={"modality": "dermoscopy", "body_region": "trunk_back"},
seed=0) # (4, 3, 256, 256) in [0, 1]
save_image(images, "samples.png") # four samples of one embedding
```
The embedding sets most of the image, and metadata is optional: fields left out count as unknown. `codec.vocab` lists the allowed values of each field. A batch of six images peaks at about 6 GB of GPU memory. From the command line:
```sh
skinmap-factory-sample --z-npy embeddings.npy --output samples/
```
The [repo README](https://github.com/Digital-Dermatology/skinmap-factory#released-model) documents the metadata values and options.
## Limits
The model learned from public dermatology datasets, and it inherits their gaps: few images of darker skin (Fitzpatrick V and VI), few children, and few images of rare diseases. Generated images are synthetic. Do not use them as clinical evidence or present them as real patients.
## License
Weights: CC BY-NC 4.0, because several training datasets allow only non-commercial use. Code: Apache-2.0.