Image Feature Extraction
timm
Safetensors
English
pathology
histopathology
feature-extraction
spectra
Instructions to use medarc/OpenMidnight-1.1-SPECTRA with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- timm
How to use medarc/OpenMidnight-1.1-SPECTRA with timm:
import timm model = timm.create_model("hf-hub:medarc/OpenMidnight-1.1-SPECTRA", pretrained=True) - Notebooks
- Google Colab
- Kaggle
|
Download README.md from medarc/OpenMidnight-1.1-SPECTRA: direct link, hf CLI and curl.
- Browser
- Download file 2.8 kB
-
https://huggingface.co/medarc/OpenMidnight-1.1-SPECTRA/resolve/main/README.md
- Command line
-
hf download hf://medarc/OpenMidnight-1.1-SPECTRA/README.md
-
curl -L -o README.md https://huggingface.co/medarc/OpenMidnight-1.1-SPECTRA/resolve/main/README.md
2.8 kB
| language: en | |
| license: apache-2.0 | |
| library_name: timm | |
| pipeline_tag: image-feature-extraction | |
| base_model: medarc/OpenMidnight-1.1 | |
| base_model_relation: finetune | |
| tags: | |
| - pathology | |
| - histopathology | |
| - feature-extraction | |
| - spectra | |
| # OpenMidnight-1.1-SPECTRA | |
| [MedARC](https://www.medarc.ai/) · [Sophont](https://sophont.med/) · [Paper (coming soon)](#paper) | |
| OpenMidnight-1.1-SPECTRA is [OpenMidnight-1.1](https://huggingface.co/medarc/OpenMidnight-1.1) adapted with **SPECTRA** by MedARC and Sophont. We trained on registered PLISM image tiles, aligning representations of the same tissue across scanners and stains with a contrastive objective. The learned updates are merged into the full model weights. | |
| The model maps RGB histology tiles to **1,536-dimensional embeddings** for downstream analysis. All inference weights are included; no separate base checkpoint, adapter, or cluster access is required. | |
| ## Inference | |
| In a Python environment: | |
| ```bash | |
| uv pip install torch torchvision "timm==1.0.30" huggingface_hub safetensors pillow | |
| ``` | |
| ```python | |
| import torch | |
| import timm | |
| from PIL import Image | |
| from timm.data import create_transform, resolve_model_data_config | |
| device = "cuda" # Use "cpu" to run without a GPU. | |
| model = timm.create_model( | |
| "hf-hub:medarc/OpenMidnight-1.1-SPECTRA", pretrained=True, | |
| ).eval().to(device) | |
| transform = create_transform(**resolve_model_data_config(model), is_training=False) | |
| image = Image.open("tile.png").convert("RGB") | |
| batch = transform(image).unsqueeze(0).to(device) | |
| with torch.inference_mode(): | |
| tokens = model.forward_features(batch) | |
| embedding = tokens[:, 0] | |
| print(embedding.shape) # torch.Size([1, 1536]) | |
| ``` | |
| The configuration uses `crop_pct=1.0` and `crop_mode="squash"` to resize the full image directly to 224 × 224 pixels. It uses bicubic interpolation and ImageNet normalization stored in `config.json`. The embedding uses the CLS token. | |
| These are tile embeddings. Slide-level predictions require a downstream model. | |
| ### Segmentation preprocessing | |
| The transform resizes the entire image to 224 × 224 without removing tissue. For square tiles, this preserves the aspect ratio; rectangular images are stretched to fit. The final center-crop operation generated by timm is a no-op because the image is already 224 × 224. | |
| The image and segmentation mask must cover the same tissue region. With a full-tile resize, decoder logits can be interpolated back to the original mask size before computing the loss or metrics, as THUNDER does. If resizing a discrete label mask instead, use nearest-neighbor interpolation. Exclude the CLS and register tokens when passing patch features to a segmentation decoder: `patch_tokens = tokens[:, model.num_prefix_tokens:]`. | |
| ## Paper | |
| Coming soon. We will add the paper link and citation here. | |