pscotti's picture
Update README.md
83f82cb verified
|
Raw History Blame Contribute Delete
2.8 kB
metadata
language: en
license: apache-2.0
library_name: timm
pipeline_tag: image-feature-extraction
base_model: medarc/OpenMidnight-1.1
base_model_relation: finetune
tags:
  - pathology
  - histopathology
  - feature-extraction
  - spectra

OpenMidnight-1.1-SPECTRA

MedARC · Sophont · Paper (coming soon)

OpenMidnight-1.1-SPECTRA is OpenMidnight-1.1 adapted with SPECTRA by MedARC and Sophont. We trained on registered PLISM image tiles, aligning representations of the same tissue across scanners and stains with a contrastive objective. The learned updates are merged into the full model weights.

The model maps RGB histology tiles to 1,536-dimensional embeddings for downstream analysis. All inference weights are included; no separate base checkpoint, adapter, or cluster access is required.

Inference

In a Python environment:

uv pip install torch torchvision "timm==1.0.30" huggingface_hub safetensors pillow
import torch
import timm
from PIL import Image
from timm.data import create_transform, resolve_model_data_config

device = "cuda"  # Use "cpu" to run without a GPU.
model = timm.create_model(
    "hf-hub:medarc/OpenMidnight-1.1-SPECTRA", pretrained=True,
).eval().to(device)
transform = create_transform(**resolve_model_data_config(model), is_training=False)

image = Image.open("tile.png").convert("RGB")
batch = transform(image).unsqueeze(0).to(device)
with torch.inference_mode():
    tokens = model.forward_features(batch)
    embedding = tokens[:, 0]
print(embedding.shape)  # torch.Size([1, 1536])

The configuration uses crop_pct=1.0 and crop_mode="squash" to resize the full image directly to 224 × 224 pixels. It uses bicubic interpolation and ImageNet normalization stored in config.json. The embedding uses the CLS token.

These are tile embeddings. Slide-level predictions require a downstream model.

Segmentation preprocessing

The transform resizes the entire image to 224 × 224 without removing tissue. For square tiles, this preserves the aspect ratio; rectangular images are stretched to fit. The final center-crop operation generated by timm is a no-op because the image is already 224 × 224.

The image and segmentation mask must cover the same tissue region. With a full-tile resize, decoder logits can be interpolated back to the original mask size before computing the loss or metrics, as THUNDER does. If resizing a discrete label mask instead, use nearest-neighbor interpolation. Exclude the CLS and register tokens when passing patch features to a segmentation decoder: patch_tokens = tokens[:, model.num_prefix_tokens:].

Paper

Coming soon. We will add the paper link and citation here.