You need to agree to share your contact information to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

This model and associated code are released under the CC-BY-NC-ND 4.0 license and may only be used for non-commercial, academic research purposes with proper attribution. Any commercial use, sale, or other monetization of the CONCH-Z model and its derivatives, which include models trained on outputs from the CONCH-Z model or datasets created from the CONCH-Z model, is prohibited and requires prior approval. Please note that the primary email used to sign up for your Hugging Face account must match your institutional email to receive approval. By downloading the model, you attest that all information (affiliation, research use) is correct and up-to-date. Downloading the model requires prior registration on Hugging Face and agreeing to the terms of use. By downloading this model, you agree not to distribute, publish or reproduce a copy of the model. If another user within your organization wishes to use the CONCH-Z model, they must register as an individual user and agree to comply with the terms of use. Users may not attempt to re-identify the deidentified data used to develop the underlying model. If you are a commercial entity, please contact the corresponding author.

CONCH-Z

The text encoder for CONCH v1.5

Beyond Sparse Captions: Aligning Slide-Level Text and Patch-Level Vision in Pathology
NeurIPS 2026 · Spotlight

Daniel Shao*, Dongmin Bang*, Luca L. Weishaupt, Sophia J. Wagner, Nic G. Reitsam,
Ming Y. Lu, Long P. Le, Richard J. Chen†, Faisal Mahmood†

_{*Co-first authors · †Co-senior authors}

Paper · Code · Quickstart · Results · Citation

CONCH-Z brings text encoding and zero-shot classification to CONCH v1.5. The previous CONCH v1.5 release provided only the vision encoder. CONCH-Z pairs that same vision encoder, unchanged, with an aligned text tower, so you can compare pathology image patches with natural-language class descriptions without training a task-specific classifier.

CONCH-Z can directly replace CONCH v1.5. The vision encoder is identical, so you don't need to re-extract patch features you already have from CONCH v1.5 (e.g. via Trident or TITAN's return_conch()).
CONCH-Z adds a text tower and the CONCH v1.5 image projection layer (proj_contrast), which the public CONCH v1.5 release leaves out. Projected image features and text embeddings share one space, which enables zero-shot classification.

Note: This release focuses on using CONCH-Z. Code for LIMIT training is coming soon in the GitHub repository.

Model details


Developed by	Mahmood Lab AI for Pathology @ Harvard/BWH
Model type	Vision-language model: CONCH v1.5 vision encoder (ViT-L/16, frozen) with a LIMIT-trained CoCa text encoder (12-layer transformer), sharing a 768-d embedding space
Parameters	418M in total: vision encoder 307M (unchanged from CONCH v1.5), image projection 0.6M, text encoder 110M
Input	RGB H&E patches resized to 448 × 448 (`model.eval_transform`), or precomputed CONCH v1.5 patch features; text prompts of up to 127 tokens
Training	LIMIT, on approximately 335,000 clinical pathology reports. The image tower and projection are frozen; only the text tower is trained.
Base model	MahmoodLab/conchv1_5
Paper	Beyond Sparse Captions: Aligning Slide-Level Text and Patch-Level Vision in Pathology, NeurIPS 2026 (Spotlight)
Repository	github.com/mahmoodlab/LIMIT
License	CC-BY-NC-ND 4.0

From clinical reports to zero-shot predictions

CONCH-Z is trained with LIMIT (Locked Image Multiple Instance PreTraining), which aligns the text encoder to a frozen visual space using approximately 335,000 clinical pathology reports. The image tower stays frozen and only the text tower is trained. At inference, candidate class descriptions become text queries for zero-shot classification.

CONCH-Z animation showing report-guided LIMIT training and zero-shot slide classification with candidate class prompts

_{LIMIT training and zero-shot slide inference. Tissue regions follow the paper's annotations; animated scores and attention are illustrative.}

Requesting access

Access requires agreeing to the terms of use in the form on this page, with the primary email of your Hugging Face account matching your institutional email. Requests from accounts whose primary email is personal (@gmail/@hotmail/@qq) are denied. To fix this, (1) add your official institutional email to your Hugging Face account and verify it, and (2) set it as your primary email. Other common reasons for denial are mistakes in the form, for example a full name with abbreviations, an affiliation that is not spelled out, an insufficient description of the research use, or an unrecognized email domain.

Installation & access

The model is gated. Request access with the form on this page (see Requesting access), then log in:

pip install torch torchvision timm transformers huggingface_hub
huggingface-cli login

Loading the model

import torch
from transformers import AutoModel

model = AutoModel.from_pretrained("MahmoodLab/CONCH-Z", trust_remote_code=True)
model = model.eval().to("cuda")

The model has three parts:

Component	Role
Vision encoder	Same as CONCH v1.5 (ViT-L/16 + attentional pooler). Maps a patch to the 768-d feature that the public CONCH v1.5 release and Trident's `conch_v15` encoder output.
Image projection (`visual.proj_contrast`)	A 768 × 768 linear map from CONCH v1.5 features into the shared image-text space. It is CONCH v1.5's own contrastive head, kept frozen during LIMIT, but it is not part of the public CONCH v1.5 weights.
Text encoder	New in CONCH-Z. Maps a prompt to a 768-d embedding in the shared space.

Important: Image features must pass through the projection before you compare them with text. encode_image does this for you. For CONCH v1.5 features computed elsewhere, call model.project_image_features first. If you skip it, zero-shot accuracy falls below chance: SPIDER-C balanced accuracy drops from 58.8 to 10.3.

Encoding images

from PIL import Image

image = Image.open("patch.png").convert("RGB")
image = model.eval_transform(image).unsqueeze(0).to("cuda")   # (1, 3, 448, 448)

with torch.inference_mode():
    image_emb = model.encode_image(image)                      # (1, 768), projected and L2-normalized

encode_image runs the vision encoder and then the projection. To get the unprojected CONCH v1.5 feature (identical to Trident's conch_v15 output), use model.encode_image_features(image).

Using precomputed CONCH v1.5 features

If you already have CONCH v1.5 patch features (for example .h5 files from Trident's conch_v15 encoder), you don't need to re-extract them. Project them into the shared space:

import h5py

with h5py.File("slide.h5", "r") as f:
    features = torch.from_numpy(f["features"][:])              # (N, 768) CONCH v1.5 features

with torch.inference_mode():
    image_emb = model.project_image_features(features)         # (N, 768), features @ proj_contrast, L2-normalized

Encoding text

prompts = [
    "a lymph node biopsy containing metastatic breast carcinoma",
    "a lymph node biopsy containing normal lymphoid tissue",
]

with torch.inference_mode():
    input_ids = model.tokenizer(prompts).to("cuda")               # (C, 128)
    text_emb = model.encode_text(input_ids, normalize=True)      # (C, 768)

Tip: as in the paper, prepend the tissue or organ type to each class name (e.g. "breast tissue, invasive ductal carcinoma"). This usually helps zero-shot accuracy.

Zero-shot classification

Option 1: built-in helper

zero_shot_classifier builds one L2-normalized embedding per class. It averages over any number of synonyms per class and prompt templates (CLASSNAME is replaced by each synonym). zero_shot takes projected image embeddings (from encode_image or project_image_features) and returns softmax probabilities scaled by the model's learned temperature:

classnames = [
    ["metastatic breast carcinoma", "tumor"],     # class 0 synonyms
    ["normal lymph node", "lymphocytes"],         # class 1 synonyms
]
templates = ["CLASSNAME", "an image of CLASSNAME", "lymph node, CLASSNAME"]

with torch.inference_mode():
    classifier = model.zero_shot_classifier(classnames, templates, device="cuda")  # (768, num_classes)
    probs = model.zero_shot(image_emb, classifier)                                 # (N, num_classes)

pred = probs.argmax(dim=-1)

Option 2: project, L2-normalize and multiply

Zero-shot classification is cosine similarity between projected image embeddings and text embeddings. You can compute it directly, which is handy when you already have a large set of precomputed CONCH v1.5 patch features:

import torch.nn.functional as F

proj = model.visual.proj_contrast                           # (768, 768) image projection

image_emb = F.normalize(features.cuda() @ proj, dim=-1)     # (N, 768)  projected patch embeddings
text_emb  = F.normalize(text_emb, dim=-1)                   # (C, 768)  one embedding per class

similarity = image_emb @ text_emb.T                         # (N, C) cosine similarities
pred = similarity.argmax(dim=-1)                            # predicted class per patch

Results: zero-shot patch-level classification

C16 is reported as AUC. SPIDER and CATCH are reported as balanced accuracy, averaged across organs and tasks. The number of classes is in parentheses. Best result in bold. See the paper for per-organ results and bootstrap standard deviations.

Benchmark	PLIP	QuiltNet	MUSK	MI-Zero	CONCH v1.5	CONCH-Z
CAMELYON-16, lymph node (2)	88.4	91.7	82.7	59.9	90.3	96.7
SPIDER-C, coarse (7–9 per organ)	34.8	30.1	51.3	44.2	49.7	58.8
SPIDER-F, fine-grained (13–24 per organ)	25.6	19.7	44.1	34.9	42.9	51.3
CATCH, canine skin (2 / 11)	49.3	48.3	59.1	51.3	47.9	64.4

Intended use and limitations

Intended use: non-commercial academic research on H&E histopathology, such as zero-shot patch classification, scoring patches against text queries, and extracting image and text embeddings for downstream models.
Out of scope: clinical diagnosis or any other clinical decision-making, and commercial use.
Prompt sensitivity: zero-shot accuracy depends on how classes are worded. Prepending the organ or tissue type, and averaging several synonyms or templates per class with zero_shot_classifier, usually helps.
No captioning: CONCH-Z ships the contrastive image and text encoders only. The CoCa caption decoder is not included.

License

The model weights are released under CC-BY-NC-ND 4.0 for non-commercial academic research only. The model is not validated for clinical diagnosis.

Citation

If you use CONCH-Z or LIMIT in your research, please cite:

@inproceedings{shao2026limit,
  title     = {Beyond Sparse Captions: Aligning Slide-Level Text and Patch-Level Vision in Pathology},
  author    = {Shao, Daniel and Bang, Dongmin and Weishaupt, Luca L. and Wagner, Sophia J. and Reitsam, Nic G. and Lu, Ming Y. and Le, Long P. and Chen, Richard J. and Mahmood, Faisal},
  booktitle = {Advances in Neural Information Processing Systems},
  year      = {2026},
  url       = {https://github.com/mahmoodlab/LIMIT}
}

Questions

Please open a GitHub issue for questions about using CONCH-Z.

Downloads last month: 2

Safetensors

Model size

0.4B params

Tensor type

F32

Model tree for MahmoodLab/CONCH-Z

Base model

MahmoodLab/conchv1_5

Finetuned

(1)

this model

Paper for MahmoodLab/CONCH-Z

Beyond Sparse Captions: Aligning Slide-Level Text and Patch-Level Vision in Pathology

Paper • 2610.17233 • Published 3 days ago