README.md

---
license: cc-by-nc-4.0
library_name: transformers
tags: [pathology, vision-language, histopathology, lora, zero-shot]
---

# Lumen

Lumen aligns frozen Virchow2 and BioMedBERT encoders using rank-4 LoRA adapters and projection heads into a shared 512-dimensional space.

This repository contains only the trained alignment parameters (2,984,961 parameters; 12.0 MB). The backbone weights are loaded from their original repositories.


## Usage

```python
import torch
from transformers import AutoModel

model = AutoModel.from_pretrained("digitalpathologybern/Lumen", trust_remote_code=True).eval()

# images: (B, 3, 224, 224), preprocessed with the Virchow2 transform
tokens = model.tokenizer(["metastatic carcinoma in a lymph node",
                          "benign lymph node"],
                         padding=True, truncation=True, max_length=128,
                         return_tensors="pt")
with torch.no_grad():
    out = model(images=images, tokens=tokens)

img = torch.nn.functional.normalize(out["image_embeds"], dim=-1)
txt = torch.nn.functional.normalize(out["text_embeds"], dim=-1)
logits = out["logit_scale"] * img @ txt.T
```

images should have shape (B, 3, 224, 224) and be preprocessed using the Virchow2 transform.

encode_image and encode_text can be used to run either encoder separately.

`timm`, `transformers`, and `peft` are required.

Both backbones are downloaded from their own repositories on the first call.

## Licence

The 2,984,961 parameters in this repository are released under CC-BY-NC-4.0.
The model also loads two third-party backbones, which are not redistributed here
and carry their own terms.

| Component | Source | Licence |
| --- | --- | --- |
| Adapters, projection heads, logit scale | this repository | CC-BY-NC-4.0 |
| Vision encoder | [paige-ai/Virchow2](https://huggingface.co/paige-ai/Virchow2) | CC-BY-NC-ND-4.0, gated |
| Text encoder | [microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract-fulltext](https://huggingface.co/microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract-fulltext) | MIT |

**The assembled model is non-commercial**, a constraint that comes from Virchow2
and holds whatever these weights say. Read the backbone terms against your
intended use rather than reading the tag above as covering the whole.

Adapting these weights further is permitted. Virchow2 additionally forbids
derivatives of itself, so fine-tune the alignment, not the vision encoder.