Feature Extraction
Transformers
Safetensors
lumen
pathology
vision-language
histopathology
lora
zero-shot
custom_code
Instructions to use digitalpathologybern/Lumen with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use digitalpathologybern/Lumen with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("feature-extraction", model="digitalpathologybern/Lumen", trust_remote_code=True)# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("digitalpathologybern/Lumen", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
Update README.md
Browse files
README.md
CHANGED
|
@@ -6,13 +6,10 @@ tags: [pathology, vision-language, histopathology, lora, zero-shot]
|
|
| 6 |
|
| 7 |
# Lumen
|
| 8 |
|
| 9 |
-
|
| 10 |
-
|
|
|
|
| 11 |
|
| 12 |
-
This repository holds **only the alignment**: 2,984,961 parameters
|
| 13 |
-
(12.0 MB) comprising rank-4 LoRA adapters on both
|
| 14 |
-
encoders, two projection heads into a shared 512-dimensional space, and a learned
|
| 15 |
-
logit scale.
|
| 16 |
|
| 17 |
## Usage
|
| 18 |
|
|
@@ -35,12 +32,13 @@ txt = torch.nn.functional.normalize(out["text_embeds"], dim=-1)
|
|
| 35 |
logits = out["logit_scale"] * img @ txt.T
|
| 36 |
```
|
| 37 |
|
| 38 |
-
|
| 39 |
-
|
| 40 |
-
|
|
|
|
|
|
|
| 41 |
|
| 42 |
Both backbones are downloaded from their own repositories on the first call.
|
| 43 |
-
`timm`, `transformers` and `peft` are required.
|
| 44 |
|
| 45 |
### The loading warning is expected
|
| 46 |
|
|
@@ -79,8 +77,3 @@ and what to do. In short, open its page on the Hub, accept the licence, then
|
|
| 79 |
authenticate with `huggingface-cli login`. Already have the weights locally? Pass
|
| 80 |
`vision_path=` or `text_path=` instead.
|
| 81 |
|
| 82 |
-
## What this is
|
| 83 |
-
|
| 84 |
-
The 740,721,664 backbone parameters withheld here are unmodified by
|
| 85 |
-
training, so this adapter plus the public backbones reconstructs the evaluated
|
| 86 |
-
model exactly.
|
|
|
|
| 6 |
|
| 7 |
# Lumen
|
| 8 |
|
| 9 |
+
Lumen aligns frozen Virchow2 and BioMedBERT encoders using rank-4 LoRA adapters and projection heads into a shared 512-dimensional space.
|
| 10 |
+
|
| 11 |
+
This repository contains only the trained alignment parameters (2,984,961 parameters; 12.0 MB). The backbone weights are loaded from their original repositories.
|
| 12 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 13 |
|
| 14 |
## Usage
|
| 15 |
|
|
|
|
| 32 |
logits = out["logit_scale"] * img @ txt.T
|
| 33 |
```
|
| 34 |
|
| 35 |
+
images should have shape (B, 3, 224, 224) and be preprocessed using the Virchow2 transform.
|
| 36 |
+
|
| 37 |
+
encode_image and encode_text can be used to run either encoder separately.
|
| 38 |
+
|
| 39 |
+
`timm`, `transformers`, and `peft` are required.
|
| 40 |
|
| 41 |
Both backbones are downloaded from their own repositories on the first call.
|
|
|
|
| 42 |
|
| 43 |
### The loading warning is expected
|
| 44 |
|
|
|
|
| 77 |
authenticate with `huggingface-cli login`. Already have the weights locally? Pass
|
| 78 |
`vision_path=` or `text_path=` instead.
|
| 79 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|