DPbutler commited on
Commit
804ef3e
·
verified ·
1 Parent(s): 78d64de

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +8 -15
README.md CHANGED
@@ -6,13 +6,10 @@ tags: [pathology, vision-language, histopathology, lora, zero-shot]
6
 
7
  # Lumen
8
 
9
- Cross-modal alignment for a frozen [Virchow2](https://huggingface.co/paige-ai/Virchow2) vision encoder
10
- and a frozen [BioMedBERT](https://huggingface.co/microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract-fulltext) text encoder.
 
11
 
12
- This repository holds **only the alignment**: 2,984,961 parameters
13
- (12.0 MB) comprising rank-4 LoRA adapters on both
14
- encoders, two projection heads into a shared 512-dimensional space, and a learned
15
- logit scale.
16
 
17
  ## Usage
18
 
@@ -35,12 +32,13 @@ txt = torch.nn.functional.normalize(out["text_embeds"], dim=-1)
35
  logits = out["logit_scale"] * img @ txt.T
36
  ```
37
 
38
- The arguments are `images` and `tokens`, not the `pixel_values` and `input_ids`
39
- that most Hub models take. `encode_image` and `encode_text` are available if you
40
- want one tower at a time.
 
 
41
 
42
  Both backbones are downloaded from their own repositories on the first call.
43
- `timm`, `transformers` and `peft` are required.
44
 
45
  ### The loading warning is expected
46
 
@@ -79,8 +77,3 @@ and what to do. In short, open its page on the Hub, accept the licence, then
79
  authenticate with `huggingface-cli login`. Already have the weights locally? Pass
80
  `vision_path=` or `text_path=` instead.
81
 
82
- ## What this is
83
-
84
- The 740,721,664 backbone parameters withheld here are unmodified by
85
- training, so this adapter plus the public backbones reconstructs the evaluated
86
- model exactly.
 
6
 
7
  # Lumen
8
 
9
+ Lumen aligns frozen Virchow2 and BioMedBERT encoders using rank-4 LoRA adapters and projection heads into a shared 512-dimensional space.
10
+
11
+ This repository contains only the trained alignment parameters (2,984,961 parameters; 12.0 MB). The backbone weights are loaded from their original repositories.
12
 
 
 
 
 
13
 
14
  ## Usage
15
 
 
32
  logits = out["logit_scale"] * img @ txt.T
33
  ```
34
 
35
+ images should have shape (B, 3, 224, 224) and be preprocessed using the Virchow2 transform.
36
+
37
+ encode_image and encode_text can be used to run either encoder separately.
38
+
39
+ `timm`, `transformers`, and `peft` are required.
40
 
41
  Both backbones are downloaded from their own repositories on the first call.
 
42
 
43
  ### The loading warning is expected
44
 
 
77
  authenticate with `huggingface-cli login`. Already have the weights locally? Pass
78
  `vision_path=` or `text_path=` instead.
79