Update README.md

README.md

@@ -11,30 +11,58 @@ tags:
 - foundation-model
 - vl-bert
 - siglip2
-pipeline_tag: image-classification
+- segmentation
+- zero-shot-classification
+pipeline_tag: image-feature-extraction
 library_name: pytorch
 ---
 
 # Luna — Retinal Foundation Model with Hybrid Clinical & Anatomical Supervision
 
-**Luna** is a multimodal, multi-task retinal foundation model: a single-stream VL-BERT (SigLIP‑2 ViT‑Base vision encoder + PubMedBERT-initialized BERT‑Base text encoder) that reconstructs **masked clinical tokens** (disease label, age, sex, eye laterality) from 512×512 fundus images with a UNETR decoder jointly supervising vessel/optic-disc segmentation.
+**Luna** is a multimodal, multi-task retinal foundation model for high-resolution color fundus photographs (CFPs).
+Instead of pixel reconstruction, it is pre-trained by **predicting masked clinical tokens** (disease label, age, sex,
+eye laterality) from the image, while a **UNETR decoder jointly supervises vessel / optic-disc segmentation**.
+
+Architecturally it is a single-stream VL-BERT: a **SigLIP-2 ViT-Base/16 (512px) vision encoder**, a **BERT-Base
+masked-LM initialized from PubMedBERT**, and a **UNETR head** for anatomy.
+
+> Paper: N/A
 
-> Paper: *Multimodal Foundation Model of High-Resolution Fundus Photos with Clinical Metadata via Predicting Demographics and Anatomic Structure* — Lim, H. et al.
 > Code: https://github.com/loopback-kr/Luna
 
 ## Model Details
 
 | Attribute | Value |
 |---|---|
-| **Architecture** | SigLIP‑2 ViT‑Base/16 vision encoder + BERT‑Base language encoder (single-stream VL-BERT fusion) + UNETR segmentation decoder |
-| **Parameters** | 93.5M (ViT‑Base backbone) |
+| **Architecture** | SigLIP-2 ViT-Base/16 vision encoder + BERT-Base masked-LM (single-stream VL-BERT fusion, 128 compressed vision tokens) + UNETR segmentation decoder |
+| **Parameters** | 243M total; ≈93.5M in the SigLIP-2 ViT-Base vision encoder that downstream tasks reuse |
 | **Input resolution** | 512 × 512 RGB (CLAHE-enhanced) |
-| **Vision encoder init.** | `google/siglip2-base-patch16-512` (ImageNet-pretrained) |
-| **Text encoder init.** | `NeuML/pubmedbert-base-embeddings`, extended with `[LABEL]`, `[AGE]`, `[SEX]`, `[DIR]` special tokens |
-| **Pre-training objective** | Masked clinical-token prediction (label/age/sex/direction) + vessel/optic-disc segmentation (Dice loss), *not* pixel-level MAE reconstruction |
+| **Vision encoder init.** | `google/siglip2-base-patch16-512` |
+| **Text encoder init.** | `NeuML/pubmedbert-base-embeddings` tokenizer, extended with `[LABEL]`, `[AGE]`, `[SEX]`, `[DIR]` special tokens |
+| **Pre-training objective** | Masked clinical-token prediction (label / age / sex / direction) + vessel & optic-disc Dice loss, *not* pixel-level MAE reconstruction |
 | **Pre-training data** | 361,519 CFPs (52K public + 309K private institutional images) |
 | **License** | MIT |
 
+Prompt templates used during pre-training: `"A CFP image of {CLS}"`, `"age is {CLS} years old"`,
+`"gender is {CLS}"`, `"this is {CLS} direction eye"`.
+
+## Files in this repository
+
+This is the full `accelerate` training state at epoch 399:
+
+| File | Contents |
+|---|---|
+| `model.safetensors` / `pytorch_model.bin` | Model weights (243,122,497 parameters, fp32) |
+| `optimizer.bin`, `scheduler.bin`, `scaler.pt` | AdamW / LR-scheduler / GradScaler state, for exact resumption |
+| `random_states_{0..3}.pkl` | RNG states of the four training processes |
+
+## Usage
+
+Full training, fine-tuning and evaluation code lives in the
+[GitHub repository](https://github.com/loopback-kr/Luna) — see its `README.md` for the *Upstream training*,
+*Downstream training*, *Zero-Shot* and *Segmentation* sections. Checkpoint paths there expect
+`pytorch_model.bin`; passing the containing directory instead resumes optimizer and scheduler state as well.
+
 ## Training Data
 
 **\*** = private institutional data, not available externally.
@@ -84,8 +112,16 @@ library_name: pytorch
 
 ```bibtex
 @article{lim_luna_2026,
-  title   = {Multimodal Foundation Model of High-Resolution Fundus Photos with Clinical Metadata via Predicting Demographics and Anatomic Structure},
-  author  = {Lim, Hyunseok and Kim, Junseok and Oh, Joonseo and Kim, Kanghyun and Lim, Jongsoo and Jeong, Jinhoon and Jeong, Hy and Kim, Yoonjeon and Kim, Namkug},
-  journal = {(manuscript in preparation)},
+  title   = {Multimodal Foundation Model for High-Resolution Fundus Photographs Incorporating
+             Clinical Metadata via Prediction of Demographics and Anatomical Structures},
+  author  = {Lim, Hyunseok and Kim, Junseok and Oh, Joonseo and Kim, Kanghyun and Lim, Jongsoo
+             and Jeong, Jinhoon and Jeong, Hy and Kim, Yoonjeon and Kim, Namkug},
+  year    = {2026},
 }
 ```
+
+## License
+
+Released under the MIT License — Copyright (c) 2026 Hyunseok Lim.
+The third-party datasets listed above carry their own licenses and access terms, and the Asan Medical Center
+institutional cohorts are not redistributable.