shantam00 commited on
Commit
42caf42
Β·
verified Β·
1 Parent(s): f8beb05

Update MIDL 2026 model card

Browse files
Files changed (1) hide show
  1. README.md +104 -5
README.md CHANGED
@@ -8,13 +8,91 @@ tags:
8
  - llava
9
  - lora
10
  - contrastive-decoding
 
11
  ---
12
 
13
- # CWCD adapters
14
 
15
- This model repository contains the eight category-specific rank-1 LoRA adapters for **CWCD: Category-Wise Contrastive Decoding for Structured Medical Report Generation**.
16
 
17
- Inference code and usage instructions are maintained at [`shantamsrivastava/CWCD`](https://github.com/shantamsrivastava/CWCD).
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
18
 
19
  ```text
20
  adapters/
@@ -28,6 +106,27 @@ adapters/
28
  └── wall/
29
  ```
30
 
31
- Each directory contains `adapter_config.json` and `adapter_model.bin`. The GitHub inference script downloads these files through `huggingface_hub.snapshot_download()` and combines the eight category-wise findings into a structured findings report.
 
 
 
 
32
 
33
- > Research use only. CWCD is not a medical device and is not a substitute for review by a qualified clinician.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
8
  - llava
9
  - lora
10
  - contrastive-decoding
11
+ - medical-report-generation
12
  ---
13
 
14
+ # 🩻 [MIDL 2026 Oral] CWCD Category-Specific LoRA Adapters
15
 
16
+ [Shantam Srivastava](https://scholar.google.com/citations?user=UXG7qiYAAAAJ&hl=en), [Mahesh Bhosale](https://bhosalems.github.io/), [David Doermann](https://scholar.google.com/citations?user=RoGOW9AAAAAJ&hl=en), [Mingchen Gao](https://cse.buffalo.edu/~mgao8/)
17
 
18
+ **University at Buffalo, The State University of New York**
19
+
20
+ [![Conference](https://img.shields.io/badge/MIDL%202026-Oral%20Presentation-1f6feb)](https://2026.midl.io/scientific-program)
21
+ [![Oral Selection](https://img.shields.io/badge/Oral%20Selection-%3C8%25-success)](https://2026.midl.io/scientific-program)
22
+ [![Paper](https://img.shields.io/badge/Paper-arXiv%3A2604.10410-b31b1b?logo=arxiv&logoColor=white)](https://arxiv.org/abs/2604.10410)
23
+ [![Code](https://img.shields.io/badge/Code-GitHub-181717?logo=github&logoColor=white)](https://github.com/shantamsrivastava/CWCD)
24
+
25
+ CWCD was accepted as a full paper at **Medical Imaging with Deep Learning (MIDL) 2026** and selected for an **oral presentation** (oral selection rate below 8%).
26
+
27
+ ## πŸ“– Model Description
28
+
29
+ This repository contains the eight category-specific rank-1 LoRA adapters released with **CWCD: Category-Wise Contrastive Decoding for Structured Medical Report Generation**. CWCD generates a structured findings report through independent forward passes under eight anatomical categories.
30
+
31
+ At inference time, each adapter is used for both the normal X-ray and its corresponding category-masked X-ray. CWCD contrasts the resulting base and masked log-probabilities, applies Plausibility-Based Vocabulary Subselection, and greedily selects the next token. The eight category-wise findings are combined into the final structured findings report.
32
+
33
+ The complete inference implementation and usage instructions are available in the [CWCD GitHub repository](https://github.com/shantamsrivastava/CWCD).
34
+
35
+ ## πŸ€— Available Adapters
36
+
37
+ | Anatomical category | Adapter directory | LoRA rank | Target modules |
38
+ |---|---|---:|---|
39
+ | Lungs and Airways | `adapters/lungs` | 1 | `q_proj`, `v_proj` |
40
+ | Musculoskeletal and Chest Wall | `adapters/wall` | 1 | `q_proj`, `v_proj` |
41
+ | Cardiovascular | `adapters/cardiovascular` | 1 | `q_proj`, `v_proj` |
42
+ | Pleura | `adapters/pleura` | 1 | `q_proj`, `v_proj` |
43
+ | Hila and Mediastinum | `adapters/hila` | 1 | `q_proj`, `v_proj` |
44
+ | Tubes, Catheters, and Support Devices | `adapters/tubes` | 1 | `q_proj`, `v_proj` |
45
+ | Abdominal | `adapters/abdominal` | 1 | `q_proj`, `v_proj` |
46
+ | Other | `adapters/other` | 1 | `q_proj`, `v_proj` |
47
+
48
+ Each directory contains:
49
+
50
+ ```text
51
+ adapter_config.json
52
+ adapter_model.bin
53
+ ```
54
+
55
+ ## 🧠 CWCD Inference
56
+
57
+ For each anatomical category, the GitHub inference pipeline:
58
+
59
+ 1. loads the corresponding category-specific adapter;
60
+ 2. obtains the base distribution from the normal X-ray;
61
+ 3. obtains the masked distribution after blacking out that category's bounding boxes;
62
+ 4. contrasts the two distributions in log-probability space;
63
+ 5. applies Plausibility-Based Vocabulary Subselection;
64
+ 6. generates the category-wise findings and combines all eight sections.
65
+
66
+ The contrastive score is:
67
+
68
+ ```text
69
+ score(token) = (1 + alpha) * log P_base(token) - alpha * log P_masked(token)
70
+ ```
71
+
72
+ The released defaults are `alpha=1.0` and `beta=0.5`.
73
+
74
+ ## πŸš€ Using the Adapters
75
+
76
+ The adapters are downloaded automatically by [`inference.py`](https://github.com/shantamsrivastava/CWCD/blob/main/inference.py) through `huggingface_hub.snapshot_download()`. Only the adapter configuration and weight files are retrieved.
77
+
78
+ ```bash
79
+ git clone https://github.com/shantamsrivastava/CWCD.git
80
+ cd CWCD
81
+
82
+ python -m pip install -r requirements.txt
83
+ python -m pip install git+https://github.com/microsoft/LLaVA-Rad.git
84
+
85
+ python inference.py \
86
+ --image chest_xray.png \
87
+ --boxes boxes.example.json \
88
+ --alpha 1.0 \
89
+ --beta 0.5 \
90
+ --output report.json
91
+ ```
92
+
93
+ See the [GitHub README](https://github.com/shantamsrivastava/CWCD#readme) for environment setup, input format, standard category-wise inference, prompts, and all CLI options.
94
+
95
+ ## πŸ“¦ Repository Structure
96
 
97
  ```text
98
  adapters/
 
106
  └── wall/
107
  ```
108
 
109
+ ## ⚠️ Intended Use and Limitations
110
+
111
+ These adapters are intended strictly for research and educational use. CWCD has not been approved or validated for clinical or diagnostic deployment and must not be used to make medical decisions or inform patient care.
112
+
113
+ Generated findings can contain omissions, incorrect statements, or hallucinations. Performance depends on the base model, input image quality, category-specific bounding boxes, and the domain represented by the evaluation data. Users are responsible for protecting patient privacy and following all applicable data-use agreements.
114
 
115
+ ## 🀝 Acknowledgements
116
+
117
+ CWCD builds on [LLaVA-Rad](https://github.com/microsoft/LLaVA-Rad), [LLaVA](https://github.com/haotian-liu/LLaVA), [Vicuna](https://github.com/lm-sys/FastChat), and [PEFT](https://github.com/huggingface/peft).
118
+
119
+ ## πŸ“‘ Citation
120
+
121
+ ```bibtex
122
+ @inproceedings{srivastava2026cwcd,
123
+ title={CWCD: Category-Wise Contrastive Decoding for Structured Medical Report Generation},
124
+ author={Srivastava, Shantam and Bhosale, Mahesh and Doermann, David and Gao, Mingchen},
125
+ booktitle={Medical Imaging with Deep Learning},
126
+ series={Proceedings of Machine Learning Research},
127
+ volume={315},
128
+ pages={868--893},
129
+ year={2026},
130
+ publisher={PMLR}
131
+ }
132
+ ```