stasaking commited on
Commit
afedf52
·
verified ·
1 Parent(s): 993154e

Add full model card: usage, training data, performance, limitations, citation

Browse files
Files changed (1) hide show
  1. README.md +148 -0
README.md CHANGED
@@ -1,3 +1,151 @@
1
  ---
2
  license: cc-by-nc-sa-4.0
 
 
 
 
 
 
 
 
 
 
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  license: cc-by-nc-sa-4.0
3
+ library_name: pytorch
4
+ tags:
5
+ - image-regression
6
+ - vgg19_bn
7
+ - cognition
8
+ - dementia
9
+ - alzheimers-disease
10
+ - drawing
11
+ - pentagon-drawing-test
12
+ - medical-imaging
13
  ---
14
+
15
+ # PentaMind
16
+
17
+ PentaMind predicts **global cognition** from a scanned drawing of **intersecting pentagons** (the
18
+ pentagon copying item of the MMSE). It is an ensemble of 10 VGG19-bn models, trained on 13,777
19
+ drawings from 3,111 older adults in the ROS, MAP and MARS cohorts of the Rush Alzheimer's Disease
20
+ Center (RADC).
21
+
22
+ - **Code and usage:** [github.com/stasaki/PentaMind](https://github.com/stasaki/PentaMind)
23
+ - **Paper:** Tasaki S, Kim N, Truty T, Zhang A, Buchman AS, Lamar M, Bennett DA.
24
+ [Explainable deep learning approach for extracting cognitive features from hand-drawn images of intersecting pentagons](https://www.nature.com/articles/s41746-023-00904-w).
25
+ *npj Digital Medicine* 6, 157 (2023). doi:10.1038/s41746-023-00904-w
26
+
27
+ ## Model details
28
+
29
+ | | |
30
+ |---|---|
31
+ | Task | Regression: one image -> one number (predicted global cognition, higher = better) |
32
+ | Architecture | torchvision `vgg19_bn`, ImageNet start, `classifier[6]` replaced by `Linear(4096, 1)` |
33
+ | Files | `models/vgg19_bn-1_model.pth` ... `models/vgg19_bn-10_model.pth` (PyTorch `state_dict`, fp32, 558 MB each) |
34
+ | Ensemble | Average the 10 model outputs. The 10 models come from 10-fold cross-validation (each trained on 9/10 of the participants) |
35
+ | Input | RGB, 224 x 224, ImageNet normalization (mean 0.485/0.456/0.406, std 0.229/0.224/0.225) |
36
+ | Output scale | ROS/MAP/MARS global cognition: composite of 19 cognitive tests on a z-score scale (0 = about the cohort average) |
37
+ | Framework | Trained with PyTorch 1.10; loads with current PyTorch (tested with 2.x, CPU and GPU) |
38
+
39
+ ## How to use
40
+
41
+ The easiest way is the script in the GitHub repository. It downloads these weights, preprocesses
42
+ the images and averages the 10 models:
43
+
44
+ ```bash
45
+ git clone https://github.com/stasaki/PentaMind.git && cd PentaMind
46
+ pip install -r requirements.txt
47
+ python pentamind_predict.py --images my_images/ --out scores.csv # add --crop for whole-page scans
48
+ ```
49
+
50
+ Minimal Python example (one image, 10 models):
51
+
52
+ ```python
53
+ import torch, torchvision
54
+ from torch import nn
55
+ from PIL import Image
56
+ from huggingface_hub import hf_hub_download
57
+
58
+ net = torchvision.models.vgg19_bn(weights=None)
59
+ net.classifier[6] = nn.Linear(4096, 1)
60
+ net.eval()
61
+
62
+ prep = torchvision.transforms.Compose([
63
+ torchvision.transforms.Resize((224, 224)),
64
+ torchvision.transforms.ToTensor(),
65
+ torchvision.transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]),
66
+ ])
67
+ x = prep(Image.open("pentagon.png").convert("RGB")).unsqueeze(0) # drawing on white, square image
68
+
69
+ scores = []
70
+ for i in range(1, 11):
71
+ path = hf_hub_download("stasaking/pentamind", f"models/vgg19_bn-{i}_model.pth")
72
+ net.load_state_dict(torch.load(path, map_location="cpu", weights_only=True))
73
+ with torch.no_grad():
74
+ scores.append(net(x).item())
75
+ print("PentaMind score:", sum(scores) / len(scores))
76
+ ```
77
+
78
+ ## Input images
79
+
80
+ Training images were scanned pencil-and-paper drawings. The pentagon region was cut out, padded
81
+ with white to a 500 x 500 square and resized to 224 x 224. So the model expects **dark lines on a
82
+ white background, the drawing centered and filling most of a square image, and nothing else**
83
+ (no text, page borders or signatures). The GitHub script's `--crop` option does this for
84
+ whole-page scans.
85
+
86
+ In the paper, each image was also scored 30 times with small random changes (test-time
87
+ augmentation) and the results were averaged. Use `--n_tta 30` in the script to do the same.
88
+
89
+ ## Training data
90
+
91
+ - 13,777 drawings from 3,111 participants: ROS (1,093 participants), MAP (1,508), MARS (510).
92
+ - Label: global cognition at the same research visit as the drawing (19 tests; the MMSE and the pentagon item are
93
+ not part of it).
94
+ - Split by participant: 80% training, 10% validation, 10% test in model selection; 10-fold
95
+ cross-validation for the 10 released models.
96
+ - The data are not public. They can be requested from the
97
+ [RADC Resource Sharing Hub](https://www.radc.rush.edu/) under a data use agreement.
98
+
99
+ ## Performance (from the paper)
100
+
101
+ - Test set: Spearman correlation 0.44 with global cognition.
102
+ - Out-of-fold: explained 23.3% of the variance in global cognition, 1.9 times more than the usual
103
+ pass/fail rating of the drawing.
104
+ - Separating dementia from no cognitive impairment: AUC 0.77 (pass/fail rating: 0.67).
105
+ - The score was also related to motor function and to several brain pathologies.
106
+
107
+ ## Intended use
108
+
109
+ - Research on cognition and drawing, for example as a cognitive variable in studies of aging, or as
110
+ a feature extractor for other drawing tasks.
111
+ - **Not** for clinical diagnosis or decisions about individuals.
112
+
113
+ ## Limitations and cautions
114
+
115
+ - **Data leakage on RADC data.** Every ROS/MAP/MARS drawing in the training data was used to train 9
116
+ of the 10 models. Scores on those drawings will look better than they really are. Do not use them
117
+ to measure performance.
118
+ - **Training population:** older adults in the US, mostly White (MARS: African American), one test
119
+ form and scanning setup. Performance in other populations, forms or scanners is unknown.
120
+ - **Other drawings** (clocks, cubes, etc.) are out of domain. The score on them is usually only weakly
121
+ related to cognition. Training a simple model (for example ridge regression) on the image features
122
+ (see `--features` in the GitHub script) on your own labels usually works better.
123
+ - **Digital pen data:** the model only sees the final image (no timing, pressure or stroke order).
124
+ Render strokes as black lines on white, with a realistic line width (very thin lines lowered the
125
+ score in the paper's simulation), and crop to the drawing.
126
+ - **Modest accuracy:** the score explains about a quarter of the variance in cognition and is pulled
127
+ toward the average.
128
+
129
+ ## License
130
+
131
+ The weights are licensed under [CC BY-NC-SA 4.0](https://creativecommons.org/licenses/by-nc-sa/4.0/)
132
+ (attribution, non-commercial, share-alike), because they were trained on ROS/MAP/MARS data. The code
133
+ on GitHub is BSD 3-Clause.
134
+
135
+ ## Citation
136
+
137
+ ```bibtex
138
+ @article{tasaki2023pentamind,
139
+ title = {Explainable deep learning approach for extracting cognitive features from hand-drawn images of intersecting pentagons},
140
+ author = {Tasaki, Shinya and Kim, Namhee and Truty, Tim and Zhang, Ada and Buchman, Aron S. and Lamar, Melissa and Bennett, David A.},
141
+ journal = {npj Digital Medicine},
142
+ volume = {6},
143
+ pages = {157},
144
+ year = {2023},
145
+ doi = {10.1038/s41746-023-00904-w}
146
+ }
147
+ ```
148
+
149
+ ## Acknowledgments
150
+
151
+ We thank all the participants of the ROS, MAP and MARS studies.