Tata24
/

Tata24 ianpan commited on
Commit
00b1fc6
·
0 Parent(s):

Duplicate from ianpan/bone-age

Browse files

Co-authored-by: Ian Pan <ianpan@users.noreply.huggingface.co>

Files changed (7) hide show
  1. .gitattributes +35 -0
  2. README.md +122 -0
  3. config.json +18 -0
  4. configuration.py +23 -0
  5. model.safetensors +3 -0
  6. modeling.py +133 -0
  7. ref_img.png +0 -0
.gitattributes ADDED
@@ -0,0 +1,35 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ *.7z filter=lfs diff=lfs merge=lfs -text
2
+ *.arrow filter=lfs diff=lfs merge=lfs -text
3
+ *.bin filter=lfs diff=lfs merge=lfs -text
4
+ *.bz2 filter=lfs diff=lfs merge=lfs -text
5
+ *.ckpt filter=lfs diff=lfs merge=lfs -text
6
+ *.ftz filter=lfs diff=lfs merge=lfs -text
7
+ *.gz filter=lfs diff=lfs merge=lfs -text
8
+ *.h5 filter=lfs diff=lfs merge=lfs -text
9
+ *.joblib filter=lfs diff=lfs merge=lfs -text
10
+ *.lfs.* filter=lfs diff=lfs merge=lfs -text
11
+ *.mlmodel filter=lfs diff=lfs merge=lfs -text
12
+ *.model filter=lfs diff=lfs merge=lfs -text
13
+ *.msgpack filter=lfs diff=lfs merge=lfs -text
14
+ *.npy filter=lfs diff=lfs merge=lfs -text
15
+ *.npz filter=lfs diff=lfs merge=lfs -text
16
+ *.onnx filter=lfs diff=lfs merge=lfs -text
17
+ *.ot filter=lfs diff=lfs merge=lfs -text
18
+ *.parquet filter=lfs diff=lfs merge=lfs -text
19
+ *.pb filter=lfs diff=lfs merge=lfs -text
20
+ *.pickle filter=lfs diff=lfs merge=lfs -text
21
+ *.pkl filter=lfs diff=lfs merge=lfs -text
22
+ *.pt filter=lfs diff=lfs merge=lfs -text
23
+ *.pth filter=lfs diff=lfs merge=lfs -text
24
+ *.rar filter=lfs diff=lfs merge=lfs -text
25
+ *.safetensors filter=lfs diff=lfs merge=lfs -text
26
+ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
27
+ *.tar.* filter=lfs diff=lfs merge=lfs -text
28
+ *.tar filter=lfs diff=lfs merge=lfs -text
29
+ *.tflite filter=lfs diff=lfs merge=lfs -text
30
+ *.tgz filter=lfs diff=lfs merge=lfs -text
31
+ *.wasm filter=lfs diff=lfs merge=lfs -text
32
+ *.xz filter=lfs diff=lfs merge=lfs -text
33
+ *.zip filter=lfs diff=lfs merge=lfs -text
34
+ *.zst filter=lfs diff=lfs merge=lfs -text
35
+ *tfevents* filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,122 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ library_name: transformers
3
+ tags:
4
+ - radiology
5
+ - medical_imaging
6
+ - bone_age
7
+ - x_ray
8
+ license: apache-2.0
9
+ base_model:
10
+ - timm/convnextv2_tiny.fcmae_ft_in22k_in1k
11
+ pipeline_tag: image-classification
12
+ ---
13
+
14
+ **Training code:** [bone-age-prediction](https://github.com/i-pan/bone-age-prediction) — data preparation, training, evaluation, and Hugging Face export.
15
+
16
+
17
+ This model has been trained and validated on 14,036 pediatric hand radiographs from the [RSNA Pediatric Bone Age Challenge](https://www.rsna.org/rsnai/ai-image-challenge/rsna-pediatric-bone-age-challenge-2017) dataset, which is publicly available.
18
+ It can be loaded using:
19
+ ```
20
+ from transformers import AutoModel
21
+
22
+ model = AutoModel.from_pretrained("ianpan/bone-age", trust_remote_code=True)
23
+ ```
24
+ The model is a 3-fold ensemble utilizing the `convnextv2_tiny` backbone.
25
+ The individual single-fold models can be accessed through `model.net0`, `model.net1`, `model.net2`. Each of these models was trained over 20,000 iterations using a batch size of 64 across 2 NVIDIA RTX 3090 GPUs.
26
+ Originally, it was trained with both a regression and classification head.
27
+ However, this model only loads the classification head, as stand-alone performance was slightly better. The classification head also generates better GradCAMs.
28
+ The softmax function is applied to the output logits and multiplied by the corresponding class indices, then summed.
29
+ This outputs a scalar float value representing the predicted bone age in units of months.
30
+
31
+ In addition to standard data augmentation, additional augmentations were also applied:
32
+ - Using a cropped radiograph (from the model <https://huggingface.co/ianpan/bone-age-crop>) with probability 0.5
33
+ - Histogram matching with a reference image (available in this repo under Files, `ref_img.png`) with probability 0.5
34
+
35
+ Note that both of the above augmentations could be applied simultaneously and in conjunction with standard data augamentations. Thus, the model accommodates a large range of variability in the appearance of a hand radiograph.
36
+
37
+ On the original challenge test set comprising 200 multi-annotated pediatric hand radiographs, this model achieves a **mean absolute error of 4.16 months** (when applying both cropping and histogram matching to the input radiograph), which surpasses the [top solutions](https://pubs.rsna.org/doi/10.1148/radiol.2018180736) from the original challenge.
38
+ Specific results as follows, with single model performance using `model.net0` in brackets:
39
+ ```
40
+ Crop (-) / Histogram Matching (-): 4.42 [4.67] months
41
+ Crop (+) / Histogram Matching (-): 4.47 [4.84] months
42
+ Crop (-) / Histogram Matching (+): 4.34 [4.59] months
43
+ Crop (+) / Histogram Matching (+): 4.16 [4.45] months
44
+ ```
45
+
46
+ Thus it is preferable to both crop and histogram match the image to obtain the optimal results. See <https://huggingface.co/ianpan/bone-age-crop> for how to crop a bone age radiograph with a pretrained model.
47
+ To histogram match with a reference image:
48
+ ```
49
+ import cv2
50
+ from skimage.exposure import match_histograms
51
+
52
+ x = cv2.imread("target_radiograph.png", 0)
53
+ ref = cv2.imread("ref_img.png", 0) # download ref_img.png from this repo
54
+ x = match_histograms(x, ref)
55
+ ```
56
+
57
+ Patient sex is an important variable affecting the model's prediction. This is passed to the model's `forward()` function using the `female` argument:
58
+ ```
59
+ # 1 indicates female, 0 male
60
+ model(x, female=torch.tensor([1, 0, 1, 0])) # assuming batch size of 4
61
+ ```
62
+
63
+ Example usage for a single image:
64
+ ```
65
+ import cv2
66
+ import torch
67
+ from skimage.exposure import match_histograms
68
+ from transformers import AutoModel
69
+
70
+ device = "cuda" if torch.cuda.is_available() else "cpu"
71
+
72
+ crop_model = AutoModel.from_pretrained("ianpan/bone-age-crop", trust_remote_code=True)
73
+ crop_model = crop_model.eval().to(device)
74
+ img = cv2.imread(..., 0)
75
+ img_shape = torch.tensor([img.shape[:2]])
76
+ x = crop_model.preprocess(img) # only takes single image as input
77
+ x = torch.from_numpy(x).unsqueeze(0).unsqueeze(0) # add channel, batch dims
78
+ x = x.float()
79
+
80
+ # if you do not provide img_shape
81
+ # model will return normalized coordinates
82
+ with torch.inference_mode():
83
+ coords = model(x.to(device), img_shape.to(device))
84
+
85
+ # only 1 sample in batch
86
+ coords = coords[0].cpu().numpy()
87
+ x, y, w, h = coords
88
+ # coords already rescaled with img_shape
89
+ cropped_img = img[y: y + h, x: x + w]
90
+
91
+ # histogram matching
92
+ ref = cv2.imread("ref_img.png", 0) # download ref_img.png from this repo
93
+ cropped_img = match_histograms(cropped_img, ref)
94
+
95
+ model = AutoModel.from_pretrained("ianpan/bone-age", trust_remote_code=True)
96
+ model = model.eval().to(device)
97
+ x = model.preprocess(cropped_img)
98
+ x = torch.from_numpy(x).unsqueeze(0).unsqueeze(0)
99
+ x = x.float()
100
+ female = torch.tensor([1])
101
+
102
+ with torch.inference_mode():
103
+ bone_age = model(x.to(device), female.to(device))
104
+ ```
105
+
106
+ If you want the raw logits (class `i` = `i` months), you can pass `return_logits=True` to `forward()`:
107
+ ```
108
+ bone_age_logits = model(x, female, return_logits=True)
109
+ ```
110
+
111
+ To run single model inference, simply access one of the nets:
112
+ ```
113
+ bone_age = model.net0(x, female)
114
+ ```
115
+
116
+ If you have `pydicom` installed, you can also load a DICOM image directly:
117
+ ```
118
+ img = model.load_image_from_dicom(path_to_dicom)
119
+ ```
120
+
121
+ This model is for demonstration and research purposes only and has NOT been approved by any regulatory agency for clinical use.
122
+ The user assumes any and all responsibility regarding their own use of this model and its outputs.
config.json ADDED
@@ -0,0 +1,18 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "BoneAgeEnsembleModel"
4
+ ],
5
+ "auto_map": {
6
+ "AutoConfig": "configuration.BoneAgeConfig",
7
+ "AutoModel": "modeling.BoneAgeEnsembleModel"
8
+ },
9
+ "backbone": "convnextv2_tiny",
10
+ "dropout": 0.1,
11
+ "feature_dim": 768,
12
+ "in_chans": 2,
13
+ "model_type": "bone_age",
14
+ "num_classes": 240,
15
+ "num_models": 3,
16
+ "torch_dtype": "float32",
17
+ "transformers_version": "4.47.0"
18
+ }
configuration.py ADDED
@@ -0,0 +1,23 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ from transformers import PretrainedConfig
2
+
3
+
4
+ class BoneAgeConfig(PretrainedConfig):
5
+ model_type = "bone_age"
6
+
7
+ def __init__(
8
+ self,
9
+ backbone="convnextv2_tiny",
10
+ feature_dim=768,
11
+ dropout=0.1,
12
+ num_classes=240,
13
+ in_chans=2,
14
+ num_models=3,
15
+ **kwargs
16
+ ):
17
+ self.backbone = backbone
18
+ self.feature_dim = feature_dim
19
+ self.dropout = dropout
20
+ self.num_classes = num_classes
21
+ self.in_chans = in_chans
22
+ self.num_models = num_models
23
+ super().__init__(**kwargs)
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2ac6cd24258802e23a9233f824f4670de4aece4523681511a03d0841a06cc2fc
3
+ size 336662916
modeling.py ADDED
@@ -0,0 +1,133 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ import albumentations as A
2
+ import cv2
3
+ import torch
4
+ import torch.nn as nn
5
+ import torch.nn.functional as F
6
+
7
+ from numpy.typing import NDArray
8
+ from transformers import PreTrainedModel
9
+ from timm import create_model
10
+ from typing import Optional
11
+ from .configuration import BoneAgeConfig
12
+
13
+
14
+ class GeM(nn.Module):
15
+ def __init__(
16
+ self, p: int = 3, eps: float = 1e-6, dim: int = 2, flatten: bool = True
17
+ ):
18
+ super().__init__()
19
+ self.p = nn.Parameter(torch.ones(1) * p)
20
+ self.eps = eps
21
+ assert dim in {2, 3}, f"dim must be one of [2, 3], not {dim}"
22
+ self.dim = dim
23
+ if self.dim == 2:
24
+ self.func = F.adaptive_avg_pool2d
25
+ elif self.dim == 3:
26
+ self.func = F.adaptive_avg_pool3d
27
+ self.flatten = nn.Flatten(1) if flatten else nn.Identity()
28
+
29
+ def forward(self, x: torch.Tensor) -> torch.Tensor:
30
+ # assumes x.shape is (n, c, [t], h, w)
31
+ x = self.func(x.clamp(min=self.eps).pow(self.p), output_size=1).pow(
32
+ 1.0 / self.p
33
+ )
34
+ return self.flatten(x)
35
+
36
+
37
+ class BoneAgeModel(nn.Module):
38
+ def __init__(
39
+ self, backbone, feature_dim=768, dropout=0.1, num_classes=240, in_chans=2
40
+ ):
41
+ super().__init__()
42
+ self.backbone = create_model(
43
+ model_name=backbone,
44
+ pretrained=False,
45
+ num_classes=0,
46
+ global_pool="",
47
+ features_only=False,
48
+ in_chans=in_chans,
49
+ )
50
+ self.pooling = GeM(p=3, dim=2)
51
+ self.dropout = nn.Dropout(p=dropout)
52
+ self.linear = nn.Linear(feature_dim, num_classes)
53
+
54
+ def normalize(self, x: torch.Tensor) -> torch.Tensor:
55
+ # [0, 255] -> [-1, 1]
56
+ mini, maxi = 0.0, 255.0
57
+ x = (x - mini) / (maxi - mini)
58
+ x = (x - 0.5) * 2.0
59
+ return x
60
+
61
+ def forward(
62
+ self, x: torch.Tensor, female: torch.Tensor, return_logits: bool = False
63
+ ) -> torch.Tensor:
64
+ assert x.size(0) == female.size(
65
+ 0
66
+ ), f"x.size(0) [{x.size(0)}] must equal female.size(0) [{female.size(0)}]"
67
+ female_ch = torch.zeros_like(x).to(x.device)
68
+ female_ch[female.bool()] = 255.0
69
+ x = torch.cat([x, female_ch], dim=1)
70
+ x = self.normalize(x)
71
+ features = self.pooling(self.backbone(x))
72
+ logits = self.linear(features)
73
+ if return_logits:
74
+ return logits
75
+ out = (logits.softmax(1) * torch.arange(logits.size(1)).to(logits.device)).sum(
76
+ 1
77
+ )
78
+ return out
79
+
80
+
81
+ class BoneAgeEnsembleModel(PreTrainedModel):
82
+ config_class = BoneAgeConfig
83
+
84
+ def __init__(self, config):
85
+ super().__init__(config)
86
+ self.num_models = config.num_models
87
+ for i in range(self.num_models):
88
+ setattr(
89
+ self,
90
+ f"net{i}",
91
+ BoneAgeModel(
92
+ config.backbone,
93
+ config.feature_dim,
94
+ config.dropout,
95
+ config.num_classes,
96
+ config.in_chans,
97
+ ),
98
+ )
99
+
100
+ @staticmethod
101
+ def load_image_from_dicom(path: str) -> Optional[NDArray]:
102
+ try:
103
+ from pydicom import dcmread
104
+ from pydicom.pixels import apply_voi_lut
105
+ except ModuleNotFoundError:
106
+ print("`pydicom` is not installed, returning None ...")
107
+ return None
108
+ dicom = dcmread(path)
109
+ arr = apply_voi_lut(dicom.pixel_array, dicom)
110
+ if dicom.PhotometricInterpretation == "MONOCHROME1":
111
+ arr = arr.max() - arr
112
+
113
+ arr = arr - arr.min()
114
+ arr = arr / arr.max()
115
+ arr = (arr * 255).astype("uint8")
116
+ return arr
117
+
118
+ @staticmethod
119
+ def preprocess(x: NDArray) -> NDArray:
120
+ x = A.LongestMaxSize(max_size=512, p=1)(image=x)["image"]
121
+ x = A.PadIfNeeded(512, 512, border_mode=cv2.BORDER_CONSTANT, p=1)(image=x)[
122
+ "image"
123
+ ]
124
+ return x
125
+
126
+ def forward(
127
+ self, x: torch.Tensor, female: torch.Tensor, return_logits: bool = False
128
+ ) -> torch.Tensor:
129
+ out = []
130
+ for i in range(self.num_models):
131
+ model = getattr(self, f"net{i}")
132
+ out.append(model(x, female, return_logits))
133
+ return torch.stack(out).mean(0)
ref_img.png ADDED