Upload 17 files
Browse filesupload Model Files
- .gitattributes +2 -0
- .gitignore +7 -0
- README.md +111 -3
- best_multitask_vision_model.pth +3 -0
- download.jpg +3 -0
- models/mobilenet_v3_large-8738ca79.pth +3 -0
- mrbean_tf_model/mrbean_vision_float16.tflite +3 -0
- mrbean_tf_model/mrbean_vision_float32.tflite +3 -0
- mrbean_tf_model/mrbean_vision_tensor_correspondence_report.json +0 -0
- mrbean_tf_model/schema.fbs +1715 -0
- mrbean_tf_model/schema_generated.py +0 -0
- mrbean_vision.onnx +3 -0
- mrbean_vision.onnx.data +3 -0
- pthToONNX.py +48 -0
- requirements.txt +10 -0
- test.py +55 -0
- test_tflite.py +53 -0
- train.py +217 -0
.gitattributes
CHANGED
|
@@ -33,3 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
|
|
|
| 33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
| 36 |
+
download.jpg filter=lfs diff=lfs merge=lfs -text
|
| 37 |
+
mrbean_vision.onnx.data filter=lfs diff=lfs merge=lfs -text
|
.gitignore
ADDED
|
@@ -0,0 +1,7 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
dataset/
|
| 2 |
+
.idea/
|
| 3 |
+
download.jpg
|
| 4 |
+
.venv_tf/
|
| 5 |
+
test.ipynb
|
| 6 |
+
.venv/
|
| 7 |
+
test.py
|
README.md
CHANGED
|
@@ -1,3 +1,111 @@
|
|
| 1 |
-
|
| 2 |
-
|
| 3 |
-
--
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# MRBEAN Vision Engine
|
| 2 |
+
|
| 3 |
+
This repository contains the offline multi-task computer vision model for **Project MRBEAN** (Mobile Resilient Broadcast for Emergency Ad-hoc Networks).
|
| 4 |
+
|
| 5 |
+
Currently in its initial development phase, this lightweight model is designed to run entirely offline on Android edge devices to analyze live camera feeds, identifying both the type of disaster and the severity of the damage simultaneously.
|
| 6 |
+
|
| 7 |
+
|
| 8 |
+
---
|
| 9 |
+
|
| 10 |
+
# Dataset
|
| 11 |
+
|
| 12 |
+
|
| 13 |
+
Link: https://crisisnlp.qcri.org/medic/#:~:text=CrisisNLP,The%20dataset%20contains%2071%2C198%20images.
|
| 14 |
+
|
| 15 |
+
|
| 16 |
+
---
|
| 17 |
+
|
| 18 |
+
## Model Architecture
|
| 19 |
+
|
| 20 |
+
The vision engine utilizes a shared **MobileNetV3 (Large)** backbone, splitting into two distinct dense classification heads:
|
| 21 |
+
* **Backbone:** MobileNetV3 (Pre-trained on ImageNet, fine-tuned on the MEDIC dataset)
|
| 22 |
+
* **Head 1 (Disaster Type):** 6 Output Classes
|
| 23 |
+
* **Head 2 (Damage Severity):** 3 Output Classes
|
| 24 |
+
|
| 25 |
+
---
|
| 26 |
+
|
| 27 |
+
## Supported Classification Classes
|
| 28 |
+
|
| 29 |
+
**Disaster Types (Head 1)**
|
| 30 |
+
* 0 = Earthquake
|
| 31 |
+
* 1 = Fire
|
| 32 |
+
* 2 = Flood
|
| 33 |
+
* 3 = Hurricane
|
| 34 |
+
* 4 = Landslide
|
| 35 |
+
* 5 = Not a Disaster
|
| 36 |
+
|
| 37 |
+
**Damage Severity (Head 2)**
|
| 38 |
+
* 0 = Little or None
|
| 39 |
+
* 1 = Mild
|
| 40 |
+
* 2 = Severe
|
| 41 |
+
|
| 42 |
+
---
|
| 43 |
+
|
| 44 |
+
## Exported Model Formats
|
| 45 |
+
|
| 46 |
+
To support cross-platform development and strict mobile deployment, the model has been exported into the following formats:
|
| 47 |
+
|
| 48 |
+
* `best_multitask_vision_model.pth`: The raw PyTorch weights for continued training or server-side inference.
|
| 49 |
+
* `mrbean_tf_model/mrbean_vision_float32.tflite` (12.7 MB): Standard TensorFlow Lite model for high-accuracy Android deployment.
|
| 50 |
+
* `mrbean_tf_model/mrbean_vision_float16.tflite` (6.39 MB): Half-precision compressed model to minimize the Flutter application bundle size.
|
| 51 |
+
|
| 52 |
+
---
|
| 53 |
+
|
| 54 |
+
## Input Specifications
|
| 55 |
+
|
| 56 |
+
The model expects images preprocessed to match the PyTorch standard:
|
| 57 |
+
* **Resolution:** 224x224 pixels
|
| 58 |
+
* **Channels:** 3 (RGB)
|
| 59 |
+
* **Normalization:** ImageNet standard (`mean=[0.485, 0.456, 0.406]`, `std=[0.229, 0.224, 0.225]`)
|
| 60 |
+
* **TFLite Input Shape:** `[1, 224, 224, 3]` (NHWC format)
|
| 61 |
+
|
| 62 |
+
---
|
| 63 |
+
|
| 64 |
+
## Python Testing (TFLite)
|
| 65 |
+
|
| 66 |
+
To test the mobile-ready `.tflite` files on a PC before integrating them into the Flutter frontend, use the following snippet. Note: Testing the `float16` model on a PC requires bypassing the XNNPACK engine.
|
| 67 |
+
|
| 68 |
+
```python
|
| 69 |
+
import os
|
| 70 |
+
os.environ["TF_LITE_DISABLE_XNNPACK"] = "1"
|
| 71 |
+
|
| 72 |
+
import numpy as np
|
| 73 |
+
import tensorflow as tf
|
| 74 |
+
from PIL import Image
|
| 75 |
+
|
| 76 |
+
# 1. Load Model
|
| 77 |
+
model_path = r"mrbean_tf_model\mrbean_vision_float16.tflite"
|
| 78 |
+
interpreter = tf.lite.Interpreter(model_path=model_path)
|
| 79 |
+
interpreter.allocate_tensors()
|
| 80 |
+
|
| 81 |
+
input_details = interpreter.get_input_details()[0]
|
| 82 |
+
output_details = interpreter.get_output_details()
|
| 83 |
+
|
| 84 |
+
# 2. Preprocess Image
|
| 85 |
+
def preprocess_image(image_path, expected_shape):
|
| 86 |
+
img = Image.open(image_path).convert('RGB').resize((224, 224))
|
| 87 |
+
img_array = np.array(img, dtype=np.float32) / 255.0
|
| 88 |
+
|
| 89 |
+
mean = np.array([0.485, 0.456, 0.406], dtype=np.float32)
|
| 90 |
+
std = np.array([0.229, 0.224, 0.225], dtype=np.float32)
|
| 91 |
+
img_array = (img_array - mean) / std
|
| 92 |
+
|
| 93 |
+
if expected_shape[1] == 3: # NCHW fallback
|
| 94 |
+
img_array = np.transpose(img_array, (2, 0, 1))
|
| 95 |
+
|
| 96 |
+
return np.expand_dims(img_array, axis=0)
|
| 97 |
+
|
| 98 |
+
# 3. Inference
|
| 99 |
+
input_data = preprocess_image("test_image.jpg", input_details['shape']).astype(input_details['dtype'])
|
| 100 |
+
interpreter.set_tensor(input_details['index'], input_data)
|
| 101 |
+
interpreter.invoke()
|
| 102 |
+
|
| 103 |
+
# 4. Extract Results
|
| 104 |
+
out_0 = interpreter.get_tensor(output_details[0]['index'])[0]
|
| 105 |
+
out_1 = interpreter.get_tensor(output_details[1]['index'])[0]
|
| 106 |
+
|
| 107 |
+
pred_disaster = np.argmax(out_0) if len(out_0) == 6 else np.argmax(out_1)
|
| 108 |
+
pred_severity = np.argmax(out_1) if len(out_0) == 6 else np.argmax(out_0)
|
| 109 |
+
|
| 110 |
+
print(f"Disaster Class ID: {pred_disaster}")
|
| 111 |
+
print(f"Severity Class ID: {pred_severity}")
|
best_multitask_vision_model.pth
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:1682ea6311d6bb95d3e48f581eebbac52292656d302c2eb5c3b81ffd9b510060
|
| 3 |
+
size 13585071
|
download.jpg
ADDED
|
Git LFS Details
|
models/mobilenet_v3_large-8738ca79.pth
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:8738ca797c879b547d18bbd15da5736ff2557b2036a9af72225393ca61759a04
|
| 3 |
+
size 22139423
|
mrbean_tf_model/mrbean_vision_float16.tflite
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:b56ba48a90db314a12b776d23855f36cbcf6b9675f1edb516f0e4f3911a5e938
|
| 3 |
+
size 6704452
|
mrbean_tf_model/mrbean_vision_float32.tflite
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:e781e3a82921d955d51ecfeae64a783fa8aa4ff62e64b2efe3f99ad43d6c7075
|
| 3 |
+
size 13364672
|
mrbean_tf_model/mrbean_vision_tensor_correspondence_report.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
mrbean_tf_model/schema.fbs
ADDED
|
|
mrbean_tf_model/schema_generated.py
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
mrbean_vision.onnx
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:93dc4dbdb093c70d477c89fd688f41cacb036c0a01bb0c6e46e312ca77b408a2
|
| 3 |
+
size 13351699
|
mrbean_vision.onnx.data
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:1d968c064a213f66248058842c6262baefba0dd944f4f2443edd71f80472b535
|
| 3 |
+
size 13321056
|
pthToONNX.py
ADDED
|
@@ -0,0 +1,48 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import torch
|
| 2 |
+
import onnxscript
|
| 3 |
+
import torch.nn as nn
|
| 4 |
+
import torchvision.models as models
|
| 5 |
+
|
| 6 |
+
class MRBEANVisionModel(nn.Module):
|
| 7 |
+
def __init__(self, num_disaster_classes=6, num_severity_classes=3):
|
| 8 |
+
super(MRBEANVisionModel, self).__init__()
|
| 9 |
+
mobilenet = models.mobilenet_v3_large(weights=None)
|
| 10 |
+
self.features = mobilenet.features
|
| 11 |
+
self.pool = nn.AdaptiveAvgPool2d(1)
|
| 12 |
+
self.flatten = nn.Flatten()
|
| 13 |
+
|
| 14 |
+
self.disaster_head = nn.Sequential(
|
| 15 |
+
nn.Linear(960, 256), nn.ReLU(), nn.Dropout(0.3), nn.Linear(256, num_disaster_classes)
|
| 16 |
+
)
|
| 17 |
+
self.severity_head = nn.Sequential(
|
| 18 |
+
nn.Linear(960, 128), nn.ReLU(), nn.Dropout(0.3), nn.Linear(128, num_severity_classes)
|
| 19 |
+
)
|
| 20 |
+
|
| 21 |
+
def forward(self, x):
|
| 22 |
+
x = self.features(x)
|
| 23 |
+
x = self.pool(x)
|
| 24 |
+
x = self.flatten(x)
|
| 25 |
+
return self.disaster_head(x), self.severity_head(x)
|
| 26 |
+
|
| 27 |
+
device = torch.device("cpu")
|
| 28 |
+
model = MRBEANVisionModel()
|
| 29 |
+
model.load_state_dict(torch.load("best_multitask_vision_model.pth", map_location=device))
|
| 30 |
+
|
| 31 |
+
model.eval()
|
| 32 |
+
|
| 33 |
+
dummy_input = torch.randn(1, 3, 224, 224)
|
| 34 |
+
onnx_filename = "mrbean_vision.onnx"
|
| 35 |
+
|
| 36 |
+
print(f"Tracing architecture and exporting to {onnx_filename}:")
|
| 37 |
+
torch.onnx.export(
|
| 38 |
+
model,
|
| 39 |
+
dummy_input,
|
| 40 |
+
onnx_filename,
|
| 41 |
+
export_params=True,
|
| 42 |
+
opset_version=17,
|
| 43 |
+
do_constant_folding=True,
|
| 44 |
+
input_names=['input_image'],
|
| 45 |
+
output_names=['disaster_output', 'severity_output']
|
| 46 |
+
)
|
| 47 |
+
|
| 48 |
+
print("\nSuccess! Your model is now framework-independent.")
|
requirements.txt
ADDED
|
@@ -0,0 +1,10 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
--extra-index-url https://download.pytorch.org/whl/cu124
|
| 2 |
+
|
| 3 |
+
pandas
|
| 4 |
+
numpy
|
| 5 |
+
matplotlib
|
| 6 |
+
torch
|
| 7 |
+
torchvision
|
| 8 |
+
onnxscript
|
| 9 |
+
tensorflow
|
| 10 |
+
onnx-tf
|
test.py
ADDED
|
@@ -0,0 +1,55 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import torch
|
| 2 |
+
import torchvision.transforms as transforms
|
| 3 |
+
from PIL import Image
|
| 4 |
+
import torch.nn as nn
|
| 5 |
+
import torchvision.models as models
|
| 6 |
+
|
| 7 |
+
class MRBEANVisionModel(nn.Module):
|
| 8 |
+
def __init__(self, num_disaster_classes=6, num_severity_classes=3):
|
| 9 |
+
super(MRBEANVisionModel, self).__init__()
|
| 10 |
+
mobilenet = models.mobilenet_v3_large(weights=None)
|
| 11 |
+
self.features = mobilenet.features
|
| 12 |
+
self.pool = nn.AdaptiveAvgPool2d(1)
|
| 13 |
+
self.flatten = nn.Flatten()
|
| 14 |
+
|
| 15 |
+
self.disaster_head = nn.Sequential(
|
| 16 |
+
nn.Linear(960, 256), nn.ReLU(), nn.Dropout(0.3), nn.Linear(256, num_disaster_classes)
|
| 17 |
+
)
|
| 18 |
+
self.severity_head = nn.Sequential(
|
| 19 |
+
nn.Linear(960, 128), nn.ReLU(), nn.Dropout(0.3), nn.Linear(128, num_severity_classes)
|
| 20 |
+
)
|
| 21 |
+
|
| 22 |
+
def forward(self, x):
|
| 23 |
+
x = self.features(x)
|
| 24 |
+
x = self.pool(x)
|
| 25 |
+
x = self.flatten(x)
|
| 26 |
+
return self.disaster_head(x), self.severity_head(x)
|
| 27 |
+
|
| 28 |
+
disaster_labels = {0: 'Earthquake', 1: 'Fire', 2: 'Flood', 3: 'Hurricane', 4: 'Landslide', 5: 'Not a Disaster'}
|
| 29 |
+
severity_labels = {0: 'Little or None', 1: 'Mild', 2: 'Severe'}
|
| 30 |
+
|
| 31 |
+
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
|
| 32 |
+
model = MRBEANVisionModel().to(device)
|
| 33 |
+
|
| 34 |
+
print("Loading trained weights...")
|
| 35 |
+
model.load_state_dict(torch.load("best_multitask_vision_model.pth", map_location=device))
|
| 36 |
+
model.eval()
|
| 37 |
+
|
| 38 |
+
transform = transforms.Compose([
|
| 39 |
+
transforms.Resize((224, 224)),
|
| 40 |
+
transforms.ToTensor(),
|
| 41 |
+
transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
|
| 42 |
+
])
|
| 43 |
+
|
| 44 |
+
test_image_path = "download.jpg"
|
| 45 |
+
image = Image.open(test_image_path).convert('RGB')
|
| 46 |
+
input_tensor = transform(image).unsqueeze(0).to(device)
|
| 47 |
+
|
| 48 |
+
with torch.no_grad():
|
| 49 |
+
out_disaster, out_severity = model(input_tensor)
|
| 50 |
+
_, pred_d = torch.max(out_disaster, 1)
|
| 51 |
+
_, pred_s = torch.max(out_severity, 1)
|
| 52 |
+
|
| 53 |
+
print("\n--- MRBEAN ANALYSIS RESULTS ---")
|
| 54 |
+
print(f"Disaster Type: {disaster_labels[pred_d.item()]}")
|
| 55 |
+
print(f"Damage Severity: {severity_labels[pred_s.item()]}")
|
test_tflite.py
ADDED
|
@@ -0,0 +1,53 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import numpy as np
|
| 2 |
+
import tensorflow as tf
|
| 3 |
+
from PIL import Image
|
| 4 |
+
|
| 5 |
+
model_path = r"mrbean_tf_model\mrbean_vision_float32.tflite"
|
| 6 |
+
print(f"Loading {model_path}...")
|
| 7 |
+
|
| 8 |
+
interpreter = tf.lite.Interpreter(model_path=model_path)
|
| 9 |
+
interpreter.allocate_tensors()
|
| 10 |
+
|
| 11 |
+
input_details = interpreter.get_input_details()[0]
|
| 12 |
+
output_details = interpreter.get_output_details()
|
| 13 |
+
|
| 14 |
+
disaster_labels = {0: 'Earthquake', 1: 'Fire', 2: 'Flood', 3: 'Hurricane', 4: 'Landslide', 5: 'Not a Disaster'}
|
| 15 |
+
severity_labels = {0: 'Little or None', 1: 'Mild', 2: 'Severe'}
|
| 16 |
+
|
| 17 |
+
def preprocess_image(image_path, expected_shape):
|
| 18 |
+
img = Image.open(image_path).convert('RGB')
|
| 19 |
+
|
| 20 |
+
img = img.resize((224, 224))
|
| 21 |
+
|
| 22 |
+
img_array = np.array(img, dtype=np.float32) / 255.0
|
| 23 |
+
|
| 24 |
+
mean = np.array([0.485, 0.456, 0.406], dtype=np.float32)
|
| 25 |
+
std = np.array([0.229, 0.224, 0.225], dtype=np.float32)
|
| 26 |
+
img_array = (img_array - mean) / std
|
| 27 |
+
|
| 28 |
+
if expected_shape[1] == 3:
|
| 29 |
+
img_array = np.transpose(img_array, (2, 0, 1))
|
| 30 |
+
|
| 31 |
+
return np.expand_dims(img_array, axis=0)
|
| 32 |
+
|
| 33 |
+
test_image_path = "download.jpg"
|
| 34 |
+
|
| 35 |
+
input_shape = input_details['shape']
|
| 36 |
+
input_data = preprocess_image(test_image_path, input_shape)
|
| 37 |
+
|
| 38 |
+
interpreter.set_tensor(input_details['index'], input_data)
|
| 39 |
+
interpreter.invoke()
|
| 40 |
+
|
| 41 |
+
out_0 = interpreter.get_tensor(output_details[0]['index'])[0]
|
| 42 |
+
out_1 = interpreter.get_tensor(output_details[1]['index'])[0]
|
| 43 |
+
|
| 44 |
+
if len(out_0) == 6:
|
| 45 |
+
pred_disaster = np.argmax(out_0)
|
| 46 |
+
pred_severity = np.argmax(out_1)
|
| 47 |
+
else:
|
| 48 |
+
pred_severity = np.argmax(out_0)
|
| 49 |
+
pred_disaster = np.argmax(out_1)
|
| 50 |
+
|
| 51 |
+
print("\n--- MRBEAN FLOAT16 OFFLINE ANALYSIS ---")
|
| 52 |
+
print(f"Disaster Type: {disaster_labels[pred_disaster]}")
|
| 53 |
+
print(f"Damage Severity: {severity_labels[pred_severity]}")
|
train.py
ADDED
|
@@ -0,0 +1,217 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import os
|
| 2 |
+
import time
|
| 3 |
+
import copy
|
| 4 |
+
import pandas as pd
|
| 5 |
+
import torch
|
| 6 |
+
import torch.nn as nn
|
| 7 |
+
import torch.optim as optim
|
| 8 |
+
import torchvision.models as models
|
| 9 |
+
import torchvision.transforms as transforms
|
| 10 |
+
from torch.utils.data import Dataset, DataLoader
|
| 11 |
+
from PIL import Image, ImageFile
|
| 12 |
+
|
| 13 |
+
ImageFile.LOAD_TRUNCATED_IMAGES = True
|
| 14 |
+
|
| 15 |
+
class MedicDataset(Dataset):
|
| 16 |
+
def __init__(self, dataframe, dataset_root, transform=None):
|
| 17 |
+
self.df = dataframe
|
| 18 |
+
self.root_dir = dataset_root
|
| 19 |
+
self.transform = transform
|
| 20 |
+
|
| 21 |
+
self.disaster_mapping = {
|
| 22 |
+
'earthquake': 0, 'fire': 1, 'flood': 2,
|
| 23 |
+
'hurricane': 3, 'landslide': 4, 'not_disaster': 5
|
| 24 |
+
}
|
| 25 |
+
self.severity_mapping = {
|
| 26 |
+
'little_or_none': 0, 'mild': 1, 'severe': 2
|
| 27 |
+
}
|
| 28 |
+
|
| 29 |
+
def __len__(self):
|
| 30 |
+
return len(self.df)
|
| 31 |
+
|
| 32 |
+
def __getitem__(self, idx):
|
| 33 |
+
relative_path = self.df.iloc[idx]['image_path']
|
| 34 |
+
img_path = os.path.join(self.root_dir, relative_path)
|
| 35 |
+
|
| 36 |
+
image = Image.open(img_path).convert('RGBA').convert('RGB')
|
| 37 |
+
|
| 38 |
+
disaster_str = self.df.iloc[idx]['disaster_types']
|
| 39 |
+
severity_str = self.df.iloc[idx]['damage_severity']
|
| 40 |
+
|
| 41 |
+
disaster_label = torch.tensor(self.disaster_mapping[disaster_str], dtype=torch.long)
|
| 42 |
+
severity_label = torch.tensor(self.severity_mapping[severity_str], dtype=torch.long)
|
| 43 |
+
|
| 44 |
+
if self.transform:
|
| 45 |
+
image = self.transform(image)
|
| 46 |
+
|
| 47 |
+
return image, disaster_label, severity_label
|
| 48 |
+
|
| 49 |
+
|
| 50 |
+
def prepare_dataframes(dataset_root):
|
| 51 |
+
target_disasters = ['earthquake', 'fire', 'flood', 'hurricane', 'landslide', 'not_disaster']
|
| 52 |
+
target_severities = ['severe', 'mild', 'little_or_none']
|
| 53 |
+
|
| 54 |
+
train_tsv = os.path.join(dataset_root, "MEDIC_train.tsv")
|
| 55 |
+
df_train_raw = pd.read_csv(train_tsv, sep='\t')
|
| 56 |
+
df_train = df_train_raw[
|
| 57 |
+
df_train_raw['disaster_types'].isin(target_disasters) &
|
| 58 |
+
df_train_raw['damage_severity'].isin(target_severities)
|
| 59 |
+
].copy()
|
| 60 |
+
df_train.reset_index(drop=True, inplace=True)
|
| 61 |
+
|
| 62 |
+
val_tsv = os.path.join(dataset_root, "MEDIC_dev.tsv")
|
| 63 |
+
df_val_raw = pd.read_csv(val_tsv, sep='\t')
|
| 64 |
+
df_val = df_val_raw[
|
| 65 |
+
df_val_raw['disaster_types'].isin(target_disasters) &
|
| 66 |
+
df_val_raw['damage_severity'].isin(target_severities)
|
| 67 |
+
].copy()
|
| 68 |
+
df_val.reset_index(drop=True, inplace=True)
|
| 69 |
+
|
| 70 |
+
return df_train, df_val
|
| 71 |
+
|
| 72 |
+
class MRBEANVisionModel(nn.Module):
|
| 73 |
+
def __init__(self, num_disaster_classes=6, num_severity_classes=3):
|
| 74 |
+
super(MRBEANVisionModel, self).__init__()
|
| 75 |
+
|
| 76 |
+
mobilenet = models.mobilenet_v3_large(weights=None)
|
| 77 |
+
|
| 78 |
+
local_weights_path = "./models/mobilenet_v3_large-8738ca79.pth"
|
| 79 |
+
if os.path.exists(local_weights_path):
|
| 80 |
+
mobilenet.load_state_dict(torch.load(local_weights_path))
|
| 81 |
+
print("Successfully loaded pre-trained MobileNet weights.")
|
| 82 |
+
else:
|
| 83 |
+
print("Warning: Local weights file not found. Training from scratch.")
|
| 84 |
+
|
| 85 |
+
self.features = mobilenet.features
|
| 86 |
+
self.pool = nn.AdaptiveAvgPool2d(1)
|
| 87 |
+
self.flatten = nn.Flatten()
|
| 88 |
+
|
| 89 |
+
self.disaster_head = nn.Sequential(
|
| 90 |
+
nn.Linear(960, 256),
|
| 91 |
+
nn.ReLU(),
|
| 92 |
+
nn.Dropout(0.3),
|
| 93 |
+
nn.Linear(256, num_disaster_classes)
|
| 94 |
+
)
|
| 95 |
+
|
| 96 |
+
self.severity_head = nn.Sequential(
|
| 97 |
+
nn.Linear(960, 128),
|
| 98 |
+
nn.ReLU(),
|
| 99 |
+
nn.Dropout(0.3),
|
| 100 |
+
nn.Linear(128, num_severity_classes)
|
| 101 |
+
)
|
| 102 |
+
|
| 103 |
+
def forward(self, x):
|
| 104 |
+
x = self.features(x)
|
| 105 |
+
x = self.pool(x)
|
| 106 |
+
x = self.flatten(x)
|
| 107 |
+
out_disaster = self.disaster_head(x)
|
| 108 |
+
out_severity = self.severity_head(x)
|
| 109 |
+
return out_disaster, out_severity
|
| 110 |
+
|
| 111 |
+
if __name__ == '__main__':
|
| 112 |
+
DATASET_ROOT = "./dataset"
|
| 113 |
+
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
|
| 114 |
+
|
| 115 |
+
df_train, df_val = prepare_dataframes(DATASET_ROOT)
|
| 116 |
+
|
| 117 |
+
basic_transform = transforms.Compose([
|
| 118 |
+
transforms.Resize((224, 224)),
|
| 119 |
+
transforms.ToTensor(),
|
| 120 |
+
transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
|
| 121 |
+
])
|
| 122 |
+
|
| 123 |
+
train_dataset = MedicDataset(df_train, DATASET_ROOT, basic_transform)
|
| 124 |
+
val_dataset = MedicDataset(df_val, DATASET_ROOT, basic_transform)
|
| 125 |
+
|
| 126 |
+
train_loader = DataLoader(
|
| 127 |
+
train_dataset,
|
| 128 |
+
batch_size=64,
|
| 129 |
+
shuffle=True,
|
| 130 |
+
num_workers=4,
|
| 131 |
+
pin_memory=True,
|
| 132 |
+
drop_last=True,
|
| 133 |
+
persistent_workers=True
|
| 134 |
+
)
|
| 135 |
+
|
| 136 |
+
val_loader = DataLoader(
|
| 137 |
+
val_dataset,
|
| 138 |
+
batch_size=64,
|
| 139 |
+
shuffle=False,
|
| 140 |
+
num_workers=4,
|
| 141 |
+
pin_memory=True,
|
| 142 |
+
persistent_workers=True
|
| 143 |
+
)
|
| 144 |
+
|
| 145 |
+
model = MRBEANVisionModel().to(device)
|
| 146 |
+
criterion_disaster = nn.CrossEntropyLoss()
|
| 147 |
+
criterion_severity = nn.CrossEntropyLoss()
|
| 148 |
+
optimizer = optim.AdamW(model.parameters(), lr=1e-4, weight_decay=1e-2)
|
| 149 |
+
scaler = torch.amp.GradScaler('cuda')
|
| 150 |
+
|
| 151 |
+
EPOCHS = 25
|
| 152 |
+
best_val_acc = 0.0
|
| 153 |
+
|
| 154 |
+
best_model_wts = copy.deepcopy(model.state_dict())
|
| 155 |
+
|
| 156 |
+
print(f"\nStarting {EPOCHS}-Epoch Training Loop...")
|
| 157 |
+
|
| 158 |
+
for epoch in range(EPOCHS):
|
| 159 |
+
print(f"\n=== Epoch {epoch + 1}/{EPOCHS} ===")
|
| 160 |
+
|
| 161 |
+
model.train()
|
| 162 |
+
running_loss = 0.0
|
| 163 |
+
start_time = time.time()
|
| 164 |
+
|
| 165 |
+
for batch_idx, (images, labels_d, labels_s) in enumerate(train_loader):
|
| 166 |
+
images = images.to(device, non_blocking=True)
|
| 167 |
+
labels_d = labels_d.to(device, non_blocking=True)
|
| 168 |
+
labels_s = labels_s.to(device, non_blocking=True)
|
| 169 |
+
|
| 170 |
+
optimizer.zero_grad()
|
| 171 |
+
|
| 172 |
+
with torch.amp.autocast('cuda'):
|
| 173 |
+
out_d, out_s = model(images)
|
| 174 |
+
loss = criterion_disaster(out_d, labels_d) + criterion_severity(out_s, labels_s)
|
| 175 |
+
|
| 176 |
+
scaler.scale(loss).backward()
|
| 177 |
+
scaler.step(optimizer)
|
| 178 |
+
scaler.update()
|
| 179 |
+
|
| 180 |
+
running_loss += loss.item()
|
| 181 |
+
|
| 182 |
+
avg_loss = running_loss / len(train_loader)
|
| 183 |
+
epoch_time = time.time() - start_time
|
| 184 |
+
print(f"Train Loss: {avg_loss:.4f} | Epoch Time: {epoch_time:.1f}s")
|
| 185 |
+
|
| 186 |
+
model.eval()
|
| 187 |
+
correct_d, correct_s, total = 0, 0, 0
|
| 188 |
+
|
| 189 |
+
with torch.no_grad():
|
| 190 |
+
for images, labels_d, labels_s in val_loader:
|
| 191 |
+
images = images.to(device, non_blocking=True)
|
| 192 |
+
labels_d = labels_d.to(device, non_blocking=True)
|
| 193 |
+
labels_s = labels_s.to(device, non_blocking=True)
|
| 194 |
+
|
| 195 |
+
with torch.amp.autocast('cuda'):
|
| 196 |
+
out_d, out_s = model(images)
|
| 197 |
+
|
| 198 |
+
_, pred_d = torch.max(out_d, 1)
|
| 199 |
+
_, pred_s = torch.max(out_s, 1)
|
| 200 |
+
|
| 201 |
+
total += labels_d.size(0)
|
| 202 |
+
correct_d += (pred_d == labels_d).sum().item()
|
| 203 |
+
correct_s += (pred_s == labels_s).sum().item()
|
| 204 |
+
|
| 205 |
+
acc_d = 100 * correct_d / total
|
| 206 |
+
acc_s = 100 * correct_s / total
|
| 207 |
+
avg_val_acc = (acc_d + acc_s) / 2
|
| 208 |
+
|
| 209 |
+
print(f"Val Disaster Acc: {acc_d:.2f}% | Val Severity Acc: {acc_s:.2f}%")
|
| 210 |
+
|
| 211 |
+
if avg_val_acc > best_val_acc:
|
| 212 |
+
best_val_acc = avg_val_acc
|
| 213 |
+
best_model_wts = copy.deepcopy(model.state_dict())
|
| 214 |
+
torch.save(best_model_wts, "best_multitask_vision_model.pth")
|
| 215 |
+
print("Checkpoint: New best model saved to disk!")
|
| 216 |
+
|
| 217 |
+
print(f"\nTraining Complete. Peak Average Validation Accuracy: {best_val_acc:.2f}%")
|