FireWatch-AI / README.md
omerfarooq223's picture
docs: refine dataset distinction, reference split labeling, checkpoint vs peak metrics, and limitations phrasing
d94c8de verified
|
Raw History Blame Contribute Delete
9.87 kB
---
language:
- en
license: agpl-3.0
tags:
- vision
- object-detection
- instance-segmentation
- yolo
- yolov8
- fire-detection
- smoke-detection
- safety
pipeline_tag: object-detection
datasets:
- openrsgis/FASDD
model-index:
- name: FireWatch-AI
results:
- task:
type: object-detection
name: Object Detection
dataset:
name: FASDD_CV (Flame and Smoke Detection Dataset)
type: openrsgis/FASDD
split: validation
metrics:
- name: Box mAP@50 (detector-best.pt, Saved Checkpoint)
type: mAP
value: 79.91
- name: Box mAP@50-95 (detector-best.pt, Saved Checkpoint)
type: mAP
value: 54.23
- name: Box Precision (detector-best.pt, Saved Checkpoint)
type: precision
value: 78.84
- name: Box Recall (detector-best.pt, Saved Checkpoint)
type: recall
value: 70.41
- task:
type: instance-segmentation
name: Instance Segmentation
dataset:
name: Clean Fire Segmentation (Derived Subset)
type: custom
split: validation
metrics:
- name: Mask mAP@50 (seg-best.pt, Saved Checkpoint)
type: mAP
value: 92.29
- name: Mask mAP@50-95 (seg-best.pt, Saved Checkpoint)
type: mAP
value: 62.29
- name: Mask Precision (seg-best.pt, Saved Checkpoint)
type: precision
value: 90.50
- name: Mask Recall (seg-best.pt, Saved Checkpoint)
type: recall
value: 86.26
---
# FireWatch AI — Dual Detection & Segmentation YOLO Models
[![Inference Space](https://img.shields.io/badge/%F0%9F%A4%97%20Space-FireWatch_Inference-blue)](https://huggingface.co/spaces/omerfarooq223/firewatch-yolo-inference)
[![GitHub](https://img.shields.io/badge/GitHub-FireWatch_AI-181717?logo=github)](https://github.com/omerfarooq223/FireWatch-AI)
[![License: AGPL-3.0](https://img.shields.io/badge/License-AGPL_3.0-orange)](https://github.com/ultralytics/ultralytics/blob/main/LICENSE)
This repository provides fine-tuned YOLOv8 weights powering the **FireWatch AI** safety monitoring and incident management framework.
Live inference microservice: [omerfarooq223/firewatch-yolo-inference](https://huggingface.co/spaces/omerfarooq223/firewatch-yolo-inference)
---
## Model Checkpoints
This repository hosts two distinct weights files:
1. **`detector-best.pt`** (~262 MB)
* **Task:** Bounding-box Object Detection
* **Base Architecture:** **YOLOv8l** (`depth=1.0, width=1.0`, nc=2)
* **Classes:**
* `0: fire`
* `1: smoke`
* **Role:** Localizing flame and smoke bounding boxes in surveillance video frames.
2. **`seg-best.pt`** (~6.7 MB)
* **Task:** Polygon Instance Segmentation
* **Base Architecture:** **YOLOv8n-seg** (`depth=0.33, width=0.25`, nc=1)
* **Classes:**
* `0: fire`
* **Role:** Extracting pixel-level flame boundaries and calculating fire surface area. (Does not segment smoke).
---
## Evaluation Metrics (Validation Split)
Below are the exact metrics recorded in the training checkpoint logs. Metrics associated with the saved `best.pt` checkpoint are distinguished from individual peak-epoch measurements:
### 1. Detection Model (`detector-best.pt` — YOLOv8l)
* **Training duration:** 17 epochs completed (target was 50; training stopped at Epoch 17).
* **Saved Checkpoint (`best.pt` / Epoch 17):**
* **`metrics/mAP50(B)`**: **79.91%** (`0.79907`)
* **`metrics/mAP50-95(B)`**: **54.23%** (`0.54227`, peak fitness for the run)
* **`metrics/precision(B)`**: **78.84%** (`0.78842`)
* **`metrics/recall(B)`**: **70.41%** (`0.70407`)
* `val/box_loss`: `1.08875`
* `val/cls_loss`: `0.87772`
* `val/dfl_loss`: `1.28927`
* **Peak Metric Comparison:**
* Box mAP@50 reached its individual peak at **Epoch 16** with **79.92%** (`0.79921`, with Precision: `78.88%`, Recall: `70.41%`, mAP50-95: `54.15%`), before `best.pt` was finalized at Epoch 17 on overall fitness.
### 2. Segmentation Model (`seg-best.pt` — YOLOv8n-seg)
* **Training duration:** 20 epochs completed. The saved `best.pt` corresponds to the final epoch evaluated (Epoch 20, fitness `1.40684`).
* **Saved Checkpoint Metrics (Epoch 20):**
* **Mask Metrics (Contour Segmentation):**
* **`metrics/mAP50(M)`**: **92.29%** (`0.92287`)
* **`metrics/mAP50-95(M)`**: **62.29%** (`0.62285`)
* **`metrics/precision(M)`**: **90.50%** (`0.90495`)
* **`metrics/recall(M)`**: **86.26%** (`0.86260`)
* **Bounding Box Metrics:**
* **`metrics/mAP50(B)`**: **96.50%** (`0.96503`)
* **`metrics/mAP50-95(B)`**: **78.40%** (`0.78399`)
* **`metrics/precision(B)`**: **92.11%** (`0.92114`)
* **`metrics/recall(B)`**: **89.65%** (`0.89654`)
* `val/box_loss`: `0.66318` | `val/seg_loss`: `1.44122` | `val/cls_loss`: `0.50684` | `val/dfl_loss`: `0.92107`
---
## Dataset Details
The detector and segmentation models were trained on separate dataset configurations:
### A. Detection Dataset (`detector-best.pt`)
* **Upstream Benchmark:** Trained on the **FASDD (Flame And Smoke Detection Dataset)** — `FASDD_CV` computer vision benchmark subset (*Wang et al., 2023*).
* **Benchmark Size:** **95,314 total images** (full official FASDD_CV benchmark across all splits combined).
* **Reference Upstream Benchmark Splits:**
> *Note: The figures below represent the official published FASDD_CV benchmark partition ratios provided for reference. The exact split counts ingested by the specific training notebook script were managed via `data.yaml` and are not recorded in the checkpoint dictionary.*
* Reference Training Partition (~50%): ~47,657 images
* Reference Validation Partition (~33.3%): ~31,771 images
* Reference Test Partition (~16.7%): ~15,886 images
* **Annotations:** Dual-class bounding box labels for `fire` and `smoke`.
### B. Segmentation Dataset (`seg-best.pt`)
* **Subset Configuration:** Trained using a derived fire segmentation configuration referenced in the training metadata as `/kaggle/working/clean_fire_seg.yaml`.
* **Subset Size & Splits:** **Not recorded.** The derived dataset files, image manifests, and exact train/val sample counts were generated in an ephemeral Kaggle working environment and are not embedded in the model checkpoint.
* **Target Scope:** Single-class polygon annotations for `fire` only (smoke masks are not included).
---
## Training Configuration
| Parameter | Detection (`detector-best.pt`) | Segmentation (`seg-best.pt`) |
| :--- | :--- | :--- |
| **Model Architecture** | YOLOv8l (Large) | YOLOv8n-seg (Nano Segmentation) |
| **Pretrained Checkpoint** | Resumed / fine-tuned from `last.pt` | `yolov8n-seg.pt` |
| **Input Image Size** | 640 × 640 | 640 × 640 |
| **Batch Size** | 16 | 16 |
| **Epochs Trained** | 17 epochs | 20 epochs |
| **Optimizer** | Auto (SGD/AdamW, momentum: 0.937, weight_decay: 0.0005) | Auto (SGD/AdamW, momentum: 0.937, weight_decay: 0.0005) |
| **Dropout** | 0.5 | 0.0 |
| **Hardware** | Kaggle GPU (`device: 0`, GPU model: *not recorded*) | Kaggle GPU (`device: 0`, GPU model: *not recorded*) |
| **Ultralytics Version** | `8.4.41` | `8.4.45` |
| **Run Name / Project** | `yolov8l-FASDD` (`Fire-Smoke-FASDD`) | `final_fire_segmentation` |
---
## How to Run Inference
### Installation
```bash
pip install ultralytics huggingface_hub opencv-python pillow
```
### Python Inference Example
```python
from huggingface_hub import hf_hub_download
from ultralytics import YOLO
# 1. Download weights from Hugging Face Hub
det_path = hf_hub_download(repo_id="omerfarooq223/FireWatch-AI", filename="detector-best.pt")
seg_path = hf_hub_download(repo_id="omerfarooq223/FireWatch-AI", filename="seg-best.pt")
# 2. Instantiate YOLO models
det_model = YOLO(det_path)
seg_model = YOLO(seg_path)
image_path = "test_frame.jpg"
# 3. Detect fire and smoke bounding boxes
det_results = det_model.predict(source=image_path, conf=0.40, imgsz=640)[0]
for box in det_results.boxes:
cls_id = int(box.cls[0])
label = det_results.names[cls_id]
conf = float(box.conf[0])
xyxy = box.xyxy[0].tolist()
print(f"Detected {label} ({conf:.2%}) at {xyxy}")
# 4. Extract polygon masks for flame regions
seg_results = seg_model.predict(source=image_path, conf=0.35, imgsz=640)[0]
if seg_results.masks is not None:
for polygon in seg_results.masks.xy:
print(f"Fire polygon coordinates count: {len(polygon)}")
```
---
## Potential Limitations & Operational Considerations
The following items represent theoretical considerations and known architectural constraints; untested scenarios have not been quantitatively evaluated:
* **Potential Optical Confounds:** In vision-based monitoring systems, strong directional solar glare, low-angle sunrise/sunset backlighting, reflections off polished surfaces, or high-intensity industrial sodium lamps may pose risks of false positive activations. These environmental conditions have not been systematically tested.
* **Atmospheric Particulates vs. Smoke:** Thin or diffuse smoke plumes share visual features with dense fog, steam plumes, and suspended dust. Performance in heavy fog or steam environments has not been benchmarked.
* **Low-Light / Nighttime Deployment:** Detection capability in poorly lit conditions without active illumination or complementary thermal/infrared sensors remains unverified.
* **Architectural Scope Constraint:** The segmentation checkpoint (`seg-best.pt`) was trained strictly for flame instance segmentation (`fire`); it does not segment smoke regions. Use `detector-best.pt` when smoke monitoring is needed.
---
## Licensing
* **Model Weights:** Ultralytics YOLOv8 weights are distributed under the [AGPL-3.0 License](https://github.com/ultralytics/ultralytics/blob/main/LICENSE).
* **Application Framework:** [MIT License](https://github.com/omerfarooq223/FireWatch-AI/blob/main/LICENSE).
* **FASDD Dataset:** Distributed under academic research and non-commercial benchmarking terms.