README.md

metadata

license: cc-by-4.0
library_name: keras
pipeline_tag: object-detection
tags:
  - person-detection
  - search-and-rescue
  - drone
  - aerial
  - tinyml
  - tflite
  - int8
  - esp32
  - centernet

VoxHOD: tiny person detector for drone search and rescue

A CenterNet-style, anchor-free person detector for downward-looking drone cameras: 136,547 parameters, 197 KB full-int8 TFLite, or 37,363 parameters, 75 KB for the nano variant, built for ESP32-S3-class microcontrollers. It marks where people are in a 160×160 RGB tile so a drone can flag places worth a closer look for a rescue team.

Türkçe özet: Arama kurtarma dronları için, mikrodenetleyicide çalışacak kadar küçük (136 bin parametre, int8 dosyası 197 KB) bir insan tespit modeli. HERIDAL test setinde int8 hâli, eşik 0,2'de insanların %83,9'unu buluyor. 75 KB'lık nano sürümü %81,2'sini buluyor. Eğitimde kullanılmayan bir WiSARD klibinde bu oranlar %61 ve %62'ye düşüyor. CC BY 4.0.

Code and training details: ErdemWilkinson/VoxHOD.

Files

File	What it is
voxhod_qat_int8.tflite	Recommended (v3). Quantization-aware trained, full int8, input [1,160,160,3] int8, output [1,40,40,3] int8
voxhod_int8.tflite	Post-training quantized int8 of the float model
voxhod.keras	Float model
voxhod_nano_int8.tflite, voxhod_nano.keras	Nano. Half width, 75 KB post-training quantized int8, same input and output
evaluation.json, evaluation_wisard.json, evaluation_nano.json	Recall / precision / false alarms per threshold

Input and output

• Input: 160×160 RGB, pixel / 127.5 - 1. A 640×640 crop of a 4000×3000 aerial photo, resized to 160, matches the test setting.
• Output: 40×40 grid (stride 4).
  • Channel 0: person-centre heatmap (0 to 1).
  • Channels 1 and 2: log box width and height in grid cells.
• Detections: local maxima of channel 0 above a threshold. Lower thresholds find more people and give more false alarms.

import numpy as np
import tensorflow as tf
from PIL import Image

interpreter = tf.lite.Interpreter(model_path="voxhod_qat_int8.tflite")
interpreter.allocate_tensors()
inp, out = interpreter.get_input_details()[0], interpreter.get_output_details()[0]
(si, zi), (so, zo) = inp["quantization"], out["quantization"]

tile = np.asarray(Image.open("tile.png").convert("RGB").resize((160, 160)), np.float32) / 127.5 - 1
interpreter.set_tensor(inp["index"], np.clip(np.round(tile / si + zi), -128, 127).astype(np.int8)[None])
interpreter.invoke()
grid = (interpreter.get_tensor(out["index"])[0].astype(np.float32) - zo) * so

heat = grid[..., 0]
peaks = heat == tf.nn.max_pool2d(heat[None, :, :, None], 3, 1, "SAME")[0, :, :, 0].numpy()
for y, x in zip(*np.nonzero(peaks & (heat >= 0.2))):
    w, h = np.exp(grid[y, x, 1:]) * 4
    print(f"person at ({(x + 0.5) * 4:.0f}, {(y + 0.5) * 4:.0f}) px, about {w:.0f}x{h:.0f}, score {heat[y, x]:.2f}")

Results (HERIDAL test set, 3,507 tiles, 372 people)

A detection is a find when its centre is inside a person's box grown by one grid cell.

int8 file, threshold	Recall	Precision	False alarms per tile
v3 QAT, 0.30	76.6%	32.5%	0.17
v3 QAT, 0.20	83.9%	20.4%	0.35
v3 QAT, 0.10	88.7%	7.7%	1.12
nano PTQ, 0.30	72.3%	25.8%	0.22
nano PTQ, 0.20	81.2%	12.7%	0.59
nano PTQ, 0.10	89.5%	4.8%	1.87

External check: WiSARD sample

One clip of 264 correlated frames from another camera (6,336 tiles, 1,271 people), never trained on. Treat it as a first signal.

int8 file, threshold	Recall	Precision	False alarms per tile
v3 QAT, 0.20	61.3%	19.1%	0.52
nano PTQ, 0.20	62.2%	13.4%	0.81

Limitations

• Trained only on HERIDAL: summer, daytime, mostly Mediterranean terrain. Recall falls by about 20 points on the WiSARD clip. Snow, forest canopy, water, night and thermal are untested.
• 90% recall is not reached on the test set.
• Not yet measured on an ESP32-S3 (speed, memory) or on real flight video.

Data and license

Not trained on the Ferhat Tatık footage. That footage is licensed to this project for non-commercial use only, so models trained on it are published separately as VoxHOD-NC (CC BY-NC 4.0). VoxHOD stays CC BY 4.0.

Trained on HERIDAL (CC BY 3.0): D. Božić-Štulić, Ž. Marušić, S. Gotovac, "Deep Learning Approach in Aerial Imagery for Supporting Land Search and Rescue Missions", IJCV 2019.

External test only, not training: WiSARD sample. D. Broyles, C. R. Hayner, K. Leung, "WiSARD: A Labeled Visual and Thermal Image Dataset for Wilderness Search and Rescue", IROS 2022.

Weights: CC BY 4.0. Training code: MIT.

Intended use

Search and rescue: helping find people who are lost or injured. It is not intended for targeting or harming people.