VoxHOD: tiny aerial person detector

A CenterNet-style, anchor-free person detector for downward-looking drone cameras: 136,547 parameters, 197 KB full-int8 TFLite, or 37,363 parameters, 75 KB for the nano variant, built for ESP32-S3-class microcontrollers. It marks where people are in a 160×160 RGB tile so a drone can flag places worth a closer look. Search and rescue is the use it has been trained and measured for so far.

Türkçe özet: Havadan çekilen görüntüler için, mikrodenetleyicide çalışacak kadar küçük (136 bin parametre, int8 dosyası 197 KB), genel amaçlı bir insan tespit modeli. Şimdiye kadar arama kurtarma verileriyle eğitilip ölçüldü. HERIDAL test setinde int8 hâli, eşik 0,2'de insanların %83,9'unu buluyor. 75 KB'lık nano sürümü %81,2'sini buluyor. Eğitimde kullanılmayan bir WiSARD klibinde bu oranlar %61 ve %62'ye düşüyor. CC BY 4.0.

Code and training details: ErdemWilkinson/VoxHOD.

Files

File	What it is
voxhod_qat_int8.tflite	Recommended (v3). Quantization-aware trained, full int8, input [1,160,160,3] int8, output [1,40,40,3] int8
voxhod_int8.tflite	Post-training quantized int8 of the float model
voxhod.keras	Float model
voxhod_nano_int8.tflite, voxhod_nano.keras	Nano. Half width, 75 KB post-training quantized int8, same input and output
evaluation.json, evaluation_wisard.json, evaluation_nano.json	Recall / precision / false alarms per threshold

Input and output

• Input: 160×160 RGB, pixel / 127.5 - 1. A 640×640 crop of a 4000×3000 aerial photo, resized to 160, matches the test setting.
• Output: 40×40 grid (stride 4).
  • Channel 0: person-centre heatmap (0 to 1).
  • Channels 1 and 2: log box width and height in grid cells.
• Detections: local maxima of channel 0 above a threshold. Lower thresholds find more people and give more false alarms.

import numpy as np
import tensorflow as tf
from PIL import Image

interpreter = tf.lite.Interpreter(model_path="voxhod_qat_int8.tflite")
interpreter.allocate_tensors()
inp, out = interpreter.get_input_details()[0], interpreter.get_output_details()[0]
(si, zi), (so, zo) = inp["quantization"], out["quantization"]

tile = np.asarray(Image.open("tile.png").convert("RGB").resize((160, 160)), np.float32) / 127.5 - 1
interpreter.set_tensor(inp["index"], np.clip(np.round(tile / si + zi), -128, 127).astype(np.int8)[None])
interpreter.invoke()
grid = (interpreter.get_tensor(out["index"])[0].astype(np.float32) - zo) * so

heat = grid[..., 0]
peaks = heat == tf.nn.max_pool2d(heat[None, :, :, None], 3, 1, "SAME")[0, :, :, 0].numpy()
for y, x in zip(*np.nonzero(peaks & (heat >= 0.2))):
    w, h = np.exp(grid[y, x, 1:]) * 4
    print(f"person at ({(x + 0.5) * 4:.0f}, {(y + 0.5) * 4:.0f}) px, about {w:.0f}x{h:.0f}, score {heat[y, x]:.2f}")

Results (HERIDAL test set, 3,507 tiles, 372 people)

A detection is a find when its centre is inside a person's box grown by one grid cell.

int8 file, threshold	Recall	Precision	False alarms per tile
v3 QAT, 0.30	76.6%	32.5%	0.17
v3 QAT, 0.20	83.9%	20.4%	0.35
v3 QAT, 0.10	88.7%	7.7%	1.12
nano PTQ, 0.30	72.3%	25.8%	0.22
nano PTQ, 0.20	81.2%	12.7%	0.59
nano PTQ, 0.10	89.5%	4.8%	1.87

External check: WiSARD sample

One clip of 264 correlated frames from another camera (6,336 tiles, 1,271 people), never trained on. Treat it as a first signal.

int8 file, threshold	Recall	Precision	False alarms per tile
v3 QAT, 0.20	61.3%	19.1%	0.52
nano PTQ, 0.20	62.2%	13.4%	0.81

Limitations

• Trained only on HERIDAL: summer, daytime, mostly Mediterranean terrain. Recall falls by about 20 points on the WiSARD clip. Snow, forest canopy, water, night and thermal are untested.
• 90% recall is not reached on the test set.
• Not yet measured on an ESP32-S3 (speed, memory) or on real flight video.

Data and license

Trained on HERIDAL (CC BY 3.0): D. Božić-Štulić, Ž. Marušić, S. Gotovac, "Deep Learning Approach in Aerial Imagery for Supporting Land Search and Rescue Missions", IJCV 2019.

External test only, not training: WiSARD sample. D. Broyles, C. R. Hayner, K. Leung, "WiSARD: A Labeled Visual and Thermal Image Dataset for Wilderness Search and Rescue", IROS 2022.

Weights: CC BY 4.0. Training code: MIT.

Intended use

VoxHOD is a general-purpose aerial person detector. Search and rescue is the use it has been trained and measured for so far: helping find people who are lost or injured. Other settings are unmeasured. It is not intended for targeting or harming people.