---
license: cc-by-4.0
library_name: keras
pipeline_tag: object-detection
tags:
- person-detection
- search-and-rescue
- drone
- aerial
- tinyml
- tflite
- int8
- esp32
- centernet
---

# VoxHOD: tiny aerial person detector

A CenterNet-style, anchor-free person detector for downward-looking drone
cameras: **136,547 parameters, 197 KB full-int8 TFLite**, or **37,363 parameters,
75 KB** for the nano variant, built for ESP32-S3-class microcontrollers. It marks where people are in a 160×160 RGB
tile so a drone can flag places worth a closer look. Search and rescue is the use it
has been trained and measured for so far.

> **Türkçe özet:** Havadan çekilen görüntüler için, mikrodenetleyicide
> çalışacak kadar küçük (136 bin parametre, int8 dosyası 197 KB), genel amaçlı
> bir insan tespit modeli. Şimdiye kadar arama kurtarma verileriyle eğitilip
> ölçüldü. HERIDAL test setinde int8 hâli, eşik 0,2'de insanların
> %83,9'unu buluyor. 75 KB'lık nano sürümü %81,2'sini buluyor. Eğitimde
> kullanılmayan bir WiSARD klibinde bu oranlar %61 ve %62'ye düşüyor. CC BY 4.0.

Code and training details: [ErdemWilkinson/VoxHOD](https://github.com/ErdemWilkinson/VoxHOD).

## Files

| File | What it is |
|---|---|
| `voxhod_qat_int8.tflite` | **Recommended (v3).** Quantization-aware trained, full int8, input `[1,160,160,3]` int8, output `[1,40,40,3]` int8 |
| `voxhod_int8.tflite` | Post-training quantized int8 of the float model |
| `voxhod.keras` | Float model |
| `voxhod_nano_int8.tflite`, `voxhod_nano.keras` | **Nano.** Half width, 75 KB post-training quantized int8, same input and output |
| `evaluation.json`, `evaluation_wisard.json`, `evaluation_nano.json` | Recall / precision / false alarms per threshold |

## Input and output

- **Input:** 160×160 RGB, `pixel / 127.5 - 1`. A 640×640 crop of a
  4000×3000 aerial photo, resized to 160, matches the test setting.
- **Output:** 40×40 grid (stride 4).
  - Channel 0: person-centre heatmap (0 to 1).
  - Channels 1 and 2: `log` box width and height in grid cells.
- **Detections:** local maxima of channel 0 above a threshold. Lower
  thresholds find more people and give more false alarms.

```python
import numpy as np
import tensorflow as tf
from PIL import Image

interpreter = tf.lite.Interpreter(model_path="voxhod_qat_int8.tflite")
interpreter.allocate_tensors()
inp, out = interpreter.get_input_details()[0], interpreter.get_output_details()[0]
(si, zi), (so, zo) = inp["quantization"], out["quantization"]

tile = np.asarray(Image.open("tile.png").convert("RGB").resize((160, 160)), np.float32) / 127.5 - 1
interpreter.set_tensor(inp["index"], np.clip(np.round(tile / si + zi), -128, 127).astype(np.int8)[None])
interpreter.invoke()
grid = (interpreter.get_tensor(out["index"])[0].astype(np.float32) - zo) * so

heat = grid[..., 0]
peaks = heat == tf.nn.max_pool2d(heat[None, :, :, None], 3, 1, "SAME")[0, :, :, 0].numpy()
for y, x in zip(*np.nonzero(peaks & (heat >= 0.2))):
    w, h = np.exp(grid[y, x, 1:]) * 4
    print(f"person at ({(x + 0.5) * 4:.0f}, {(y + 0.5) * 4:.0f}) px, about {w:.0f}x{h:.0f}, score {heat[y, x]:.2f}")
```

## Results (HERIDAL test set, 3,507 tiles, 372 people)

A detection is a find when its centre is inside a person's box grown by one
grid cell.

![Recall and precision of v3 and nano on the HERIDAL test set](voxhod-test-v3-nano-light.svg)

| int8 file, threshold | Recall | Precision | False alarms per tile |
|---|---:|---:|---:|
| v3 QAT, 0.30 | 76.6% | 32.5% | 0.17 |
| **v3 QAT, 0.20** | **83.9%** | 20.4% | 0.35 |
| v3 QAT, 0.10 | 88.7% | 7.7% | 1.12 |
| nano PTQ, 0.30 | 72.3% | 25.8% | 0.22 |
| **nano PTQ, 0.20** | **81.2%** | 12.7% | 0.59 |
| nano PTQ, 0.10 | 89.5% | 4.8% | 1.87 |

## External check: WiSARD sample

One clip of 264 correlated frames from another camera (6,336 tiles, 1,271
people), never trained on. Treat it as a first signal.

| int8 file, threshold | Recall | Precision | False alarms per tile |
|---|---:|---:|---:|
| v3 QAT, 0.20 | 61.3% | 19.1% | 0.52 |
| nano PTQ, 0.20 | 62.2% | 13.4% | 0.81 |

## Limitations

- Trained only on HERIDAL: summer, daytime, mostly Mediterranean terrain.
  Recall falls by about 20 points on the WiSARD clip. Snow, forest canopy,
  water, night and thermal are untested.
- 90% recall is not reached on the test set.
- Not yet measured on an ESP32-S3 (speed, memory) or on real flight video.

## Data and license

Trained on [HERIDAL](https://zenodo.org/records/5662351) (CC BY 3.0):
D. Božić-Štulić, Ž. Marušić, S. Gotovac, "Deep Learning Approach in Aerial
Imagery for Supporting Land Search and Rescue Missions", IJCV 2019.

External test only, not training: WiSARD sample. D. Broyles, C. R. Hayner,
K. Leung, "WiSARD: A Labeled Visual and Thermal Image Dataset for Wilderness
Search and Rescue", IROS 2022.

Weights: **CC BY 4.0**. Training code: MIT.

## Intended use

VoxHOD is a general-purpose aerial person detector. Search and rescue is the
use it has been trained and measured for so far: helping find people who are
lost or injured. Other settings are unmeasured. It is not intended for
targeting or harming people.