| --- |
| license: cc-by-4.0 |
| library_name: keras |
| pipeline_tag: object-detection |
| tags: |
| - person-detection |
| - search-and-rescue |
| - drone |
| - aerial |
| - tinyml |
| - tflite |
| - int8 |
| - esp32 |
| - centernet |
| --- |
| |
| # VoxHOD: tiny aerial person detector |
|
|
| A CenterNet-style, anchor-free person detector for downward-looking drone |
| cameras: **136,547 parameters, 197 KB full-int8 TFLite**, or **37,363 parameters, |
| 75 KB** for the nano variant, built for ESP32-S3-class microcontrollers. It marks where people are in a 160×160 RGB |
| tile so a drone can flag places worth a closer look. Search and rescue is the use it |
| has been trained and measured for so far. |
|
|
| > **Türkçe özet:** Havadan çekilen görüntüler için, mikrodenetleyicide |
| > çalışacak kadar küçük (136 bin parametre, int8 dosyası 197 KB), genel amaçlı |
| > bir insan tespit modeli. Şimdiye kadar arama kurtarma verileriyle eğitilip |
| > ölçüldü. HERIDAL test setinde int8 hâli, eşik 0,2'de insanların |
| > %83,9'unu buluyor. 75 KB'lık nano sürümü %81,2'sini buluyor. Eğitimde |
| > kullanılmayan bir WiSARD klibinde bu oranlar %61 ve %62'ye düşüyor. CC BY 4.0. |
|
|
| Code and training details: [ErdemWilkinson/VoxHOD](https://github.com/ErdemWilkinson/VoxHOD). |
|
|
| ## Files |
|
|
| | File | What it is | |
| |---|---| |
| | `voxhod_qat_int8.tflite` | **Recommended (v3).** Quantization-aware trained, full int8, input `[1,160,160,3]` int8, output `[1,40,40,3]` int8 | |
| | `voxhod_int8.tflite` | Post-training quantized int8 of the float model | |
| | `voxhod.keras` | Float model | |
| | `voxhod_nano_int8.tflite`, `voxhod_nano.keras` | **Nano.** Half width, 75 KB post-training quantized int8, same input and output | |
| | `evaluation.json`, `evaluation_wisard.json`, `evaluation_nano.json` | Recall / precision / false alarms per threshold | |
|
|
| ## Input and output |
|
|
| - **Input:** 160×160 RGB, `pixel / 127.5 - 1`. A 640×640 crop of a |
| 4000×3000 aerial photo, resized to 160, matches the test setting. |
| - **Output:** 40×40 grid (stride 4). |
| - Channel 0: person-centre heatmap (0 to 1). |
| - Channels 1 and 2: `log` box width and height in grid cells. |
| - **Detections:** local maxima of channel 0 above a threshold. Lower |
| thresholds find more people and give more false alarms. |
|
|
| ```python |
| import numpy as np |
| import tensorflow as tf |
| from PIL import Image |
| |
| interpreter = tf.lite.Interpreter(model_path="voxhod_qat_int8.tflite") |
| interpreter.allocate_tensors() |
| inp, out = interpreter.get_input_details()[0], interpreter.get_output_details()[0] |
| (si, zi), (so, zo) = inp["quantization"], out["quantization"] |
| |
| tile = np.asarray(Image.open("tile.png").convert("RGB").resize((160, 160)), np.float32) / 127.5 - 1 |
| interpreter.set_tensor(inp["index"], np.clip(np.round(tile / si + zi), -128, 127).astype(np.int8)[None]) |
| interpreter.invoke() |
| grid = (interpreter.get_tensor(out["index"])[0].astype(np.float32) - zo) * so |
| |
| heat = grid[..., 0] |
| peaks = heat == tf.nn.max_pool2d(heat[None, :, :, None], 3, 1, "SAME")[0, :, :, 0].numpy() |
| for y, x in zip(*np.nonzero(peaks & (heat >= 0.2))): |
| w, h = np.exp(grid[y, x, 1:]) * 4 |
| print(f"person at ({(x + 0.5) * 4:.0f}, {(y + 0.5) * 4:.0f}) px, about {w:.0f}x{h:.0f}, score {heat[y, x]:.2f}") |
| ``` |
|
|
| ## Results (HERIDAL test set, 3,507 tiles, 372 people) |
|
|
| A detection is a find when its centre is inside a person's box grown by one |
| grid cell. |
|
|
|  |
|
|
| | int8 file, threshold | Recall | Precision | False alarms per tile | |
| |---|---:|---:|---:| |
| | v3 QAT, 0.30 | 76.6% | 32.5% | 0.17 | |
| | **v3 QAT, 0.20** | **83.9%** | 20.4% | 0.35 | |
| | v3 QAT, 0.10 | 88.7% | 7.7% | 1.12 | |
| | nano PTQ, 0.30 | 72.3% | 25.8% | 0.22 | |
| | **nano PTQ, 0.20** | **81.2%** | 12.7% | 0.59 | |
| | nano PTQ, 0.10 | 89.5% | 4.8% | 1.87 | |
|
|
| ## External check: WiSARD sample |
|
|
| One clip of 264 correlated frames from another camera (6,336 tiles, 1,271 |
| people), never trained on. Treat it as a first signal. |
|
|
| | int8 file, threshold | Recall | Precision | False alarms per tile | |
| |---|---:|---:|---:| |
| | v3 QAT, 0.20 | 61.3% | 19.1% | 0.52 | |
| | nano PTQ, 0.20 | 62.2% | 13.4% | 0.81 | |
|
|
| ## Limitations |
|
|
| - Trained only on HERIDAL: summer, daytime, mostly Mediterranean terrain. |
| Recall falls by about 20 points on the WiSARD clip. Snow, forest canopy, |
| water, night and thermal are untested. |
| - 90% recall is not reached on the test set. |
| - Not yet measured on an ESP32-S3 (speed, memory) or on real flight video. |
|
|
| ## Data and license |
|
|
| Trained on [HERIDAL](https://zenodo.org/records/5662351) (CC BY 3.0): |
| D. Božić-Štulić, Ž. Marušić, S. Gotovac, "Deep Learning Approach in Aerial |
| Imagery for Supporting Land Search and Rescue Missions", IJCV 2019. |
|
|
| External test only, not training: WiSARD sample. D. Broyles, C. R. Hayner, |
| K. Leung, "WiSARD: A Labeled Visual and Thermal Image Dataset for Wilderness |
| Search and Rescue", IROS 2022. |
|
|
| Weights: **CC BY 4.0**. Training code: MIT. |
|
|
| ## Intended use |
|
|
| VoxHOD is a general-purpose aerial person detector. Search and rescue is the |
| use it has been trained and measured for so far: helping find people who are |
| lost or injured. Other settings are unmeasured. It is not intended for |
| targeting or harming people. |
|
|
