issai/fes-face

FES Face Bounding Box Detection (Event Cameras)

Environment Installation & Set up

To launch training or inference, first of all it is required to install Metavision SDK environment.

• If you use Linux OS, install Metavision SDK environment from this link.

• If you use Windows OS, install Metavision SDK environment from this link.

Git repo:

https://github.com/IS2AI/faces-in-event-streams

The following scripts can be found withhin our github repo:

Face detection model

• Inference

To run inference for face bounding box model, use the following command:

python3 detection_and_tracking_pipeline.py --object_detector_dir /path/to/model --record_file <RAW file to process> --display

alternatively you can proceed with instructions from this link.

• Training

Before launching training, please place label_map_dictionary.json, which comes as part of this repo, in the same folder where your train/val/test folders are located.

1. To train the model to detect face bounding box in event streams, run:

cd <path to train_detection.py>
python3 train_detection.py <path to output directory> <path to dataset>

alternatively, you can follow instructions from here.

2. To select the feature extractor, you need to define --feature_extractor option using:

python3 train_detection.py <path to output directory> <path to dataset> --feature_extractor <select feature extractor: Vanilla, ResNet_18, ResNet_34, ResNet_50>

Facial landmark detection model

To train or detect five point facial landmark detection, the change to the following files should be done:

• feature_extractors.py
• box_processing.py
• display_frame.py
• modules.py

Changes which need to be done can be found in our repo in files:

• box_processing_difference.py
• display_frame_difference.py
• modules_difference.py
• feature_extractors_difference.py

Further, train and inference instructions are the same as for face detection part, declared above.

Since

• feature_extractors.py
• box_processing.py
• display_frame.py
• modules.py

are files which originally comes as part of Metavision SDK which comes under Prophesee Metavision Licensing, we will share only changes that has to be done on the installed Metavision SDK package (once you install Metavision SDK, you will have direct access to the original files).

If you use the dataset/source code/pre-trained models in your research, please cite our work:

@Article{s24051409,
AUTHOR = {Bissarinova, Ulzhan and Rakhimzhanova, Tomiris and Kenzhebalin, Daulet and Varol, Huseyin Atakan},
TITLE = {Faces in Event Streams (FES): An Annotated Face Dataset for Event Cameras},
JOURNAL = {Sensors},
VOLUME = {24},
YEAR = {2024},
NUMBER = {5},
ARTICLE-NUMBER = {1409},
URL = {https://www.mdpi.com/1424-8220/24/5/1409},
ISSN = {1424-8220},
DOI = {10.3390/s24051409}
}