README.md

---
license: cc-by-nc-4.0
language:
  - en
  - zh
tags:
  - micro-expression
  - facial-expression-recognition
  - video-classification
  - spotting
  - pytorch
pipeline_tag: video-classification
---

# Micro-Expression Spotting and Recognition

**[中文文档](README_zh.md)** | English

A PyTorch-based end-to-end framework for **micro-expression recognition** (short clips) and **long-video micro-expression spotting**, with a built-in web visualization platform.

Supports datasets: CASME2 / SAMM / SMIC-HS / CAS(ME)³

---

## Quick Start

### 1. Set up environment

```bash
conda create -n me-env python=3.10
conda activate me-env

# GPU (CUDA 12.1)
conda install pytorch torchvision torchaudio pytorch-cuda=12.1 -c pytorch -c nvidia

# CPU only
# conda install pytorch torchvision torchaudio cpuonly -c pytorch

pip install -r requirements.txt
```

### 2. Download model weights

```python
from huggingface_hub import snapshot_download

snapshot_download(
    repo_id="ghy-cmd/micro-expression-weights",
    local_dir=".",  # restores checkpoints/ and weights/ under project root
)
```

### 3. Start the web app

```bash
python -m webapp.app
# open http://localhost:5001
```

---

## Features

| Feature | Description |
|---------|-------------|
| **Micro-Expression Recognition** | Upload a short video clip, auto-preprocess and classify emotion via PGCAN model |
| **Long-Video Spotting** | Upload any long video, auto face-crop + sliding-window inference, locates all micro-expression intervals |

---

## Model Description

### Task 1 · Micro-Expression Recognition

Classifies short clips into **3 categories**: Positive / Negative / Surprise

- HRNet feature extractor on onset & apex frames → (2, 256, 64, 64)
- TV-L1 optical flow (u, v, strain) → (3, 256, 256)
- Transformer encoder with attention fusion (16 heads, 4 stages, full guidance)
- Trained with LOSO cross-validation on Composite dataset (CASME2 + SMIC-HS + SAMM)

### Task 2 · Long-Video Micro-Expression Spotting

Detects all micro-expression intervals in arbitrary-length videos, classifies each into **4 categories**: Positive / Negative / Surprise / Others

- Sliding-window inference (window = 0.4s, stride = 0.2s @ 30 fps)
- Multi-scale temporal features (6 scales)
- Dual-task head: detection score + emotion classification
- Trained on CAS(ME)³-LV dataset
- Evaluation metric: overlap-based F1 (any frame overlap = TP)

---

## Preprocessing Pipeline

Both tasks share the same pipeline, consistent with dataset preprocessing:

1. **Face crop** (dlib 68-point landmarks) → 256×256
2. **Apex frame detection** (optical strain + UPC) — recognition only
3. **HRNet feature extraction** → (2, 256, 64, 64)
4. **Optical flow** (TV-L1) → (3, 256, 256)

---

## Training

```bash
# Recognition — LOSO training
python train_loso.py --config configs/train_loso.yaml --dataset composite

# Detection — LOSO training
python detection/train_detection.py --config configs/detection_config.yaml
```

See [docs/TRAINING_GUIDE.md](docs/TRAINING_GUIDE.md) for full training guide.

---

## Project Structure

```
├── webapp/                  # Web visualization platform (Flask)
│   ├── app.py               # Entry point → http://localhost:5000
│   ├── api/                 # REST API (recognition + detection)
│   └── core/                # Inference wrappers
├── models/                  # Model architectures
├── detection/               # Spotting training & evaluation
├── trainers/                # Recognition training utilities
├── configs/                 # YAML config files
├── datasets/                # Dataset loaders
├── utils/                   # Shared utilities
├── docs/                    # Documentation & visualization scripts
├── train_loso.py            # Recognition LOSO training script
└── requirements.txt
```

---

## License

[CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/) — free for academic & personal use, **commercial use prohibited**.