ljchang commited on
Commit
cdb00be
·
verified ·
1 Parent(s): 52bb85d

Add v2.5 model: face_multitask_v2.safetensors (config in metadata) + 52 blendshapes; update model card

Browse files
Files changed (2) hide show
  1. README.md +64 -58
  2. face_multitask_v2.safetensors +3 -0
README.md CHANGED
@@ -1,76 +1,82 @@
1
  ---
2
  license: other
3
  license_name: research-only
 
 
4
  tags:
5
- - facial-action-units
6
- - facial-expression-recognition
 
7
  - gaze-estimation
8
- - facial-landmarks
 
 
9
  - multitask
10
- - py-feat
11
- library_name: py-feat
12
  ---
13
 
14
  # face_multitask_v2
15
 
16
- Multitask facial-behavior model for **[py-feat](https://github.com/cosanlab/py-feat)**'s
17
- `Detectorv2`. A single ConvNeXt V2-Tiny backbone with lightweight task heads jointly
18
- predicts, from one forward pass:
19
-
20
- - **20 facial action units** (FACS, presence probabilities)
21
- - **7-class emotion** (Neutral, Happy, Sad, Surprise, Fear, Disgust, Anger)
22
- - **valence / arousal** (continuous, [-1, 1])
23
- - **gaze** (yaw, pitch)
24
- - **478-point MediaPipe-topology face mesh**
25
- - **6-DoF head pose**
26
-
27
- `face_multitask_v2.pt` contains `{model: state_dict, config: ModelV2Config, ...}`.
28
- py-feat loads it via `feat.multitask.inference.MultitaskModel` inside `Detectorv2`.
29
-
30
- ## Architecture
31
-
32
- - **Backbone:** ConvNeXt V2-Tiny (FCMAE + ImageNet-22k/1k pretrain).
33
- - **AU head:** ANFL graph — AFG (per-AU branches) → FGG (k-NN cosine GCN) → cosine-
34
- similarity classifier. (No MEFL edge head — ablation showed it was inert and ~4×
35
- slower; see notes.)
36
- - **Unified features:** backbone GAP ∥ projected mesh-xy, feeding the emotion and gaze
37
- heads (OF3-style). Emotion head additionally conditioned on the AU probabilities.
38
- - **Gaze:** L2CS-style 4-FC head. **Landmark / pose:** MLP heads (frozen after stage 1).
39
- - Multi-task loss with Kendall homoscedastic uncertainty weighting. ~37.6M params.
40
-
41
- Input: a 256×256 aligned face crop → center-crop 224 → ImageNet normalize.
42
-
43
- ## What changed vs v1 (face_multitask_v1)
44
-
45
- - Dropped 4 poorly-represented AUs (AU16/18/27/45) → **20 AUs** (v1's 24).
46
- - Dropped **Contempt** → **7-class** emotion (v1's 8).
47
- - Removed the MEFL edge head (≈4× faster model forward, single-image latency ≈ OpenFace 3.0).
48
- - Emotion improved substantially; valence/arousal added/strengthened.
49
-
50
- ## Benchmarks (py-feat end-to-end harness)
51
-
52
- | task | v2 | v1 (v2.3) | OpenFace 3.0 |
53
  |---|---|---|---|
54
- | DISFA+ AU macro-F1 (8-AU matched) | 0.756 | 0.757 | 0.732 |
55
- | AffectNet-7 macro-F1 | 0.330 | 0.264 | ~0.40* |
56
- | RAF-DB test macro-F1 | **0.849** | 0.751 | — |
57
- | Aff-Wild2 valence / arousal CCC | **0.816 / 0.783** | 0.79 / 0.74 | (no V/A head) |
58
- | MPIIGaze mean angular err | 3.92° | 3.33° | 2.56° |
59
- | Gaze360 mean angular err | **6.81°** | 5.81° | 10.6° |
60
-
61
- \* OF3 emotion measured on our chips (approximate crop); its paper reports higher.
62
- Numbers are macro-F1 / CCC / degrees on held-out external test sets.
 
 
 
 
 
 
63
 
64
  ## Usage
65
 
66
  ```python
67
- from feat.detector_v2 import Detectorv2
68
- det = Detectorv2(device="cuda") # downloads this model
69
- fex = det.detect("face.jpg", data_type="image")
70
  ```
71
 
72
- ## Notes / license
 
 
 
 
73
 
74
- Research use. Trained on a mix of public AU, emotion, gaze, and landmark datasets;
75
- respect each source dataset's license. The optional ArcFace identity branch in
76
- Detectorv2 is a separate non-commercial-research model (not part of this checkpoint).
 
 
1
  ---
2
  license: other
3
  license_name: research-only
4
+ license_link: LICENSE
5
+ library_name: py-feat
6
  tags:
7
+ - facial-expression-analysis
8
+ - action-units
9
+ - emotion-recognition
10
  - gaze-estimation
11
+ - face-landmarks
12
+ - head-pose
13
+ - blendshapes
14
  - multitask
15
+ pipeline_tag: image-classification
 
16
  ---
17
 
18
  # face_multitask_v2
19
 
20
+ A single multi-task convolutional model for facial behavior analysis, used by
21
+ [py-feat](https://github.com/cosanlab/py-feat)'s `Detectorv2`. From one face crop
22
+ it jointly predicts **action units, categorical emotion, valence/arousal,
23
+ eye gaze, a 478-point face mesh, 6-DoF head pose, and 52 MediaPipe/ARKit
24
+ blendshapes** (the v2.5 model; replaces v2.4).
25
+
26
+ - **Backbone:** ConvNeXt-V2 Tiny (FCMAE + IN-22k/IN-1k pretrained)
27
+ - **Heads:** ME-GraphAU AU graph (AFG/FGG/SC) + unified-feature emotion/V-A and
28
+ gaze heads + landmark, pose, and **blendshape** regression heads
29
+ - **Params:** ~30M · **Input:** 224×224 RGB (from a 256×256 face crop)
30
+ - **File:** `face_multitask_v2.safetensors` (safetensors; `ModelV2Config` JSON in the file metadata)
31
+
32
+ ## Outputs
33
+
34
+ | Task | Output | Notes |
35
+ |---|---|---|
36
+ | Action Units | 20 probabilities [0,1] | AU01,02,04,05,06,07,09,10,11,12,14,15,17,20,23,24,25,26,28,43 |
37
+ | Emotion | 7-class softmax | Neutral, Happy, Sad, Surprise, Fear, Disgust, Anger |
38
+ | Valence / Arousal | 2 × [−1,1] | tanh |
39
+ | Gaze | (yaw, pitch) radians | head-centric; yaw+ = right, pitch+ = up |
40
+ | Face mesh | 478 × (x,y,z) | MediaPipe topology, chip-pixel coords (z = relative depth) |
41
+ | Head pose | (yaw, pitch, roll, tx, ty, tz) | radians / pixels |
42
+ | 68 landmarks | derived | dlib-68 subset sampled from the 478 mesh |
43
+ | Blendshapes | 52 coefficients [0,1] | MediaPipe/ARKit standard names (browInnerUp, jawOpen, mouthSmileLeft, …) |
44
+
45
+ ## Benchmarks (held-out, file-verified — v2.5 deployed checkpoint)
46
+
47
+ | Task | Dataset | Metric | Score |
 
 
 
 
 
 
 
 
 
48
  |---|---|---|---|
49
+ | AU | DISFA+ (12-AU, Cheong protocol) | macro-F1 | **0.693** |
50
+ | AU | DISFA+ (8-AU subset) | macro-F1 | **0.740** |
51
+ | Emotion | RAF-DB official test (7-cls) | acc / macro-F1 | **0.910 / 0.885** |
52
+ | Emotion | AffectNet val (7-cls, drop Contempt) | acc / macro-F1 | **0.616 / 0.612** |
53
+ | Valence/Arousal | Aff-Wild2 official validation | CCC (V / A) | **0.852 / 0.799** |
54
+ | Gaze | MPIIGaze (leave-subject-out) | mean angular err | 7.05° |
55
+ | Gaze | Gaze360 (held-out split) | mean angular err | 12.89° |
56
+
57
+ Notes: **v2.5 = v2.4 architecture + a blendshape head**, and it beats v2.4 on every
58
+ accuracy benchmark — most dramatically AffectNet emotion (acc 0.35→0.62) and
59
+ Aff-Wild2 V/A (0.82/0.78 → 0.85/0.80). **Gaze numbers are now leave-subject-out
60
+ held-out** (honest generalization); the lower v2.4 figures (3.92°/6.81°) came from a
61
+ leaky evaluation that included training subjects, so they are not comparable — the
62
+ v2.5 numbers are the real ones. Numbers are from the deployed checkpoint
63
+ (`v25c_release_ep14`), weight-verified against the published `.safetensors`.
64
 
65
  ## Usage
66
 
67
  ```python
68
+ from feat import Detectorv2
69
+ detector = Detectorv2(device="cuda")
70
+ fex = detector.detect("image.jpg") # returns a py-feat Fex
71
  ```
72
 
73
+ The model expects a face crop produced by RetinaFace + py-feat's
74
+ `extract_face_from_bbox_torch(frame, bbox, face_size=256, expand_bbox=1.2)`,
75
+ then center-cropped to 224 and ImageNet-normalized. `Detectorv2` handles this.
76
+
77
+ ## License
78
 
79
+ **Research / non-commercial use only.** Trained on datasets (AffectNet, DISFA+,
80
+ RAF-DB, Aff-Wild2, BP4D, etc.) whose licenses restrict use to academic research.
81
+ The ConvNeXt-V2 backbone is MIT-licensed. Confirm each constituent dataset's
82
+ terms before any non-research use.
face_multitask_v2.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:076901ef580497066d0bee7d7d451e85c1a4554befd97e5d1b240a9ed6c8edde
3
+ size 153026660