File size: 2,137 Bytes
50a8c5d 2d0b3c8 50a8c5d 2d0b3c8 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 | ---
license: mit
tags:
- onnx
- facial-expression
- arkit
- vr
- mediapipe
- gru
library_name: onnx
pipeline_tag: other
---
# FaceFill
Causal GRU that predicts **11 upper-face ARKit morphs** (brows / eyes / blinks) from **30 lower-face** channels when an HMD occludes the upper face.
~3.36M params · streaming `step` with GRU hidden state.
## Files
| File | Description |
|------|-------------|
| `model.onnx` | Stateful single-frame step (opset 17) |
| `eval.json` | Autoregressive test metrics (held-out MEAD W009) |
## Inputs / outputs
**Inputs**
| Name | Shape | Meaning |
|------|-------|---------|
| `current` | `[B, 80]` | lower + vel + head + conf + blink state |
| `history` | `[B, 176]` | prior full morphs + vel + obs mask + … |
| `u_prev` | `[B, 11]` | previous upper prediction |
| `hidden_in` | `[3, B, 384]` | GRU hidden |
**Outputs**
| Name | Shape |
|------|-------|
| `u_hat` | `[B, 11]` |
| `u_abs` | `[B, 11]` |
| `u_delta` | `[B, 11]` |
| `blink_onset_logits` | `[B, 3]` |
| `blink_duration` | `[B]` |
| `blink_amplitude` | `[B]` |
| `hidden_out` | `[3, B, 384]` |
Combine rule: `û = clamp(α·(u_prev + Δu) + (1−α)·u_abs)`, α=0.8.
## Training data
Features derived from **[MEAD](https://wywu.github.io/projects/MEAD/MEAD.html)** frontal videos via MediaPipe Face Landmarker (person-disjoint M033 / W015 / W009 for this release). Does **not** redistribute MEAD videos.
## Eval (AR, test)
- mean MAE **0.075**
- mean Pearson **0.82**
- blink F1 **0.73**
## Usage
```python
import onnxruntime as ort
import numpy as np
sess = ort.InferenceSession("model.onnx")
B = 1
feeds = {
"current": np.zeros((B, 80), np.float32),
"history": np.zeros((B, 176), np.float32),
"u_prev": np.zeros((B, 11), np.float32),
"hidden_in": np.zeros((3, B, 384), np.float32),
}
outs = sess.run(None, feeds)
# u_hat, u_abs, u_delta, blink_onset_logits, blink_duration, blink_amplitude, hidden_out
```
Browser / WebXR companion: load with ONNX Runtime Web (see FaceFill / WebXREmotion projects).
## License
MIT (this model). Respect MEAD terms for any further training on that dataset.
|