README.md

---
license: mit
tags:
  - onnx
  - facial-expression
  - arkit
  - vr
  - mediapipe
  - gru
library_name: onnx
pipeline_tag: other
---

# FaceFill

Causal GRU that predicts **11 upper-face ARKit morphs** (brows / eyes / blinks) from **30 lower-face** channels when an HMD occludes the upper face.

~3.36M params · streaming `step` with GRU hidden state.

## Files

| File | Description |
|------|-------------|
| `model.onnx` | Stateful single-frame step (opset 17) |
| `eval.json` | Autoregressive test metrics (held-out MEAD W009) |

## Inputs / outputs

**Inputs**

| Name | Shape | Meaning |
|------|-------|---------|
| `current` | `[B, 80]` | lower + vel + head + conf + blink state |
| `history` | `[B, 176]` | prior full morphs + vel + obs mask + … |
| `u_prev` | `[B, 11]` | previous upper prediction |
| `hidden_in` | `[3, B, 384]` | GRU hidden |

**Outputs**

| Name | Shape |
|------|-------|
| `u_hat` | `[B, 11]` |
| `u_abs` | `[B, 11]` |
| `u_delta` | `[B, 11]` |
| `blink_onset_logits` | `[B, 3]` |
| `blink_duration` | `[B]` |
| `blink_amplitude` | `[B]` |
| `hidden_out` | `[3, B, 384]` |

Combine rule: `û = clamp(α·(u_prev + Δu) + (1−α)·u_abs)`, α=0.8.

## Training data

Features derived from **[MEAD](https://wywu.github.io/projects/MEAD/MEAD.html)** frontal videos via MediaPipe Face Landmarker (person-disjoint M033 / W015 / W009 for this release). Does **not** redistribute MEAD videos.

## Eval (AR, test)

- mean MAE **0.075**
- mean Pearson **0.82**
- blink F1 **0.73**

## Usage

```python
import onnxruntime as ort
import numpy as np

sess = ort.InferenceSession("model.onnx")
B = 1
feeds = {
    "current": np.zeros((B, 80), np.float32),
    "history": np.zeros((B, 176), np.float32),
    "u_prev": np.zeros((B, 11), np.float32),
    "hidden_in": np.zeros((3, B, 384), np.float32),
}
outs = sess.run(None, feeds)
# u_hat, u_abs, u_delta, blink_onset_logits, blink_duration, blink_amplitude, hidden_out
```

Browser / WebXR companion: load with ONNX Runtime Web (see FaceFill / WebXREmotion projects).

## License

MIT (this model). Respect MEAD terms for any further training on that dataset.