Duplicate from IVPL/CaReDiff

Co-authored-by: Jihoon Choi <JihoonChoi@users.noreply.huggingface.co>

.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 IVPL
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -0,0 +1,186 @@
+---
+license: mit
+tags:
+  - facial-reaction-generation
+  - dyadic-interaction
+  - diffusion-model
+  - affective-computing
+  - react2026
+  - mafrg
+language:
+  - en
+pipeline_tag: other
+---
+
+# CaReDiff — Causal Reaction Diffusion (REACT 2026)
+
+
+<img src="https://cdn-uploads.huggingface.co/production/uploads/65fdba70f485b5180897c642/q5J2ScpexbMFlRiEllA4p.png" alt="react_challenge_figure" width="600">
+
+Model checkpoints for **CaReDiff**, a submission to the [REACT 2026 Challenge](https://sites.google.com/view/react2026/home) (ACM MM 2026, Multiple Appropriate Facial Reaction Generation).
+
+Given speaker behaviour (audio, video, 3DMM coefficients, facial attributes), CaReDiff generates multiple appropriate listener facial reactions (25-d: 15 AUs + valence/arousal + 8 expressions) with an auxiliary EEG prediction head.
+
+- **Code:** https://github.com/smu-ivpl/CaReDiff
+- **Checkpoints (this repo):** https://huggingface.co/IVPL/CaReDiff
+
+## Tracks
+
+| Track | Architecture | Description |
+|-------|-------------|-------------|
+| Generic Online | PerFRDiff + EEG | Diffusion-based generation over autoregressive windows |
+| Generic Offline | PerFRDiff + EEG | Diffusion-based full-sequence generation |
+| Personalised Online | PerFRDiff + PRA + EEG | Frozen generic backbone + Personalised Residual Adapter (autoregressive windows) |
+| Personalised Offline | PerFRDiff + PRA + EEG | Frozen generic backbone + Personalised Residual Adapter (full-sequence) |
+
+## Repository Layout
+
+```
+CaReDiff/
+├── generic/
+│   ├── online/          prior + denoiser + EEG head (checkpoint_120.pth)
+│   └── offline/         prior + denoiser + EEG head (checkpoint_120.pth)
+└── personalised/
+    ├── online/          shared backbone + 3 adapters (personality / lhfb / both)
+    └── offline/         shared backbone + 3 adapters (personality / lhfb / both)
+```
+
+Each generic track folder contains:
+
+```
+<track>/
+├── DiffusionPriorNetwork/checkpoint_120.pth
+├── CausalTransformerDenoiser/checkpoint_120.pth
+└── EEGPredictionHead/checkpoint_120.pth
+```
+
+## Usage
+
+1. Clone the code repository:
+   ```bash
+   git clone https://github.com/smu-ivpl/CaReDiff
+   ```
+2. Set up the environment (see the [code README](https://github.com/smu-ivpl/CaReDiff)):
+   ```bash
+   conda create -n react python=3.10 && conda activate react
+   conda install pytorch==2.0.0 torchvision==0.15.0 torchaudio==2.0.0 pytorch-cuda=11.8 -c pytorch -c nvidia
+   conda install -c fvcore -c iopath -c conda-forge fvcore iopath
+   pip install -r generic/code/requirements.txt
+   ```
+3. Download the checkpoints from this repo and place them under `save/`. For the generic tracks:
+   ```
+   save/motion_diffusion/react_2025/
+   ├── online/checkpoints/pretrained/
+   │   ├── DiffusionPriorNetwork/checkpoint_120.pth
+   │   ├── CausalTransformerDenoiser/checkpoint_120.pth
+   │   └── EEGPredictionHead/checkpoint_120.pth
+   └── offline/checkpoints/pretrained/
+       └── ... (same structure)
+   ```
+4. Run evaluation (see below).
+
+Full placement instructions, SHA-256 checksums, and per-track details are in each variant's `checkpoints/README.md` in the code repository.
+
+### Generic Evaluation
+
+```bash
+# Generic Online
+python main.py --config-name generic_online/motion_diffusion \
+    stage=test data_dir=./datasets/REACT2026/ \
+    trainer.batch_size=1 resume_id=pretrained
+
+# Generic Offline
+python main.py --config-name generic_offline/motion_diffusion \
+    stage=test data_dir=./datasets/REACT2026/ \
+    trainer.batch_size=1 resume_id=pretrained
+```
+
+### Personalised Evaluation
+
+Each personalised track uses a frozen generic backbone plus one of three
+condition adapters (`personality` / `lhfb` / `both`). Download the
+`personalised/<track>/` folder and set `PKG` to its parent absolute path; the
+commands reference every weight explicitly through `PKG`.
+
+```bash
+PKG=<absolute path of the downloaded personalised folder>
+
+# Personalised Offline (personality condition)
+python main.py --config-name g2p_delta stage=test task=offline \
+  data_dir=<MARS_ROOT> run_id=eval_offline_personality \
+  trainer.batch_size=4 num_gts=10 \
+  trainer.generic.eval_condition_mode=matched \
+  trainer.generic.eval_eeg=false \
+  trainer.main_model.args.personal_condition_mode=personality_only \
+  resume_id=personality \
+  trainer.ckpt_dir=$PKG/offline/adapters \
+  trainer.pretrained.diffusion_decoder=$PKG/offline/backbone/CausalTransformerDenoiser/checkpoint_120.pth \
+  trainer.pretrained.diffusion_prior=$PKG/offline/backbone/DiffusionPriorNetwork/checkpoint_120.pth \
+  trainer.pretrained.eeg_head_checkpoint=$PKG/offline/backbone/EEGPredictionHead/checkpoint_120.pth
+
+# Personalised Online (personality condition)
+python main.py --config-name g2p_delta_online stage=test task=online \
+  data_dir=<MARS_ROOT> run_id=eval_online_personality \
+  trainer.batch_size=4 num_gts=10 \
+  trainer.generic.eval_condition_mode=matched \
+  trainer.generic.eval_eeg=false \
+  trainer.main_model.args.personal_condition_mode=personality_only \
+  trainer.model.diff_model.diffusion_decoder.args.past_l_emotion_drop_prob=0.2 \
+  resume_id=personality \
+  trainer.ckpt_dir=$PKG/online/adapters \
+  trainer.pretrained.diffusion_decoder=$PKG/online/backbone/CausalTransformerDenoiser/checkpoint_120.pth \
+  trainer.pretrained.diffusion_prior=$PKG/online/backbone/DiffusionPriorNetwork/checkpoint_120.pth \
+  trainer.pretrained.eeg_head_checkpoint=$PKG/online/backbone/EEGPredictionHead/checkpoint_120.pth
+```
+
+For the `lhfb` / `both` conditions, change `resume_id` (`lhfb` or `both`) and
+`personal_condition_mode` (`3dmm_only` or `3dmm_personality`) accordingly. For
+the online track, keep `past_l_emotion_drop_prob=0.2` — it enables the
+past-listener conditioning pathway and is required to reproduce the reported
+numbers.
+
+## Additional Requirements
+
+- **Post-processor (EmotionVAE) checkpoint** — required for evaluation on every `stage=test` run. Obtain from the [official REACT 2026 baseline](https://github.com/reactmultimodalchallenge/baseline_react2026) and place at `pretrained_models/post_processor/checkpoint.pth`.
+- **PIRender + FaceVerse** — needed only for FRRea (rendered-frame FID) evaluation.
+- **MARS dataset** — obtain through the challenge organisers.
+
+## Metrics
+
+| Metric | Description |
+|--------|-------------|
+| FRCorr ↑ | Facial Reaction Correlation (CCC against GT) |
+| FRDist ↓ | Facial Reaction Distance (DTW against GT) |
+| FRDiv ↑ | Diversity across the 10 generated predictions (pairwise MSE) |
+| FRVar ↑ | Temporal variance within a generated reaction |
+| FRRea ↓ | Realism (FID on rendered frames) |
+| FRSyn ↓ | Synchrony (Time-Lagged Cross-Correlation) |
+
+## Citation
+
+```bibtex
+@article{song2023multiple,
+  title={Multiple Appropriate Facial Reaction Generation in Dyadic Interaction Settings: What, Why and How?},
+  author={Song, Siyang and Spitale, Micol and Luo, Yiming and Bal, Batuhan and Gunes, Hatice},
+  journal={arXiv preprint arXiv:2302.06514},
+  year={2023}
+}
+
+@inproceedings{song2025react,
+  title={React 2025: the third multiple appropriate facial reaction generation challenge},
+  author={Song, Siyang and Spitale, Micol and Kong, Xiangyu and Zhu, Hengde and Luo, Cheng and Palmero, Cristina and Barquero, German and others},
+  booktitle={Proceedings of the 33rd ACM International Conference on Multimedia},
+  pages={13979--13984},
+  year={2025}
+}
+```
+
+## Acknowledgement
+
+- [FaceVerse](https://github.com/LizhenWangT/FaceVerse)
+- [PIRender](https://github.com/RenYurui/PIRender)
+- [REACT 2026 Baseline](https://github.com/reactmultimodalchallenge/baseline_react2026)
+
+## License
+
+MIT

generic/offline/CausalTransformerDenoiser/checkpoint_120.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:68faca9700415c949eecbe7bd3e381877a76b5e1b24bdab9c30e6fd5b628faa2
+size 561047170

generic/offline/DiffusionPriorNetwork/checkpoint_120.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d1b66e87f51afd9bb93bdcef1b9e350e6366aa8f995920e400d7e7dd4e299357
+size 481319450

generic/offline/EEGPredictionHead/checkpoint_120.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:750c49999a180cda330b88d771f99d1dca0fd94a810470ea77a45561cfd58780
+size 418238964

generic/online/CausalTransformerDenoiser/checkpoint_120.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f4fc53506fc94a65e86b52bfe1491669a73ca3429ad0b1ab51c62488854242f0
+size 561047170

generic/online/DiffusionPriorNetwork/checkpoint_120.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8b717d619cd37fc793f80f37a4af607bda5e9709c83b82f17916b2467a4380a6
+size 481319450

generic/online/EEGPredictionHead/checkpoint_120.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:60c7a7ae4e6a233fdb59c0ee1e099daf1158931d876a5f46386a781fa2a52a52
+size 418238964

personalised/offline/adapters/both/ModifierNetwork/checkpoint_best.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:73434669c633bc6384acc9845e62c0c4302c9322be27f04e30005e55dda3ab92
+size 12226442

personalised/offline/adapters/lhfb/ModifierNetwork/checkpoint_best.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0ddfde5284c580c3cc461006b2b7cd4df73d2715a8700d3838d8d5e5db8eb7f4
+size 9420474

personalised/offline/adapters/personality/ModifierNetwork/checkpoint_best.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8e0a501237c9b80b8c9e9524bd089fa5ca54ad747bdf9ed65dd97b8d883bf928
+size 6804442

personalised/offline/backbone/CausalTransformerDenoiser/checkpoint_120.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:68faca9700415c949eecbe7bd3e381877a76b5e1b24bdab9c30e6fd5b628faa2
+size 561047170

personalised/offline/backbone/DiffusionPriorNetwork/checkpoint_120.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d1b66e87f51afd9bb93bdcef1b9e350e6366aa8f995920e400d7e7dd4e299357
+size 481319450

personalised/offline/backbone/EEGPredictionHead/checkpoint_120.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:750c49999a180cda330b88d771f99d1dca0fd94a810470ea77a45561cfd58780
+size 418238964

personalised/online/adapters/both/ModifierNetwork/checkpoint_best.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:77a744f486c46484dc7a357c484bd5c3582345ea5f6947aea1aa625ba00e660a
+size 12226442

personalised/online/adapters/lhfb/ModifierNetwork/checkpoint_best.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c5d8c774774b8a1994be3149c3bb475384326e24a9af64edbf15cda0292da868
+size 9420474

personalised/online/adapters/personality/ModifierNetwork/checkpoint_best.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fad7691aea8c1895a11fcc7d40873d83ea44e0692e6dfb75f736d89cdd9e62d0
+size 6804442

personalised/online/backbone/CausalTransformerDenoiser/checkpoint_120.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f4fc53506fc94a65e86b52bfe1491669a73ca3429ad0b1ab51c62488854242f0
+size 561047170

personalised/online/backbone/DiffusionPriorNetwork/checkpoint_120.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8b717d619cd37fc793f80f37a4af607bda5e9709c83b82f17916b2467a4380a6
+size 481319450

personalised/online/backbone/EEGPredictionHead/checkpoint_120.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:60c7a7ae4e6a233fdb59c0ee1e099daf1158931d876a5f46386a781fa2a52a52
+size 418238964