Add CAM-AH ensemble-5 original weights (official Macro-F1 0.7456 @ thr 0.595)

README.md

@@ -0,0 +1,135 @@
+---
+tags:
+  - audio
+  - text
+  - multimodal
+  - cross-attention
+  - multiple-instance-learning
+  - ambivalence
+  - hesitancy
+  - abaw
+  - pytorch-lightning
+---
+
+# CAM-AH — Cross-Attention-Multimodal-Ambivalence-Hesitation
+
+Original weights of the model behind our **best official submission** to the
+*AH Video Recognition Challenge, 3rd edition (ABAW11 @ ECCV 2026)*: a **5-seed ensemble** of an
+audio ⟷ text cross-attention network with psycholinguistic support features and gated
+attention-MIL pooling, run at a **fixed decision threshold of 0.595**.
+
+| Official private test (152 videos) | Value |
+|---|---|
+| Macro-F1 | **0.7456** |
+| Average precision (class 1) | 0.8182 |
+| Decision threshold | 0.595 (60 predicted positives) |
+
+- Paper: [*Audio-Text Cross-Attention with Psycholinguistic Support Features for Ambivalence/Hesitancy Recognition*](https://arxiv.org/abs/2607.13345)
+- Code: [Liga-de-IA-PUCPR/abaw-11-ah-challenge](https://github.com/Liga-de-IA-PUCPR/abaw-11-ah-challenge)
+
+## Files
+
+```
+members/<run>/best-epoch=XX-val_ap=0.XXXX.ckpt   # original PyTorch Lightning checkpoints (5 members)
+members/<run>/trainer_state.json                 # per-member dims / fusion options
+ensemble_config.json                             # members, combination rule, threshold, hyperparameters
+```
+
+Each checkpoint is the unmodified Lightning file (weights + optimizer/loop state). The model
+weights live in `state_dict` under the `model.` prefix, about 2.96M parameters per member.
+The `threshold` field of each `trainer_state.json` is a per-member calibration. The ensemble
+ignores it and uses the fixed 0.595 from `ensemble_config.json`.
+
+## Model
+
+Each video is sliced into **5 s windows (hop 2.5 s)** aligned to the Whisper transcript by
+timestamp. Each window gets three feature vectors:
+
+| Input | Dim | Source |
+|---|---|---|
+| `feat_a` audio | 320 | librosa prosodic features (80 features × 4 statistics) |
+| `feat_b` text | 768 | [`cardiffnlp/twitter-roberta-base-emotion`](https://huggingface.co/cardiffnlp/twitter-roberta-base-emotion), masked mean pool + L2 |
+| `feat_tab` support | 74 | question type (7) + acoustic hesitation (18) + textual A/H cues (49) |
+
+Per member, with inputs of shape `(B, T, ·)` and a padding mask `(B, T)`:
+
+1. Project audio and text to 512-d. Audio is the query and text is the key/value in an 8-head
+   cross-attention, followed by a residual connection and LayerNorm.
+2. **Token fusion.** The support features go through a masked BatchNorm, then Linear + ReLU +
+   LayerNorm, and are concatenated with each window token before a fusion layer (512-d).
+3. **Gated attention-MIL pooling** (Ilse et al., 2018) over the windows produces one vector
+   per video.
+4. An MLP head (512 → 512 → 1, dropout 0.3) outputs one logit per video.
+
+**Ensemble:** average the 5 members' sigmoid probabilities. A video is predicted positive
+(`1`) when the mean probability is at least **0.595**.
+
+## Training
+
+- Data: BAH `train` + `val` splits (902 videos). Checkpoint selection and early stopping
+  used AP on the internal `test` split (525 videos); this is the `val_ap` in the file names.
+- Members: 5 identical models with seeds `{42, 1, 2, 3, 4}`.
+- Optimization: AdamW, lr 3e-4, weight decay 0.05, gradient clip 1.0, batch size 32, early
+  stopping patience 20. The models saturate in 1–2 epochs.
+- Threshold: the ranking comes from the ensemble, and the operating point was fixed at 0.595
+  for the official test set. Its positive rate is lower than that of the internal
+  calibration split, and the original calibrated threshold was 0.45 (official Macro-F1 0.6837).
+
+## Usage
+
+The features have to be produced by the project's pipeline (`make data`) or by the
+self-contained notebook `notebooks/ensemble5_pipeline.ipynb` in the code repository. The
+network definition is `src/models/cross_attention.py` (`_CrossAttentionFusionModule`).
+
+```python
+import json, torch
+from huggingface_hub import snapshot_download
+
+path = snapshot_download("LF-BF/CAM-AH")
+cfg = json.load(open(f"{path}/ensemble_config.json"))
+
+def load_member(ckpt_file, module):
+    sd = torch.load(f"{path}/{ckpt_file}", map_location="cpu", weights_only=False)["state_dict"]
+    module.load_state_dict({k.removeprefix("model."): v for k, v in sd.items()})
+    return module.eval()
+
+# members = [load_member(f, build_module(cfg["architecture"])) for f in cfg["members"]]
+# with torch.no_grad():
+#     p = torch.stack([m(feat_a, feat_b, pad_mask, feat_tab).sigmoid() for m in members]).mean(0)
+# pred = (p >= cfg["threshold"]).long()
+```
+
+## Data and terms of use
+
+The model was trained on the **BAH dataset** (Behavioural Ambivalence/Hesitancy). The
+dataset is **not** included here and is distributed only by its authors under their EULA
+([github.com/sbelharbi/bah-dataset](https://github.com/sbelharbi/bah-dataset)). These
+weights are intended for non-commercial academic research.
+
+This work was carried out using the BAH dataset: a Multimodal Dataset – created and managed by
+the Montréal Behavioural Medicine Centre (MBMC: https://mbmc-cmcm.ca) and the Imaging, Vision
+and Artificial Intelligence Laboratory (LIVIA: https://www.etsmtl.ca/laboratoires/livia).
+
+## Citation
+
+```bibtex
+@article{martins2026audiotext,
+  title   = {Audio-Text Cross-Attention with Psycholinguistic Support Features
+             for Ambivalence/Hesitancy Recognition},
+  author  = {Martins, Luiz F. B. F. and Pisaia, Rodrigo W. and Girardi, Matheus M.
+             and Berkembrock, Isabella and Almeida, Jo{\~a}o A. and Hochuli, Andr{\'e} G.
+             and Laroca, Rayson and Britto Jr., Alceu S.},
+  journal = {arXiv preprint arXiv:2607.13345},
+  year    = {2026},
+  doi     = {10.48550/arXiv.2607.13345},
+  url     = {https://arxiv.org/abs/2607.13345}
+}
+
+@inproceedings{gonzalez-26-bah,
+  title     = {{BAH} Dataset for Ambivalence/Hesitancy Recognition in Videos for Digital Behavioural Change},
+  author    = {González-González, M. and Belharbi, S. and Zeeshan, M. O. and Sharafi, M. and
+               Aslam, M. H and Pedersoli, M. and Koerich, A. L. and Bacon, S. L. and Granger, E.},
+  booktitle = {ICLR},
+  year      = {2026}
+}
+```

ensemble_config.json

@@ -0,0 +1,53 @@
+{
+  "name": "CAM-AH",
+  "full_name": "Cross-Attention-Multimodal-Ambivalence-Hesitation",
+  "task": "video-level binary classification (1 = ambivalence/hesitancy present)",
+  "members": [
+    "members/20260711_191444/best-epoch=00-val_ap=0.8790.ckpt",
+    "members/20260711_191555/best-epoch=00-val_ap=0.8749.ckpt",
+    "members/20260711_191626/best-epoch=01-val_ap=0.8639.ckpt",
+    "members/20260711_191656/best-epoch=00-val_ap=0.8769.ckpt",
+    "members/20260711_191725/best-epoch=01-val_ap=0.8747.ckpt"
+  ],
+  "seeds": [42, 1, 2, 3, 4],
+  "state_dict_prefix": "model.",
+  "combine": "mean of per-video sigmoid probabilities",
+  "threshold": 0.595,
+  "threshold_note": "Fixed decision threshold used in the best official submission (predict 1 if mean probability >= 0.595). Overrides the per-member thresholds stored in each trainer_state.json.",
+  "architecture": {
+    "dim_a": 320,
+    "dim_b": 768,
+    "dim_tab": 74,
+    "common_dim": 512,
+    "num_heads": 8,
+    "dropout": 0.3,
+    "pool": "attention",
+    "tab_fusion": "token",
+    "use_tabular": true,
+    "num_outputs": 1
+  },
+  "inputs": {
+    "window": {"size_s": 5.0, "hop_s": 2.5},
+    "feat_a": "librosa prosodic features per window (80 features x 4 stats = 320-d)",
+    "feat_b": "cardiffnlp/twitter-roberta-base-emotion, masked mean pool + L2 norm, max_length 128 (768-d)",
+    "feat_tab": "74 support features per window: question type (7) + acoustic hesitation (18) + textual A/H cues (49)",
+    "key_padding_mask": "(B, T) bool, True = padded window"
+  },
+  "training": {
+    "train_splits": ["train", "val"],
+    "n_train_videos": 902,
+    "monitor_split": "test (internal, 525 videos)",
+    "monitor": "val_ap (max)",
+    "optimizer": "AdamW",
+    "lr": 3e-4,
+    "weight_decay": 0.05,
+    "gradient_clip_val": 1.0,
+    "batch_size": 32,
+    "max_epochs": 400,
+    "early_stopping_patience": 20
+  },
+  "results": {
+    "official_private_test": {"n_videos": 152, "macro_f1": 0.745595, "average_precision": 0.818211, "threshold": 0.595},
+    "internal_test_525": {"average_precision": 0.877}
+  }
+}

members/20260711_191444/best-epoch=00-val_ap=0.8790.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5c1fc408b39eab9461acd7ce76d7acedd68bc4ac03979e39f9493b4f4f15e952
+size 35608709

members/20260711_191444/trainer_state.json

@@ -0,0 +1,10 @@
+{
+  "threshold": 0.5076700448989868,
+  "ckpt_path": "/Users/lfbf/Documents/Projetos_Academicos/Liga_IA_OneDrive/abaw-11-ah-challenge/outputs/cross_attention/20260711_191444/checkpoints/best-epoch=00-val_ap=0.8790.ckpt",
+  "dim_a": 320,
+  "dim_b": 768,
+  "dim_tab": 74,
+  "use_tabular": true,
+  "pool": "attention",
+  "tab_fusion": "token"
+}

members/20260711_191555/best-epoch=00-val_ap=0.8749.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8c6dcf8cba25a41a5644a025e3935cd8f9d8c446843741ad8cd986888c514f07
+size 35608709

members/20260711_191555/trainer_state.json

@@ -0,0 +1,10 @@
+{
+  "threshold": 0.24121101200580597,
+  "ckpt_path": "/Users/lfbf/Documents/Projetos_Academicos/Liga_IA_OneDrive/abaw-11-ah-challenge/outputs/cross_attention/20260711_191555/checkpoints/best-epoch=00-val_ap=0.8749.ckpt",
+  "dim_a": 320,
+  "dim_b": 768,
+  "dim_tab": 74,
+  "use_tabular": true,
+  "pool": "attention",
+  "tab_fusion": "token"
+}

members/20260711_191626/best-epoch=01-val_ap=0.8639.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4705a333555e4f61d0e03134e6a3c6b036310c37e2b6f94ded19731588d3a7e3
+size 35608709

members/20260711_191626/trainer_state.json

@@ -0,0 +1,10 @@
+{
+  "threshold": 0.4241614043712616,
+  "ckpt_path": "/Users/lfbf/Documents/Projetos_Academicos/Liga_IA_OneDrive/abaw-11-ah-challenge/outputs/cross_attention/20260711_191626/checkpoints/best-epoch=01-val_ap=0.8639.ckpt",
+  "dim_a": 320,
+  "dim_b": 768,
+  "dim_tab": 74,
+  "use_tabular": true,
+  "pool": "attention",
+  "tab_fusion": "token"
+}

members/20260711_191656/best-epoch=00-val_ap=0.8769.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:629eefa76235685943d39b8133437db82448a4f79df26b301c3fde6d24792d03
+size 35608709

members/20260711_191656/trainer_state.json

@@ -0,0 +1,10 @@
+{
+  "threshold": 0.2708229422569275,
+  "ckpt_path": "/Users/lfbf/Documents/Projetos_Academicos/Liga_IA_OneDrive/abaw-11-ah-challenge/outputs/cross_attention/20260711_191656/checkpoints/best-epoch=00-val_ap=0.8769.ckpt",
+  "dim_a": 320,
+  "dim_b": 768,
+  "dim_tab": 74,
+  "use_tabular": true,
+  "pool": "attention",
+  "tab_fusion": "token"
+}

members/20260711_191725/best-epoch=01-val_ap=0.8747.ckpt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d4b9e223a961a44aacf8ea85fac0cb06034d7285bcc238a9fe90bac84e660419
+size 35608709

members/20260711_191725/trainer_state.json

@@ -0,0 +1,10 @@
+{
+  "threshold": 0.5483624935150146,
+  "ckpt_path": "/Users/lfbf/Documents/Projetos_Academicos/Liga_IA_OneDrive/abaw-11-ah-challenge/outputs/cross_attention/20260711_191725/checkpoints/best-epoch=01-val_ap=0.8747.ckpt",
+  "dim_a": 320,
+  "dim_b": 768,
+  "dim_tab": 74,
+  "use_tabular": true,
+  "pool": "attention",
+  "tab_fusion": "token"
+}