---
license: mit
tags:
- pytorch-lightning
- image-captioning
- vision
---

# CLIPCap fine-tuned for Human-Human Interaction captioning (cc_ft)

This is the best-performing checkpoint from **"Learning Human-Human Interactions in Images from Weak Textual Supervision" (ICCV 2023)**: a [CLIPCap](https://github.com/rmokady/CLIP_prefix_caption) model fine-tuned on pseudo-labels (pHHI) for describing human-human interactions (HHI) in images.

- **Paper:** [arXiv:2304.14104](https://arxiv.org/abs/2304.14104)
- **Code:** [github.com/tau-vailab/learning-interactions](https://github.com/tau-vailab/learning-interactions)
- **Project page:** https://learning-interactions.github.io/

Note: this checkpoint has been computed from scratch and may differ slightly from the results reported in the paper due to randomness during training.

## Usage

This checkpoint is a PyTorch Lightning checkpoint (`cc_ft.ckpt`), intended for use with the inference code in the repo above (`modeling/src/test.py`):

```bash
hf download malper/learning-interactions-cc-ft cc_ft.ckpt --local-dir .
python src/test.py -d <image_dir> -o <output.json> -c cc_ft.ckpt -cc <clipcap_pretrained_weights>
```

See the [modeling documentation](https://github.com/tau-vailab/learning-interactions/tree/main/modeling) for full details, including the required CLIPCap pretrained MLP mapping network weights.

## Context

This is research code from 2023, prior to the widespread availability of general-purpose vision-language models (VLMs). It is provided as-is for reproducibility of the paper's results.

## Citation

```bibtex
@InProceedings{alper2023learning,
    author    = {Morris Alper and Hadar Averbuch-Elor},
    title     = {Learning Human-Human Interactions in Images from Weak Textual Supervision},
    booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
    year      = {2023}
}
```