| --- |
| license: mit |
| tags: |
| - pytorch-lightning |
| - image-captioning |
| - vision |
| --- |
| |
| # CLIPCap fine-tuned for Human-Human Interaction captioning (cc_ft) |
| |
| This is the best-performing checkpoint from **"Learning Human-Human Interactions in Images from Weak Textual Supervision" (ICCV 2023)**: a [CLIPCap](https://github.com/rmokady/CLIP_prefix_caption) model fine-tuned on pseudo-labels (pHHI) for describing human-human interactions (HHI) in images. |
| |
| - **Paper:** [arXiv:2304.14104](https://arxiv.org/abs/2304.14104) |
| - **Code:** [github.com/tau-vailab/learning-interactions](https://github.com/tau-vailab/learning-interactions) |
| - **Project page:** https://learning-interactions.github.io/ |
| |
| Note: this checkpoint has been computed from scratch and may differ slightly from the results reported in the paper due to randomness during training. |
| |
| ## Usage |
| |
| This checkpoint is a PyTorch Lightning checkpoint (`cc_ft.ckpt`), intended for use with the inference code in the repo above (`modeling/src/test.py`): |
|
|
| ```bash |
| hf download malper/learning-interactions-cc-ft cc_ft.ckpt --local-dir . |
| python src/test.py -d <image_dir> -o <output.json> -c cc_ft.ckpt -cc <clipcap_pretrained_weights> |
| ``` |
|
|
| See the [modeling documentation](https://github.com/tau-vailab/learning-interactions/tree/main/modeling) for full details, including the required CLIPCap pretrained MLP mapping network weights. |
|
|
| ## Context |
|
|
| This is research code from 2023, prior to the widespread availability of general-purpose vision-language models (VLMs). It is provided as-is for reproducibility of the paper's results. |
|
|
| ## Citation |
|
|
| ```bibtex |
| @InProceedings{alper2023learning, |
| author = {Morris Alper and Hadar Averbuch-Elor}, |
| title = {Learning Human-Human Interactions in Images from Weak Textual Supervision}, |
| booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)}, |
| year = {2023} |
| } |
| ``` |
|
|
