Idan/fga-ensemble

5xFGA — Factor Graph Attention ensemble

The five members of the 5xFGA row of Factor Graph Attention (CVPR 2019), trained from different seeds on VisDial v1.0 with F-RCNN image features.

Code: github.com/idansc/fga.

Results on VisDial v1.0 val

	NDCG	MRR	R@1	R@5	R@10	Mean rank
best single member	56.07	65.46	51.76	82.51	90.47	4.01
5xFGA, score-averaged	60.86	68.43	55.26	85.06	92.52	3.47
5xFGA, rank-averaged	60.82	67.37	53.90	84.07	92.11	3.56

Published 5xFGA: MRR 69, R@1 56%.

Averaging scores works better than averaging ranks for members of one architecture, whose scores already share a scale. Ranks help when the members disagree in confidence — mixing these with a dense-finetuned model gains 0.7 NDCG that way.

Two things that did not help: selecting each seed's best-MRR checkpoint instead of its last gave 68.27, and stacking all 26 checkpoints of the five runs gave 68.40. The diversity has to come from the seeds.

Usage

from fga import FGAForVisualDialog

members = [FGAForVisualDialog.from_pretrained("Idan/fga-ensemble", subfolder=name)
           for name in ["frcnn", "seed1", "seed2", "seed3", "seed4"]]

Or evaluate the ensemble directly:

python scripts/ensemble_eval.py --models <member dirs> \
    --image_features_path data/frcnn_features_new.h5 --combine score rank

Citation

@inproceedings{schwartz2019factor,
  title={Factor graph attention},
  author={Schwartz, Idan and Yu, Seunghak and Hazan, Tamir and Schwing, Alexander G},
  booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
  pages={2039--2048},
  year={2019}
}