facesim / README.md
seanmamasde's picture
FaceSIM weights: main model, sync-only, DINOv2-B/L, FEC-supervised reference
f958f2b verified
|
Raw History Blame Contribute Delete
2.03 kB
metadata
license: other
tags:
  - facial-expression
  - perceptual-similarity
  - vision-transformer

FaceSIM weights (unpublished; do not redistribute)

Expression-similarity encoders fine-tuned on Ava-256 avatar-capture correspondences (synchronized views, and cross-identity neighbours from the released avatar expression codes). Code: GitHub seanmamasde/facesim (private).

The metric is the cosine distance between backbone CLS embeddings of two images, d(x, y) = 1 - cos(f(x), f(y)), on full frames resized (squashed) to 224×224, inputs in [0, 1] (the backbone wrapper applies its own normalization; the ImageNet ViT deliberately uses none).

folder model seeds FE70k SRCC (all / held-out) FEC acc.
imagenet_vitb16/ours_s{1,2,3} ImageNet ViT-B/16, synchronized views + code neighbours (main model) 3 0.865 / 0.856 44.3
imagenet_vitb16/sync_only_s{1,2,3} ImageNet ViT-B/16, synchronized views only 3 0.842 / 0.829 43.7
dinov2_vitb14/sync_only_s{1,2,3} DINOv2 ViT-B/14, synchronized views only, layer-wise lr decay 3 0.805 / 0.805 57.8
dinov2_vitl14/sync_only_s{1,2} DINOv2 ViT-L/14, synchronized views only 2 0.822 / 0.824 61.0
imagenet_vitb16/fec_supervised_reference_s{1,2,3} FEC-supervised ViT reference (human FEC triplets, FECNet triplet loss, margins 0.1 / 0.2) 3 0.793 / 0.788 78.6

Each folder: final.pth (state dict of the student: backbone.*, projector.*, predictor.*; for the FEC reference only backbone.*), args.json (training arguments), metrics.json (FE70k all / dev / held-out actors, FEC test). All checkpoints are the last training step (no checkpoint selection), except the FEC reference, which is selected on FEC development triplets.

# with the GitHub code on PYTHONPATH
from eval_ckpt import load_backbone
f = load_backbone("imagenet_vitb16/ours_s1/final.pth").cuda().eval()   # reads args.json next to the file
z = f(images)   # images: float [B, 3, 224, 224] in [0, 1]