CLIP ViT-B/16 task experts: 8 vision tasks, 10 epochs

Full fine-tunes of the image encoder of openai/clip-vit-base-patch16 on the 8-task vision benchmark commonly used for model merging and task arithmetic: SUN397, Stanford Cars, RESISC45, EuroSAT, GTSRB, DTD, Food101 and FER2013. There are 10 runs: one per task, plus a second learning rate for EuroSAT and Stanford Cars. Every epoch was checkpointed, so the repo also supports checkpoint-selection and training-duration studies.

Training recipe

All runs use the same recipe. Only the learning rate differs.

  • Model: a CLIP image encoder followed by a linear head, logits = head(normalize(clip.get_image_features(x))).
  • Head: initialized from the zero-shot text embeddings of the class prompts times logit_scale, with zero bias, and frozen. Only the image tower (vision_model and visual_projection) is trained. The text tower in the checkpoint is unchanged from the base model.
  • Optimizer: AdamW (torch fused), betas (0.9, 0.999), weight decay 0, max grad norm 1.0.
  • Schedule: linear decay with 10% warmup over 10 epochs, batch size 32, fp16 mixed precision, seed 42.
  • Preprocessing: the CLIP image processor only (resize to 224, center crop, normalize), with no data augmentation. The training log prints Data augmentation: True, but that flag is not applied anywhere.
  • Evaluation: once per epoch on the evaluation split (the test split, or validation for Food101, which has no test split).
  • Model selection: load_best_model_at_end=True with metric_for_best_model=accuracy, evaluated on that same split.
  • Software: transformers 4.57.6 and torch 2.9.1.

Note on model selection. The top-level model.safetensors of each run is the best epoch by accuracy on the evaluation split, and that split is the test set for 7 of the 8 tasks. Its accuracy is therefore an optimistic number. For a selection-free comparison, use a fixed epoch (for example checkpoint-<10 x steps/epoch>, the last epoch) from the tables below.

Layout

<task>_openai_clip-vit-base-patch16_ep10_bs32_lr<LR>/
  model.safetensors          # best epoch (see table), weights of the wrapper: model.* = CLIPModel, classifier.* = frozen head
  checkpoint-<step>/         # one per epoch, step = epoch x steps/epoch; weights + trainer_state.json (optimizer state not saved)
  train.log                  # raw training log
  runs/                      # tensorboard events
  preprocessor_config.json, tokenizer files, training_args.bin

Results

Accuracy is in %, on the evaluation split described above.

Folder Task LR Train Eval Steps/epoch model.safetensors acc Best epoch Epoch-10 acc
dtd_openai_clip-vit-base-patch16_ep10_bs32_lr3e-5/ dtd 3e-5 3,760 1,880 (test) 118 79.84 10 (checkpoint-1180) 79.84
eurosat_openai_clip-vit-base-patch16_ep10_bs32_lr1e-5/ eurosat 1e-5 21,600 2,700 (test) 675 99.15 9 (checkpoint-6075) 99.04
eurosat_openai_clip-vit-base-patch16_ep10_bs32_lr3e-5/ eurosat 3e-5 21,600 2,700 (test) 675 98.89 10 (checkpoint-6750) 98.89
fer2013_openai_clip-vit-base-patch16_ep10_bs32_lr3e-5/ fer2013 3e-5 28,709 7,178 (test) 898 72.14 10 (checkpoint-8980) 72.14
food101_openai_clip-vit-base-patch16_ep10_bs32_lr3e-5/ food101 3e-5 75,750 25,250 (validation) 2368 89.94 10 (checkpoint-23680) 89.94
gtsrb_openai_clip-vit-base-patch16_ep10_bs32_lr3e-5/ gtsrb 3e-5 26,640 12,630 (test) 833 99.07 8 (checkpoint-6664) 98.99
resisc45_openai_clip-vit-base-patch16_ep10_bs32_lr1e-5/ resisc45 1e-5 18,900 6,300 (test) 591 96.60 10 (checkpoint-5910) 96.60
stanford-cars_openai_clip-vit-base-patch16_ep10_bs32_lr1e-5/ stanford-cars 1e-5 8,144 8,041 (test) 255 88.37 9 (checkpoint-2295) 88.37
stanford-cars_openai_clip-vit-base-patch16_ep10_bs32_lr3e-5/ stanford-cars 3e-5 8,144 8,041 (test) 255 87.27 10 (checkpoint-2550) 87.27
sun397_openai_clip-vit-base-patch16_ep10_bs32_lr3e-5/ sun397 3e-5 19,850 19,850 (test) 621 74.61 10 (checkpoint-6210) 74.61

Accuracy per epoch

Task LR ep1 ep2 ep3 ep4 ep5 ep6 ep7 ep8 ep9 ep10
dtd 3e-5 66.70 70.32 71.49 73.99 75.64 76.86 77.93 78.67 79.47 79.84
eurosat 1e-5 96.52 97.85 98.67 98.67 97.74 98.89 98.78 98.89 99.15 99.04
eurosat 3e-5 94.85 97.63 97.37 97.04 97.78 98.26 97.81 98.63 98.78 98.89
fer2013 3e-5 61.99 63.68 68.50 69.21 71.01 69.20 70.30 70.20 71.50 72.14
food101 3e-5 78.66 82.11 84.93 85.04 84.95 86.29 87.31 88.43 89.41 89.94
gtsrb 3e-5 94.15 97.17 97.37 98.58 97.78 98.57 98.91 99.07 98.92 98.99
resisc45 1e-5 88.79 93.49 94.30 94.52 95.59 95.51 96.48 96.51 96.49 96.60
stanford-cars 1e-5 71.62 77.13 81.49 83.39 85.00 85.67 86.93 87.97 88.37 88.37
stanford-cars 3e-5 62.90 74.63 74.62 77.93 81.12 83.12 82.85 86.17 86.28 87.27
sun397 3e-5 63.83 66.48 66.29 67.29 67.93 70.30 71.48 73.40 74.44 74.61

Loading

import torch
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
from transformers import CLIPModel, CLIPImageProcessor

repo = "theblackcat102/clip-vit-base-patch16-task-experts-ep10"
run = "dtd_openai_clip-vit-base-patch16_ep10_bs32_lr3e-5"   # or f"{run}/checkpoint-<step>/model.safetensors"
sd = load_file(hf_hub_download(repo, f"{run}/model.safetensors"))

clip = CLIPModel.from_pretrained("openai/clip-vit-base-patch16")
clip.load_state_dict({k[len("model."):]: v for k, v in sd.items() if k.startswith("model.")})
W, b = sd["classifier.weight"], sd["classifier.bias"]          # [num_classes, 512], frozen zero-shot head
processor = CLIPImageProcessor.from_pretrained("openai/clip-vit-base-patch16")

@torch.no_grad()
def predict(images):
    x = processor(images=[im.convert("RGB") for im in images], return_tensors="pt")["pixel_values"]
    f = clip.get_image_features(pixel_values=x)
    return ((f / f.norm(dim=-1, keepdim=True)) @ W.T + b).argmax(-1)

We checked this snippet on the DTD run: it gets 79.79% on the 1,880 DTD test images, against 79.84% in the training log (fp16 eval). The class order follows the label feature of each dataset listed above.

For task vectors, use the difference between vision_model.* / visual_projection.* and the same keys of openai/clip-vit-base-patch16.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for theblackcat102/clip-vit-base-patch16-task-experts-ep10

Finetuned
(68)
this model

Datasets used to train theblackcat102/clip-vit-base-patch16-task-experts-ep10