CLIP ViT-B/16 task experts: 8 vision tasks, 10 epochs
Full fine-tunes of the image encoder of openai/clip-vit-base-patch16
on the 8-task vision benchmark commonly used for model merging and task arithmetic:
SUN397, Stanford Cars, RESISC45, EuroSAT, GTSRB, DTD, Food101 and FER2013.
There are 10 runs: one per task, plus a second learning rate for EuroSAT and Stanford Cars.
Every epoch was checkpointed, so the repo also supports checkpoint-selection and training-duration studies.
Training recipe
All runs use the same recipe. Only the learning rate differs.
- Model: a CLIP image encoder followed by a linear head,
logits = head(normalize(clip.get_image_features(x))). - Head: initialized from the zero-shot text embeddings of the class prompts times
logit_scale, with zero bias, and frozen. Only the image tower (vision_modelandvisual_projection) is trained. The text tower in the checkpoint is unchanged from the base model. - Optimizer: AdamW (torch fused), betas (0.9, 0.999), weight decay 0, max grad norm 1.0.
- Schedule: linear decay with 10% warmup over 10 epochs, batch size 32, fp16 mixed precision, seed 42.
- Preprocessing: the CLIP image processor only (resize to 224, center crop, normalize), with no data augmentation.
The training log prints
Data augmentation: True, but that flag is not applied anywhere. - Evaluation: once per epoch on the evaluation split (the test split, or validation for Food101, which has no test split).
- Model selection:
load_best_model_at_end=Truewithmetric_for_best_model=accuracy, evaluated on that same split. - Software: transformers 4.57.6 and torch 2.9.1.
Note on model selection. The top-level
model.safetensorsof each run is the best epoch by accuracy on the evaluation split, and that split is the test set for 7 of the 8 tasks. Its accuracy is therefore an optimistic number. For a selection-free comparison, use a fixed epoch (for examplecheckpoint-<10 x steps/epoch>, the last epoch) from the tables below.
Layout
<task>_openai_clip-vit-base-patch16_ep10_bs32_lr<LR>/
model.safetensors # best epoch (see table), weights of the wrapper: model.* = CLIPModel, classifier.* = frozen head
checkpoint-<step>/ # one per epoch, step = epoch x steps/epoch; weights + trainer_state.json (optimizer state not saved)
train.log # raw training log
runs/ # tensorboard events
preprocessor_config.json, tokenizer files, training_args.bin
Results
Accuracy is in %, on the evaluation split described above.
| Folder | Task | LR | Train | Eval | Steps/epoch | model.safetensors acc |
Best epoch | Epoch-10 acc |
|---|---|---|---|---|---|---|---|---|
dtd_openai_clip-vit-base-patch16_ep10_bs32_lr3e-5/ |
dtd | 3e-5 | 3,760 | 1,880 (test) | 118 | 79.84 | 10 (checkpoint-1180) |
79.84 |
eurosat_openai_clip-vit-base-patch16_ep10_bs32_lr1e-5/ |
eurosat | 1e-5 | 21,600 | 2,700 (test) | 675 | 99.15 | 9 (checkpoint-6075) |
99.04 |
eurosat_openai_clip-vit-base-patch16_ep10_bs32_lr3e-5/ |
eurosat | 3e-5 | 21,600 | 2,700 (test) | 675 | 98.89 | 10 (checkpoint-6750) |
98.89 |
fer2013_openai_clip-vit-base-patch16_ep10_bs32_lr3e-5/ |
fer2013 | 3e-5 | 28,709 | 7,178 (test) | 898 | 72.14 | 10 (checkpoint-8980) |
72.14 |
food101_openai_clip-vit-base-patch16_ep10_bs32_lr3e-5/ |
food101 | 3e-5 | 75,750 | 25,250 (validation) | 2368 | 89.94 | 10 (checkpoint-23680) |
89.94 |
gtsrb_openai_clip-vit-base-patch16_ep10_bs32_lr3e-5/ |
gtsrb | 3e-5 | 26,640 | 12,630 (test) | 833 | 99.07 | 8 (checkpoint-6664) |
98.99 |
resisc45_openai_clip-vit-base-patch16_ep10_bs32_lr1e-5/ |
resisc45 | 1e-5 | 18,900 | 6,300 (test) | 591 | 96.60 | 10 (checkpoint-5910) |
96.60 |
stanford-cars_openai_clip-vit-base-patch16_ep10_bs32_lr1e-5/ |
stanford-cars | 1e-5 | 8,144 | 8,041 (test) | 255 | 88.37 | 9 (checkpoint-2295) |
88.37 |
stanford-cars_openai_clip-vit-base-patch16_ep10_bs32_lr3e-5/ |
stanford-cars | 3e-5 | 8,144 | 8,041 (test) | 255 | 87.27 | 10 (checkpoint-2550) |
87.27 |
sun397_openai_clip-vit-base-patch16_ep10_bs32_lr3e-5/ |
sun397 | 3e-5 | 19,850 | 19,850 (test) | 621 | 74.61 | 10 (checkpoint-6210) |
74.61 |
Accuracy per epoch
| Task | LR | ep1 | ep2 | ep3 | ep4 | ep5 | ep6 | ep7 | ep8 | ep9 | ep10 |
|---|---|---|---|---|---|---|---|---|---|---|---|
| dtd | 3e-5 | 66.70 | 70.32 | 71.49 | 73.99 | 75.64 | 76.86 | 77.93 | 78.67 | 79.47 | 79.84 |
| eurosat | 1e-5 | 96.52 | 97.85 | 98.67 | 98.67 | 97.74 | 98.89 | 98.78 | 98.89 | 99.15 | 99.04 |
| eurosat | 3e-5 | 94.85 | 97.63 | 97.37 | 97.04 | 97.78 | 98.26 | 97.81 | 98.63 | 98.78 | 98.89 |
| fer2013 | 3e-5 | 61.99 | 63.68 | 68.50 | 69.21 | 71.01 | 69.20 | 70.30 | 70.20 | 71.50 | 72.14 |
| food101 | 3e-5 | 78.66 | 82.11 | 84.93 | 85.04 | 84.95 | 86.29 | 87.31 | 88.43 | 89.41 | 89.94 |
| gtsrb | 3e-5 | 94.15 | 97.17 | 97.37 | 98.58 | 97.78 | 98.57 | 98.91 | 99.07 | 98.92 | 98.99 |
| resisc45 | 1e-5 | 88.79 | 93.49 | 94.30 | 94.52 | 95.59 | 95.51 | 96.48 | 96.51 | 96.49 | 96.60 |
| stanford-cars | 1e-5 | 71.62 | 77.13 | 81.49 | 83.39 | 85.00 | 85.67 | 86.93 | 87.97 | 88.37 | 88.37 |
| stanford-cars | 3e-5 | 62.90 | 74.63 | 74.62 | 77.93 | 81.12 | 83.12 | 82.85 | 86.17 | 86.28 | 87.27 |
| sun397 | 3e-5 | 63.83 | 66.48 | 66.29 | 67.29 | 67.93 | 70.30 | 71.48 | 73.40 | 74.44 | 74.61 |
Loading
import torch
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
from transformers import CLIPModel, CLIPImageProcessor
repo = "theblackcat102/clip-vit-base-patch16-task-experts-ep10"
run = "dtd_openai_clip-vit-base-patch16_ep10_bs32_lr3e-5" # or f"{run}/checkpoint-<step>/model.safetensors"
sd = load_file(hf_hub_download(repo, f"{run}/model.safetensors"))
clip = CLIPModel.from_pretrained("openai/clip-vit-base-patch16")
clip.load_state_dict({k[len("model."):]: v for k, v in sd.items() if k.startswith("model.")})
W, b = sd["classifier.weight"], sd["classifier.bias"] # [num_classes, 512], frozen zero-shot head
processor = CLIPImageProcessor.from_pretrained("openai/clip-vit-base-patch16")
@torch.no_grad()
def predict(images):
x = processor(images=[im.convert("RGB") for im in images], return_tensors="pt")["pixel_values"]
f = clip.get_image_features(pixel_values=x)
return ((f / f.norm(dim=-1, keepdim=True)) @ W.T + b).argmax(-1)
We checked this snippet on the DTD run: it gets 79.79% on the 1,880 DTD test images, against 79.84% in the training log (fp16 eval).
The class order follows the label feature of each dataset listed above.
For task vectors, use the difference between vision_model.* / visual_projection.* and the same keys of openai/clip-vit-base-patch16.
Model tree for theblackcat102/clip-vit-base-patch16-task-experts-ep10
Base model
openai/clip-vit-base-patch16