git-vqav2-medium

microsoft/git-base-vqav2 fine-tuned on VQAv2 (50,000 training questions).

Model

Architecture	GitForCausalLM (GIT-base)
Vision encoder	CLIP ViT-B/16, 12 layers, 480×480 input, 901 visual tokens
Text decoder	6-layer causal Transformer, hidden 768, 12 heads, max 1024 positions
Tokenizer	BERT WordPiece (uncased), vocab 30,522
Parameters	177,159,738 (trainable during fine-tuning: 177,159,738, strategy full)
Weights	fp32, safetensors

Input / output format

• Image: CLIPImageProcessor — resize shortest edge to 480 (bicubic), center crop 480×480, rescale 1/255, normalize with CLIP mean/std.
• Text prompt: [CLS] question (no [SEP] between question and answer).
• Output: the answer tokens generated after the prompt, terminated by [SEP] (eos id 102).
• Training sequence: [CLS] question answer [SEP]; loss only on answer [SEP] (all other positions -100).

Training data

Dataset	HuggingFaceM4/VQAv2, parquet conversion refs/convert/parquet @ ac4a1a047466a7019b5357ce94b6b5794fc9f867
Split	train (COCO train2014 images)
Shards / questions	[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19] / 50,000
Rows skipped (missing question/answer)	0
Target answer	majority vote over the 10 human answers (lower-cased, whitespace/edge punctuation stripped; ties → multiple_choice_answer)
Model selection	1,000 questions from validation shards [2, 7], best vqa_accuracy

Training procedure

Hyperparameter	Value
Epochs	2
Optimizer	AdamW (β=(0.9, 0.999), ε=1e-8), no decay on biases/LayerNorm
Learning rate	2e-06, linear decay, warmup 156 steps
Batch size	8 × 4 accumulation = 32
Weight decay	0.01
Gradient clipping	1.0
Precision	bf16 autocast, fp32 master weights
Vision attention	sdpa
Optimizer steps	3,126
Best checkpoint	step 500 (vqa_accuracy = 84.39)
Seed	42
Hardware	NVIDIA GeForce RTX 5060 Ti
Software	torch 2.11.0+cu128, transformers 5.17.0, datasets 4.3.0

Evaluation

VQAv2 consensus accuracy (official vqaEval.py normalization; per question mean over 10 leave-one-out subsets of min(1, matches/3)) and exact match against the majority answer (not an official metric). Deterministic decoding: max_new_tokens=10, num_beams=1, do_sample=False.

Evaluation set: 500 questions sampled with a fixed seed from validation shard(s) [0] (subset fingerprint 1daf0de4dae278b4), disjoint from the model-selection questions. Both models were evaluated on exactly the same questions.

Metric	microsoft/git-base-vqav2	This model	Δ
VQA accuracy	83.32	82.54	-0.78
Exact match	73.40	72.00	-1.40
VQA accuracy (number)	79.47	77.07	-2.40
VQA accuracy (other)	75.74	74.84	-0.90
VQA accuracy (yes/no)	95.14	95.19	+0.05

Usage

import torch
from PIL import Image
from transformers import AutoProcessor, AutoModelForCausalLM

repo = "thealper2/git-vqav2-medium"
processor = AutoProcessor.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo).eval()

image = Image.open("example.jpg").convert("RGB")
question = "what color is the car?"
pixel_values = processor(images=image, return_tensors="pt").pixel_values
input_ids = [processor.tokenizer.cls_token_id] + processor.tokenizer(question, add_special_tokens=False).input_ids
input_ids = torch.tensor([input_ids])

out = model.generate(pixel_values=pixel_values, input_ids=input_ids, max_new_tokens=10)
print(processor.batch_decode(out[:, input_ids.shape[1]:], skip_special_tokens=True)[0])

Batched generation: GIT does not derive text position ids from the attention mask, so padded prompts change the output. Batch only prompts of equal token length.

Limitations

• Scores are computed on a subset of the official validation split, not on the test-dev evaluation server, and are not comparable to published test-dev numbers.
• The base checkpoint scores well above published GIT-base test-dev results on these validation questions, which indicates that VQAv2 validation data was part of its original fine-tuning. Absolute accuracies on validation are therefore optimistic for both models.
• Answers are lower-case (uncased tokenizer). WordPiece splits punctuation, so decoded answers such as 11 : 10 need detokenization (11:10).
• English only; COCO-domain images.