Upload fine-tuned GIT-base VQAv2 (git-vqav2-medium)

README.md

@@ -0,0 +1,110 @@
+---
+license: mit
+base_model: microsoft/git-base-vqav2
+datasets:
+- HuggingFaceM4/VQAv2
+language:
+- en
+pipeline_tag: visual-question-answering
+library_name: transformers
+tags:
+- git
+- vqa
+- visual-question-answering
+- image-text-to-text
+---
+
+# git-vqav2-medium
+
+`microsoft/git-base-vqav2` fine-tuned on VQAv2 (50,000 training questions).
+
+## Model
+
+| | |
+|---|---|
+| Architecture | `GitForCausalLM` (GIT-base) |
+| Vision encoder | CLIP ViT-B/16, 12 layers, 480×480 input, 901 visual tokens |
+| Text decoder | 6-layer causal Transformer, hidden 768, 12 heads, max 1024 positions |
+| Tokenizer | BERT WordPiece (uncased), vocab 30,522 |
+| Parameters | 177,159,738 (trainable during fine-tuning: 177,159,738, strategy `full`) |
+| Weights | fp32, safetensors |
+
+## Input / output format
+
+- Image: `CLIPImageProcessor` — resize shortest edge to 480 (bicubic), center crop 480×480, rescale 1/255, normalize with CLIP mean/std.
+- Text prompt: `[CLS] question` (no `[SEP]` between question and answer).
+- Output: the answer tokens generated after the prompt, terminated by `[SEP]` (eos id 102).
+- Training sequence: `[CLS] question answer [SEP]`; loss only on `answer [SEP]` (all other positions `-100`).
+
+## Training data
+
+| | |
+|---|---|
+| Dataset | `HuggingFaceM4/VQAv2`, parquet conversion `refs/convert/parquet` @ `ac4a1a047466a7019b5357ce94b6b5794fc9f867` |
+| Split | `train` (COCO train2014 images) |
+| Shards / questions | [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19] / 50,000 |
+| Rows skipped (missing question/answer) | 0 |
+| Target answer | majority vote over the 10 human answers (lower-cased, whitespace/edge punctuation stripped; ties → `multiple_choice_answer`) |
+| Model selection | 1,000 questions from `validation` shards [2, 7], best `vqa_accuracy` |
+
+## Training procedure
+
+| Hyperparameter | Value |
+|---|---|
+| Epochs | 2 |
+| Optimizer | AdamW (β=(0.9, 0.999), ε=1e-8), no decay on biases/LayerNorm |
+| Learning rate | 2e-06, linear decay, warmup 156 steps |
+| Batch size | 8 × 4 accumulation = 32 |
+| Weight decay | 0.01 |
+| Gradient clipping | 1.0 |
+| Precision | bf16 autocast, fp32 master weights |
+| Vision attention | sdpa |
+| Optimizer steps | 3,126 |
+| Best checkpoint | step 500 (vqa_accuracy = 84.39) |
+| Seed | 42 |
+| Hardware | NVIDIA GeForce RTX 5060 Ti |
+| Software | torch 2.11.0+cu128, transformers 5.17.0, datasets 4.3.0 |
+
+## Evaluation
+
+VQAv2 consensus accuracy (official `vqaEval.py` normalization; per question `mean over 10 leave-one-out subsets of min(1, matches/3)`) and exact match against the majority answer (not an official metric). Deterministic decoding: max_new_tokens=10, num_beams=1, do_sample=False.
+
+Evaluation set: 500 questions sampled with a fixed seed from `validation` shard(s) [0] (subset fingerprint `1daf0de4dae278b4`), disjoint from the model-selection questions. Both models were evaluated on exactly the same questions.
+
+| Metric | microsoft/git-base-vqav2 | This model | Δ |
+|---|---|---|---|
+| VQA accuracy | 83.32 | 82.54 | -0.78 |
+| Exact match | 73.40 | 72.00 | -1.40 |
+| VQA accuracy (number) | 79.47 | 77.07 | -2.40 |
+| VQA accuracy (other) | 75.74 | 74.84 | -0.90 |
+| VQA accuracy (yes/no) | 95.14 | 95.19 | +0.05 |
+
+## Usage
+
+```python
+import torch
+from PIL import Image
+from transformers import AutoProcessor, AutoModelForCausalLM
+
+repo = "thealper2/git-vqav2-medium"
+processor = AutoProcessor.from_pretrained(repo)
+model = AutoModelForCausalLM.from_pretrained(repo).eval()
+
+image = Image.open("example.jpg").convert("RGB")
+question = "what color is the car?"
+pixel_values = processor(images=image, return_tensors="pt").pixel_values
+input_ids = [processor.tokenizer.cls_token_id] + processor.tokenizer(question, add_special_tokens=False).input_ids
+input_ids = torch.tensor([input_ids])
+
+out = model.generate(pixel_values=pixel_values, input_ids=input_ids, max_new_tokens=10)
+print(processor.batch_decode(out[:, input_ids.shape[1]:], skip_special_tokens=True)[0])
+```
+
+Batched generation: GIT does not derive text position ids from the attention mask, so padded prompts change the output. Batch only prompts of equal token length.
+
+## Limitations
+
+- Scores are computed on a subset of the official validation split, not on the test-dev evaluation server, and are not comparable to published test-dev numbers.
+- The base checkpoint scores well above published GIT-base test-dev results on these validation questions, which indicates that VQAv2 validation data was part of its original fine-tuning. Absolute accuracies on validation are therefore optimistic for both models.
+- Answers are lower-case (uncased tokenizer). WordPiece splits punctuation, so decoded answers such as `11 : 10` need detokenization (`11:10`).
+- English only; COCO-domain images.

config.json

@@ -0,0 +1,63 @@
+{
+  "architectures": [
+    "GitForCausalLM"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "bos_token_id": 101,
+  "classifier_dropout": null,
+  "dtype": "float32",
+  "eos_token_id": 102,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "layer_norm_eps": 1e-12,
+  "max_position_embeddings": 1024,
+  "model_type": "git",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 6,
+  "num_image_with_embedding": null,
+  "pad_token_id": 0,
+  "position_embedding_type": "absolute",
+  "tie_word_embeddings": false,
+  "transformers_version": "5.17.0",
+  "use_cache": true,
+  "vision_config": {
+    "add_cross_attention": false,
+    "attention_dropout": 0.0,
+    "bos_token_id": null,
+    "cross_attention_hidden_size": null,
+    "decoder_start_token_id": null,
+    "dropout": 0.0,
+    "dtype": "float32",
+    "eos_token_id": null,
+    "finetuning_task": null,
+    "hidden_act": "quick_gelu",
+    "hidden_size": 768,
+    "image_size": 480,
+    "initializer_factor": 1.0,
+    "initializer_range": 0.02,
+    "intermediate_size": 3072,
+    "is_decoder": false,
+    "layer_norm_eps": 1e-05,
+    "model_type": "git_vision_model",
+    "num_attention_heads": 12,
+    "num_channels": 3,
+    "num_hidden_layers": 12,
+    "pad_token_id": null,
+    "patch_size": 16,
+    "prefix": null,
+    "projection_dim": 512,
+    "pruned_heads": {},
+    "sep_token_id": null,
+    "task_specific_params": null,
+    "tf_legacy_loss": false,
+    "tie_encoder_decoder": false,
+    "tie_word_embeddings": true,
+    "tokenizer_class": null,
+    "torchscript": false,
+    "use_bfloat16": false
+  },
+  "vocab_size": 30522
+}

eval_results.json

@@ -0,0 +1,191 @@
+{
+  "finetuned": {
+    "vqa_accuracy": 82.54,
+    "exact_match": 72.0,
+    "vqa_accuracy_by_answer_type": {
+      "number": {
+        "n": 75,
+        "vqa_accuracy": 77.07
+      },
+      "other": {
+        "n": 244,
+        "vqa_accuracy": 74.84
+      },
+      "yes/no": {
+        "n": 181,
+        "vqa_accuracy": 95.19
+      }
+    },
+    "num_examples": 500,
+    "subset_fingerprint": "1daf0de4dae278b4",
+    "generation": {
+      "max_new_tokens": 10,
+      "num_beams": 1,
+      "do_sample": false,
+      "eos_token_id": 102,
+      "pad_token_id": 0
+    }
+  },
+  "comparison": {
+    "subset_fingerprint": "1daf0de4dae278b4",
+    "num_examples": 500,
+    "base_model": "microsoft/git-base-vqav2",
+    "finetuned_model": "outputs/git-vqav2-medium/best",
+    "metrics": {
+      "vqa_accuracy": {
+        "base": 83.32,
+        "finetuned": 82.54,
+        "absolute_improvement": -0.78
+      },
+      "exact_match": {
+        "base": 73.4,
+        "finetuned": 72.0,
+        "absolute_improvement": -1.4
+      },
+      "vqa_accuracy_number": {
+        "base": 79.47,
+        "finetuned": 77.07,
+        "absolute_improvement": -2.4
+      },
+      "vqa_accuracy_other": {
+        "base": 75.74,
+        "finetuned": 74.84,
+        "absolute_improvement": -0.9
+      },
+      "vqa_accuracy_yes/no": {
+        "base": 95.14,
+        "finetuned": 95.19,
+        "absolute_improvement": 0.05
+      }
+    }
+  },
+  "training_summary": {
+    "global_step": 3126,
+    "epochs": 2,
+    "runtime_sec_this_session": 5464.2,
+    "best_metric_name": "vqa_accuracy",
+    "best_metric": 84.39,
+    "best_step": 500,
+    "skipped_images": 0,
+    "skipped_question_ids": [],
+    "truncated_sequences": 2,
+    "total_steps": 3126,
+    "warmup_steps": 156,
+    "peak_gpu_memory": {
+      "gpu": "NVIDIA GeForce RTX 5060 Ti",
+      "vram_total_gb": 15.9,
+      "vram_free_gb": 12.5,
+      "vram_allocated_gb": 2.0,
+      "vram_reserved_gb": 2.26,
+      "vram_max_allocated_gb": 11.23,
+      "vram_max_reserved_gb": 11.77
+    }
+  },
+  "eval_history": [
+    {
+      "type": "eval",
+      "step": 500,
+      "epoch": 0.32,
+      "val_loss": 0.51991,
+      "vqa_accuracy": 84.39,
+      "exact_match": 76.0,
+      "val_skipped_images": 0,
+      "vram_total_gb": 15.9,
+      "vram_free_gb": 2.64,
+      "vram_allocated_gb": 2.0,
+      "vram_reserved_gb": 11.71,
+      "vram_max_allocated_gb": 11.23,
+      "vram_max_reserved_gb": 11.71
+    },
+    {
+      "type": "eval",
+      "step": 1000,
+      "epoch": 0.64,
+      "val_loss": 0.52613,
+      "vqa_accuracy": 83.62,
+      "exact_match": 75.5,
+      "val_skipped_images": 0,
+      "vram_total_gb": 15.9,
+      "vram_free_gb": 2.58,
+      "vram_allocated_gb": 2.0,
+      "vram_reserved_gb": 11.71,
+      "vram_max_allocated_gb": 11.23,
+      "vram_max_reserved_gb": 11.71
+    },
+    {
+      "type": "eval",
+      "step": 1500,
+      "epoch": 0.96,
+      "val_loss": 0.51468,
+      "vqa_accuracy": 83.82,
+      "exact_match": 75.4,
+      "val_skipped_images": 0,
+      "vram_total_gb": 15.9,
+      "vram_free_gb": 2.68,
+      "vram_allocated_gb": 2.0,
+      "vram_reserved_gb": 11.71,
+      "vram_max_allocated_gb": 11.23,
+      "vram_max_reserved_gb": 11.71
+    },
+    {
+      "type": "eval",
+      "step": 2000,
+      "epoch": 1.28,
+      "val_loss": 0.51831,
+      "vqa_accuracy": 84.06,
+      "exact_match": 75.9,
+      "val_skipped_images": 0,
+      "vram_total_gb": 15.9,
+      "vram_free_gb": 2.32,
+      "vram_allocated_gb": 2.0,
+      "vram_reserved_gb": 11.77,
+      "vram_max_allocated_gb": 11.23,
+      "vram_max_reserved_gb": 11.77
+    },
+    {
+      "type": "eval",
+      "step": 2500,
+      "epoch": 1.6,
+      "val_loss": 0.51739,
+      "vqa_accuracy": 83.76,
+      "exact_match": 75.1,
+      "val_skipped_images": 0,
+      "vram_total_gb": 15.9,
+      "vram_free_gb": 1.79,
+      "vram_allocated_gb": 2.0,
+      "vram_reserved_gb": 11.71,
+      "vram_max_allocated_gb": 11.23,
+      "vram_max_reserved_gb": 11.77
+    },
+    {
+      "type": "eval",
+      "step": 3000,
+      "epoch": 1.92,
+      "val_loss": 0.51595,
+      "vqa_accuracy": 83.62,
+      "exact_match": 74.7,
+      "val_skipped_images": 0,
+      "vram_total_gb": 15.9,
+      "vram_free_gb": 2.42,
+      "vram_allocated_gb": 2.0,
+      "vram_reserved_gb": 11.77,
+      "vram_max_allocated_gb": 11.23,
+      "vram_max_reserved_gb": 11.77
+    },
+    {
+      "type": "eval",
+      "step": 3126,
+      "epoch": 2.0,
+      "val_loss": 0.51609,
+      "vqa_accuracy": 83.69,
+      "exact_match": 74.7,
+      "val_skipped_images": 0,
+      "vram_total_gb": 15.9,
+      "vram_free_gb": 2.52,
+      "vram_allocated_gb": 2.0,
+      "vram_reserved_gb": 11.71,
+      "vram_max_allocated_gb": 11.23,
+      "vram_max_reserved_gb": 11.77
+    }
+  ]
+}

experiment_config.yaml

@@ -0,0 +1,87 @@
+experiment_name: medium
+seed: 42
+deterministic: false
+precision: auto
+model:
+  name: microsoft/git-base-vqav2
+  finetune_strategy: full
+  gradient_checkpointing: false
+  vision_attn_implementation: sdpa
+  image_processor_backend: null
+dataset:
+  name: HuggingFaceM4/VQAv2
+  revision: ac4a1a047466a7019b5357ce94b6b5794fc9f867
+  subset_seed: 42
+  max_question_tokens: 32
+  max_answer_tokens: 16
+  train:
+    split: train
+    shards:
+    - 0
+    - 1
+    - 2
+    - 3
+    - 4
+    - 5
+    - 6
+    - 7
+    - 8
+    - 9
+    - 10
+    - 11
+    - 12
+    - 13
+    - 14
+    - 15
+    - 16
+    - 17
+    - 18
+    - 19
+    samples: 50000
+  val:
+    split: validation
+    shards:
+    - 2
+    - 7
+    samples: 1000
+  test:
+    split: validation
+    shards:
+    - 0
+    samples: 500
+training:
+  epochs: 2
+  learning_rate: 2.0e-06
+  batch_size: 8
+  gradient_accumulation_steps: 4
+  weight_decay: 0.01
+  warmup_ratio: 0.05
+  lr_scheduler: linear
+  max_grad_norm: 1.0
+  num_workers: 6
+  logging_steps: 10
+  save_steps: 500
+  save_total_limit: 3
+  memory_test: true
+  gpu_memory_reserve_gb: 0.5
+  tensorboard: true
+evaluation:
+  batch_size: 32
+  eval_steps: 500
+  metric_for_best: vqa_accuracy
+  generate_during_training: true
+  output_dir: outputs/eval
+generation:
+  max_new_tokens: 10
+  num_beams: 1
+  do_sample: false
+  temperature: 1.0
+  top_p: 1.0
+overfit:
+  samples: 20
+  batch_size: 4
+  steps: 100
+  learning_rate: 0.0001
+  output_dir: outputs/overfit
+output:
+  directory: outputs/git-vqav2-medium

generation_config.json

@@ -0,0 +1,7 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 101,
+  "eos_token_id": 102,
+  "pad_token_id": 0,
+  "transformers_version": "5.17.0"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4d332dae2575605d7f63aa4c60a4cb275edd3ca211a3ac07caeb9286fe5031ba
+size 708678728

processor_config.json

@@ -0,0 +1,30 @@
+{
+  "image_processor": {
+    "crop_size": {
+      "height": 480,
+      "width": 480
+    },
+    "do_center_crop": true,
+    "do_convert_rgb": true,
+    "do_normalize": true,
+    "do_rescale": true,
+    "do_resize": true,
+    "image_mean": [
+      0.48145466,
+      0.4578275,
+      0.40821073
+    ],
+    "image_processor_type": "CLIPImageProcessor",
+    "image_std": [
+      0.26862954,
+      0.26130258,
+      0.27577711
+    ],
+    "resample": 3,
+    "rescale_factor": 0.00392156862745098,
+    "size": {
+      "shortest_edge": 480
+    }
+  },
+  "processor_class": "GitProcessor"
+}

tokenizer_config.json

@@ -0,0 +1,20 @@
+{
+  "backend": "tokenizers",
+  "cls_token": "[CLS]",
+  "do_lower_case": true,
+  "is_local": false,
+  "local_files_only": false,
+  "mask_token": "[MASK]",
+  "model_input_names": [
+    "input_ids",
+    "attention_mask"
+  ],
+  "model_max_length": 512,
+  "pad_token": "[PAD]",
+  "processor_class": "GitProcessor",
+  "sep_token": "[SEP]",
+  "strip_accents": null,
+  "tokenize_chinese_chars": true,
+  "tokenizer_class": "BertTokenizer",
+  "unk_token": "[UNK]"
+}

training_info.json

@@ -0,0 +1,117 @@
+{
+  "best": {
+    "type": "eval",
+    "step": 500,
+    "epoch": 0.32,
+    "val_loss": 0.51991,
+    "vqa_accuracy": 84.39,
+    "exact_match": 76.0,
+    "val_skipped_images": 0,
+    "vram_total_gb": 15.9,
+    "vram_free_gb": 2.64,
+    "vram_allocated_gb": 2.0,
+    "vram_reserved_gb": 11.71,
+    "vram_max_allocated_gb": 11.23,
+    "vram_max_reserved_gb": 11.71
+  },
+  "model_name": "microsoft/git-base-vqav2",
+  "dataset_name": "HuggingFaceM4/VQAv2",
+  "dataset_revision": "ac4a1a047466a7019b5357ce94b6b5794fc9f867",
+  "train_data": {
+    "rows_total": 80000,
+    "rows_valid": 80000,
+    "rows_skipped": 0,
+    "skipped_by_reason": {
+      "missing_question": 0,
+      "missing_answers": 0
+    },
+    "skipped_example_question_ids": {},
+    "split": "train",
+    "shards": [
+      0,
+      1,
+      2,
+      3,
+      4,
+      5,
+      6,
+      7,
+      8,
+      9,
+      10,
+      11,
+      12,
+      13,
+      14,
+      15,
+      16,
+      17,
+      18,
+      19
+    ],
+    "requested_samples": 50000,
+    "selected_samples": 50000,
+    "subset_seed": 42,
+    "revision": "ac4a1a047466a7019b5357ce94b6b5794fc9f867"
+  },
+  "val_data": {
+    "rows_total": 8000,
+    "rows_valid": 8000,
+    "rows_skipped": 0,
+    "skipped_by_reason": {
+      "missing_question": 0,
+      "missing_answers": 0
+    },
+    "skipped_example_question_ids": {},
+    "split": "validation",
+    "shards": [
+      2,
+      7
+    ],
+    "requested_samples": 1000,
+    "selected_samples": 1000,
+    "subset_seed": 43,
+    "revision": "ac4a1a047466a7019b5357ce94b6b5794fc9f867"
+  },
+  "parameters": {
+    "total": 177159738,
+    "trainable": 177159738,
+    "frozen": 0,
+    "trainable_pct": 100.0
+  },
+  "precision": "bf16",
+  "seed": 42,
+  "environment": {
+    "python": "3.12.3",
+    "platform": "Linux-6.18.33.2-microsoft-standard-WSL2-x86_64-with-glibc2.39",
+    "torch": "2.11.0+cu128",
+    "torch_cuda": "12.8",
+    "cudnn": 91900,
+    "transformers": "5.17.0",
+    "datasets": "4.3.0",
+    "cuda_available": true,
+    "hf_home": "/mnt/d/huggingface",
+    "gpu": "NVIDIA GeForce RTX 5060 Ti",
+    "gpu_total_gb": 15.9,
+    "compute_capability": "12.0",
+    "bf16_supported": true
+  },
+  "hyperparameters": {
+    "epochs": 2,
+    "learning_rate": 2e-06,
+    "batch_size": 8,
+    "gradient_accumulation_steps": 4,
+    "weight_decay": 0.01,
+    "warmup_ratio": 0.05,
+    "lr_scheduler": "linear",
+    "max_grad_norm": 1.0,
+    "num_workers": 6,
+    "logging_steps": 10,
+    "save_steps": 500,
+    "save_total_limit": 3,
+    "memory_test": true,
+    "gpu_memory_reserve_gb": 0.5,
+    "tensorboard": true
+  },
+  "resumed_from": null
+}