Knowledge-Guided VQA Models
This repository contains trained model weights for Knowledge-Guided Visual Question Answering on the OK-VQA dataset.
These models explore different strategies for injecting external LLM-derived knowledge (visual focus cues and semantic knowledge) into a frozen Vision Transformer (ViT) + Flan-T5 pipeline for open-ended VQA.
🏗️ Architecture
| Component | Model |
|---|---|
| Vision Encoder | google/vit-base-patch16-224 |
| Text Encoder | CLIP Text (openai/clip-vit-base-patch16) |
| Decoder (Answer Generator) | google/flan-t5-small |
| Dataset | OK-VQA (train + val split) |
📦 Available Models
1. best_baseline_model.pth (309 MB)
- Strategy: No external knowledge. Standard ViT → CLS projection → T5 decoder.
- Purpose: Serves as the baseline for comparison.
2. best_fusion_model.pth (313 MB)
- Strategy: Late Fusion. Concatenates ViT CLS token with CLIP-encoded guidance vector, then projects to T5.
- Knowledge Source: LLM-generated visual focus + semantic knowledge via CLIP text encoder.
3. best_attention_model.pth (325 MB)
- Strategy: Attention Modification. Injects per-layer guidance projections into the top 6 ViT self-attention layers.
- Knowledge Source: LLM-generated visual focus + semantic knowledge via CLIP text encoder.
🔧 Usage
import torch
from transformers import ViTModel, AutoTokenizer, AutoModelForSeq2SeqLM
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
# Load base models
t5_tokenizer = AutoTokenizer.from_pretrained("google/flan-t5-small")
t5 = AutoModelForSeq2SeqLM.from_pretrained("google/flan-t5-small").to(device)
vit = ViTModel.from_pretrained("google/vit-base-patch16-224").to(device)
# Define your model class (see notebook for full definitions)
# Then load weights:
model = YourModelClass().to(device)
model.load_state_dict(torch.load("best_baseline_model.pth", map_location=device))
📊 Training Details
- Optimizer: Adam (lr=3e-5)
- Early Stopping: Patience = 3-4 epochs
- Max Epochs: 15
- Batch Size: 8
- Evaluation Metrics: Exact Match, Edit Similarity, Token F1, BERTScore
📄 Citation
If you use these models in your research, please cite:
@misc{knowledge-guided-vqa-2026,
title={Knowledge-Guided Visual Question Answering with Adaptive Vision-Language Integration},
author={Aneerban},
year={2026},
publisher={Hugging Face},
url={https://huggingface.co/rishii100/knowledge-guided-vqa-models}
}
📜 License
MIT License