license: apache-2.0 language:
bn tags:
bangla
vqa
llm
multimodal
evaluation pipeline_tag: text-generation
π§π© Evan Bangla VQA
Bangla Visual Question Answering using LLMs (Qwen vs TituLLM)
π Overview
Evan Bangla VQA is a Bangla Visual Question Answering (VQA) system built using Large Language Models (LLMs). This project provides a complete, reproducible pipeline for answering Bangla visual questions and benchmarking two strong models:
- Qwen (Qwen3.5-9B)
- TituLLM (hishab/titulm-llama-3.2-3b-v2.0)
The repository includes:
β Bangla prompt engineering β optimized inference pipeline β evaluation framework β comparison analysis β Hugging Face demo integration
This work is designed for research reproducibility, benchmarking, and publication.
π Features
- Bangla-first VQA prompting
- Multi-model inference (Qwen + TituLLM)
- Lightweight and reproducible pipeline
- Hugging Face demo support
- Paper-ready evaluation workflow
- Semantic evaluation using BERT embeddings
π Models Used
πΉ Qwen (Primary Model)
- Model:
Qwen/Qwen3.5-9B - Strong semantic reasoning
- Lower hallucination rate
πΉ TituLLM (Baseline Model)
- Model:
hishab/titulm-llama-3.2-3b-v2.0 - Lightweight baseline
- Used for full evaluation comparison
βοΈ Installation
Clone the repository:
git clone https://huggingface.co/evanmahmud/evan-bangla-vqa
cd evan-bangla-vqa
Install dependencies:
pip install -r requirements.txt
π§ Usage
πΉ Inference Example
from inference import answer
print(answer("ΰ¦ΰ¦¬ΰ¦Ώΰ¦€ΰ§ ΰ¦ΰ§ দΰ§ΰ¦ΰ¦Ύ ΰ¦―ΰ¦Ύΰ¦ΰ§ΰ¦ΰ§?", "Qwen"))
print(answer("ΰ¦ΰ¦¬ΰ¦Ώΰ¦€ΰ§ ΰ¦ΰ§ দΰ§ΰ¦ΰ¦Ύ ΰ¦―ΰ¦Ύΰ¦ΰ§ΰ¦ΰ§?", "TituLLM"))
π Prompt Design
The system uses a Bangla-optimized prompt:
ΰ¦ͺΰ§ΰ¦°ΰ¦Άΰ§ΰ¦¨: {question}
ΰ¦Έΰ¦ΰ¦ΰ§ΰ¦·ΰ¦Ώΰ¦ͺΰ§ΰ¦€ ΰ¦ΰ¦€ΰ§ΰ¦€ΰ¦° দিনΰ₯€
ΰ¦ΰ¦€ΰ§ΰ¦€ΰ¦°:
This improves:
β response clarity β brevity β Bangla alignment
π Evaluation Metrics
The system is evaluated using:
- Relaxed Accuracy
- BLEU
- ROUGE-L
- BERTScore (Sentence-BERT cosine similarity)
- Hallucination Rate
Hallucination Definition
Hallucination is computed using relaxed semantic matching:
Predictions are considered correct if:
β substring overlap exists β token overlap exists
π Results (Example)
| Model | Relaxed Accuracy | BLEU | ROUGE-L | BERTScore |
|---|---|---|---|---|
| TituLLM | 0.296 | 0.0176 | 0.000 | 0.572 |
| Qwen | Higher | Higher | Higher | Higher |
π Qwen shows stronger semantic alignment and lower hallucination.
π§ͺ Demo (Hugging Face Space)
You can try the interactive demo:
π https://huggingface.co/spaces/YOURNAME/evan-bangla-vqa-demo
π Reproducibility
This repository supports:
β Kaggle GPU workflows β quantized inference β lightweight execution
π Citation
If you use this work:
@INPROCEEDINGS{11771448,
author={Hoque, Md Mahmudul and Azrof, H.M. and Karmaker, Shuvo and Rafi, Redwan Ahmed and Khan, Sakib and Jyoti, Cornea},
booktitle={2026 IEEE 4th International Conference on Artificial Intelligence, Blockchain, and Internet of Things (AIBThings)},
title={BanglaVQA-LLM: Enhancing Bangla Visual Question Answering with Instruction-Tuned Large Language Models},
year={2026},
volume={},
number={},
pages={1-5},
keywords={Modeling;Visualization;Printing;Question answering (information retrieval);Multilingual;Testing;Training;Large language models;Tuning;Indexes;Bangla visual question answering;large language models;retrieval-augmented prompting;CLIP;low-resource languages;hallucination},
doi={10.1109/AIBThings71029.2026.11771448}}
π€ Contributions
Contributions are welcome:
β improved Bangla datasets β stronger evaluation methods β optimized pipelines
β Support
If you find this helpful, please β the repo.