prodigalbeef/qwen3vl-chartqa-qlora

Qwen3-VL-4B · QLoRA on ChartQA

Fine-tune a 4B vision-language model on a single free-tier T4 GPU to answer questions about charts.

The base model is loaded in 4-bit NF4. LoRA adapters are trained on the attention layers of both the language model and the vision encoder. The run is evaluated against the untouched base model on the same 500 held-out questions.

Built by Yashveer Singh Bhayan (CB.EN.U4ARE23057) for 25ARE471 Introduction to Generative AI, Amrita Vishwa Vidyapeetham.

Base model	Qwen/Qwen3-VL-4B-Instruct, 4-bit NF4 + double quant (bitsandbytes)
Adapter	LoRA r=16, α=16, 192 modules (144 LLM + 48 ViT), 14.2M trainable params (0.32%)
Data	ChartQA train (28,299 QA pairs), 1,900 steps × batch 8 ≈ 0.54 epoch
Hardware	1× Tesla T4 16 GB (Kaggle), fp16 · 2.67 h training, 2.96 h end-to-end
Result	83.8% → 85.6% relaxed accuracy (+1.8 pts) on 500 ChartQA test questions

---

Results

	Relaxed accuracy
Base (4-bit, zero-shot)	0.838 (419/500)
QLoRA fine-tuned	0.856 (428/500)
Delta	+1.8 pts

Answer type	n	Base	Fine-tuned	Δ
Numeric	382	0.859	0.872	+1.3
Text	90	0.789	0.833	+4.4
Yes/No	28	0.714	0.714	0.0
Human-written questions	245	0.743	0.759	+1.6
Machine-generated questions	255	0.929	0.949	+2.0

Error patterns. The errors that shrank most are format mismatches, which fell from 6 to 3. One example is answering "2011" when the gold answer is "March 2011". Large numeric misses also fell, from 30 to 26. Reasoning-heavy questions remain the main failure mode on human-written charts, such as counting bars above a threshold or averaging series. Full tables, including the error taxonomy and sample failures, are in results/report.md.

How solid is the +1.8? At the level of individual questions, fine-tuning fixed 24 answers and broke 15, for a net gain of 9. An exact McNemar test on those 39 flips gives p ≈ 0.20. The gain is consistent in direction across every category except yes/no, but on 500 questions it is not statistically significant. A full-test-set evaluation (2,500 questions) would settle it.

Training loss

The loss is computed on answer tokens only. Its mean fell from 0.69 over the first 20 steps to 0.35 over the last 20. The per-step loss is noisy because answers are only 1–3 tokens long. The plot shows 50-step means, also available in results/loss_binned.csv.

---

Method

ChartQA image + question
   │  resize longest side ≤ 768 px · prompt: "Answer … with a single word or number."
   ▼
Qwen3-VL-4B-Instruct  (frozen, 4-bit NF4)
   ├─ Vision encoder  ── LoRA on attn.qkv, attn.proj        (48 modules)
   └─ Language model  ── LoRA on q_proj, k_proj, v_proj, o_proj (144 modules)
   ▼
SFT (TRL + Unsloth) · loss masked to the assistant answer only
   ▼
Greedy decode (≤24 tokens) → first line → ChartQA relaxed accuracy

• Why both towers. ChartQA errors are often perception errors, such as misreading a bar height or an axis label, not just reasoning errors. Adapting only the LLM would leave the vision encoder's features fixed. The code asserts that LoRA is attached to both towers.
• Memory on a T4. The setup uses 4-bit base weights, Unsloth gradient checkpointing with gradient offload, 8-bit AdamW and a 768 px image cap. Images are decoded lazily per batch, so the 28k-image train set never sits in RAM.
• Metric. This is ChartQA's standard relaxed accuracy: numeric answers count as correct within 5% relative error, and everything else needs a case-insensitive exact match. It is implemented in chartqa_qlora/metrics.py and unit-tested.
• Fair baseline. The same quantized model, prompt, image size, decoding settings and seeded 500-question subset are used before and after training.

Hyperparameters

LoRA rank / α / dropout	16 / 16 / 0
Learning rate / schedule	2e-4, linear, 50 warmup steps
Batch	2 × 4 grad-accum = 8
Steps	1,900 (≈15.2k examples, 0.54 epoch)
Optimizer	AdamW 8-bit, weight decay 0.01
Precision	fp16 (T4 has no bf16)
Seed	3407

---

Quick start

Requirements: an NVIDIA GPU with at least 16 GB (a T4 works) and Python 3.11+.

git clone https://huggingface.co/prodigalbeef/qwen3vl-chartqa-qlora
cd qwen3vl-chartqa-qlora
pip install -r requirements.txt

Ask a question about a chart with the trained adapter (stored at the repo root):

python scripts/infer.py --image my_chart.png --question "What was the value in 2019?"

Or load it with PEFT directly:

from unsloth import FastVisionModel
model, tokenizer = FastVisionModel.from_pretrained("prodigalbeef/qwen3vl-chartqa-qlora", load_in_4bit=True)
FastVisionModel.for_inference(model)

Reproduce the experiment (baseline eval → train → fine-tuned eval → outputs/report.md):

python scripts/run_experiment.py --smoke     # ~15 min sanity check
python scripts/run_experiment.py             # full run, ~3 h on one T4
python scripts/run_experiment.py --max-side 640   # if you hit OOM

Re-score saved predictions on a CPU: python scripts/rescore.py baseline.jsonl finetuned.jsonl. Tests: pytest.

The original Kaggle notebook, with outputs stripped, is in notebooks/.

---

Project layout

chartqa_qlora/        core library
  config.py           hyperparameters, prompt, LoRA target regex
  data.py             ChartQA loading, image resize, chat formatting
  model.py            4-bit loading, LoRA attachment (asserts both towers)
  train.py            TRL SFT with answer-only loss + time-cap callback
  evaluate.py         greedy decoding + relaxed accuracy, writes predictions
  metrics.py          relaxed accuracy, answer types, error taxonomy
  report.py           baseline vs fine-tuned Markdown report
scripts/              run_experiment.py · infer.py · rescore.py
notebooks/            original Kaggle notebook
results/              report.md (reported run) · loss_binned.csv
assets/               figures
adapter_config.json   trained LoRA adapter (repo root, so PEFT/Unsloth load the repo id directly)
adapter_model.safetensors
tests/                metric unit tests

Limitations

• Small eval. The evaluation uses 500 of 2,500 test questions. The overall gain is real in direction but not statistically significant (see above). Yes/no has only 28 questions.
• Partial epoch. Training covered about 0.54 epoch because of the free-tier time budget. Longer training, or a higher rank, may help more.
• Single seed. There is no variance estimate across seeds.
• Strong base model. Qwen3-VL-4B already scores 83.8% zero-shot, so there is limited headroom. Most of the remaining errors are multi-step reasoning, which LoRA SFT on short answers does not target.
• Short-answer style. The model is tuned to reply with a single word or number. Expect terse outputs and no explanations.

Acknowledgements

I am deeply grateful to my faculty mentor, Dr Rakesh Kumar Siddhartan, Amrita Vishwa Vidyapeetham, for his guidance and support throughout this project.

This project also builds on Qwen3-VL by the Qwen team, the ChartQA dataset (Masry et al., 2022), and the Unsloth, TRL, PEFT and bitsandbytes libraries. Compute was provided by Kaggle's free GPU tier.

License

The code is released under Apache-2.0, matching the base model's license. ChartQA is subject to its own terms.