--- language: - vi library_name: transformers pipeline_tag: visual-question-answering base_model: - VietAI/vit5-base - openai/clip-vit-base-patch16 tags: - visual-question-answering - scene-text-vqa - vietnamese - ocr - t5 --- # ViSceT5-ViTextVQA ViSceT5 is a Vietnamese scene-text visual question answering model: a ViT5 encoder-decoder that reads the image, the question and the OCR tokens of the image, and generates the answer. This checkpoint is the full model fine-tuned on **ViTextVQA**. ## Model overview - **Backbone:** [VietAI/vit5-base](https://huggingface.co/VietAI/vit5-base) (encoder-decoder), about 346M parameters in total (fp32, 1.38 GB). - **QA-ViT:** question-aware CLIP ViT-B/16 vision features ([openai/clip-vit-base-patch16](https://huggingface.co/openai/clip-vit-base-patch16)) with a gated adapter and a grounding loss. - **OCR input:** OCR text with box, detection, recognition and character features, fed as a block of the encoder input. - **Consformer:** self-attention with a constituent prior over OCR words, and a span loss. - **DOVE:** training on two views of the OCR block, the original and a "related" copy in which a few OCR words are edited. - **OACT (OCR Adversarial Contrastive Training):** a training strategy, not an inference module. Each OCR view gets a noise step along the sign of its answer-loss gradient (epsilon 0.05 of the token RMS), and a joint supervised-contrastive loss ties the four views (two clean, two perturbed; separate projection heads for clean and perturbed). It is switched on in this checkpoint (`dove_adv=ocr_both`) and unused at inference. ## Results (ViTextVQA) Test: 10,028 questions, the full public test split. Main metric: EM / F1. Scored with `training/benchmark_eval.py` of the code repository (the benchmark protocol: EM, F1, BLEU, CIDEr, BERTScore with mBERT). Dev is the validation split at the selected checkpoint. | Metric | Test | Dev | |---|---|---| | Exact Match (EM) | 36.01 | 34.51 | | F1 (token) | 58.93 | 59.46 | | F1 (char) | 72.27 | - | | BLEU-1 | 58.88 | 58.28 | | BLEU-4 | 45.12 | 44.70 | | BLEU-avg | 51.65 | - | | CIDEr | 3.3181 | 3.3852 | | BERTScore F1 | 87.25 | - | ### Ablation on ViTextVQA (test) Each row is a separate training run (single seed); row 4 uses batch size 32, the others 16. | Configuration | EM | F1 | |---|---|---| | Baseline (ViT5 + question) | 29.94 | 54.91 | | [+ Consformer](https://huggingface.co/Kus669/ViSceT5-ViTextVQA-Abl-Consformer) | 32.55 | 56.40 | | [+ QA-ViT](https://huggingface.co/Kus669/ViSceT5-ViTextVQA-Abl-QAViT) | 32.88 | 56.77 | | [+ DOVE (batch 32)](https://huggingface.co/Kus669/ViSceT5-ViTextVQA-Abl-DOVE) | 34.60 | 58.26 | | [+ QA-ViT + Consformer + DOVE](https://huggingface.co/Kus669/ViSceT5-ViTextVQA-Abl-QAViT-Consformer-DOVE) | 34.85 | 58.20 | | **+ OACT (this model)** | **36.01** | **58.93** | Differences below about 0.6 EM are within the run-to-run noise we observed, so read the gain of each module as indicative, not as a significance test. Single training run (one seed); the checkpoint was selected on the dev set, never on test. ## Usage The architecture is custom (`model_type: openvivqa`), so it does not load with `AutoModel`. Use the code at commit [`d350d49`](https://github.com/Kussssssss/ViSceT5/tree/d350d49), which is the code this checkpoint was trained with, and follow the notebook `notebooks/twc_colab.ipynb` (cell 6b, "predict from a Hugging Face repo"). In short: ```bash git clone https://github.com/Kussssssss/ViSceT5 && cd ViSceT5 && git checkout d350d49 bash setup.sh huggingface-cli download Kus669/ViSceT5-ViTextVQA --local-dir ckpt MOCK_TEST=false STAGE=predict PREDICT_SPLIT=both PREDICT_DATASET=ViTextVQA \ PREDICT_CKPT_DIR=ckpt HF_REPO= python3 run_pipeline.py ``` The script prepares the dataset and its OCR features, generates answers with beam search (4 beams, up to 56 new tokens) and writes a submission CSV. ## Limitations - Trained and evaluated on ViTextVQA only; no claim is made for other Vietnamese VQA data or other languages. - The answer depends on the OCR tokens: errors or missing text in the OCR output limit what the model can answer. - Results are from a single seed. ## Citation A paper describing ViSceT5 is in preparation; the citation will be added here.