Visual Question Answering
Transformers
Safetensors
Vietnamese
openvivqa
scene-text-vqa
vietnamese
ocr
t5
Instructions to use Kus669/ViSceT5-ViTextVQA with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use Kus669/ViSceT5-ViTextVQA with Transformers:
# Use a pipeline as a high-level helper # Warning: Pipeline type "visual-question-answering" is no longer supported in transformers v5. # You must load the model directly (see below) or downgrade to v4.x with: # pip install "transformers<5.0.0" from transformers import pipeline pipe = pipeline("visual-question-answering", model="Kus669/ViSceT5-ViTextVQA")# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("Kus669/ViSceT5-ViTextVQA", device_map="auto") - Notebooks
- Google Colab
- Kaggle
|
Download README.md from Kus669/ViSceT5-ViTextVQA: direct link, hf CLI and curl.
- Browser
- Download file 4.28 kB
-
https://huggingface.co/Kus669/ViSceT5-ViTextVQA/resolve/main/README.md
- Command line
-
hf download hf://Kus669/ViSceT5-ViTextVQA/README.md
-
curl -L -o README.md https://huggingface.co/Kus669/ViSceT5-ViTextVQA/resolve/main/README.md
4.28 kB
| language: | |
| - vi | |
| library_name: transformers | |
| pipeline_tag: visual-question-answering | |
| base_model: | |
| - VietAI/vit5-base | |
| - openai/clip-vit-base-patch16 | |
| tags: | |
| - visual-question-answering | |
| - scene-text-vqa | |
| - vietnamese | |
| - ocr | |
| - t5 | |
| # ViSceT5-ViTextVQA | |
| ViSceT5 is a Vietnamese scene-text visual question answering model: a ViT5 encoder-decoder that reads the | |
| image, the question and the OCR tokens of the image, and generates the answer. This checkpoint is the full | |
| model fine-tuned on **ViTextVQA**. | |
| ## Model overview | |
| - **Backbone:** [VietAI/vit5-base](https://huggingface.co/VietAI/vit5-base) (encoder-decoder), about 346M parameters in total (fp32, 1.38 GB). | |
| - **QA-ViT:** question-aware CLIP ViT-B/16 vision features ([openai/clip-vit-base-patch16](https://huggingface.co/openai/clip-vit-base-patch16)) with a gated adapter and a grounding loss. | |
| - **OCR input:** OCR text with box, detection, recognition and character features, fed as a block of the encoder input. | |
| - **Consformer:** self-attention with a constituent prior over OCR words, and a span loss. | |
| - **DOVE:** training on two views of the OCR block, the original and a "related" copy in which a few OCR words are edited. | |
| - **OACT (OCR Adversarial Contrastive Training):** a training strategy, not an inference module. Each OCR view gets a noise step | |
| along the sign of its answer-loss gradient (epsilon 0.05 of the token RMS), and a joint supervised-contrastive loss ties the four views | |
| (two clean, two perturbed; separate projection heads for clean and perturbed). It is switched on in this checkpoint | |
| (`dove_adv=ocr_both`) and unused at inference. | |
| ## Results (ViTextVQA) | |
| Test: 10,028 questions, the full public test split. Main metric: EM / F1. Scored with `training/benchmark_eval.py` of the code repository | |
| (the benchmark protocol: EM, F1, BLEU, CIDEr, BERTScore with mBERT). Dev is the validation split at the selected checkpoint. | |
| | Metric | Test | Dev | | |
| |---|---|---| | |
| | Exact Match (EM) | 36.01 | 34.51 | | |
| | F1 (token) | 58.93 | 59.46 | | |
| | F1 (char) | 72.27 | - | | |
| | BLEU-1 | 58.88 | 58.28 | | |
| | BLEU-4 | 45.12 | 44.70 | | |
| | BLEU-avg | 51.65 | - | | |
| | CIDEr | 3.3181 | 3.3852 | | |
| | BERTScore F1 | 87.25 | - | | |
| ### Ablation on ViTextVQA (test) | |
| Each row is a separate training run (single seed); row 4 uses batch size 32, the others 16. | |
| | Configuration | EM | F1 | | |
| |---|---|---| | |
| | Baseline (ViT5 + question) | 29.94 | 54.91 | | |
| | [+ Consformer](https://huggingface.co/Kus669/ViSceT5-ViTextVQA-Abl-Consformer) | 32.55 | 56.40 | | |
| | [+ QA-ViT](https://huggingface.co/Kus669/ViSceT5-ViTextVQA-Abl-QAViT) | 32.88 | 56.77 | | |
| | [+ DOVE (batch 32)](https://huggingface.co/Kus669/ViSceT5-ViTextVQA-Abl-DOVE) | 34.60 | 58.26 | | |
| | [+ QA-ViT + Consformer + DOVE](https://huggingface.co/Kus669/ViSceT5-ViTextVQA-Abl-QAViT-Consformer-DOVE) | 34.85 | 58.20 | | |
| | **+ OACT (this model)** | **36.01** | **58.93** | | |
| Differences below about 0.6 EM are within the run-to-run noise we observed, so read the gain of each | |
| module as indicative, not as a significance test. | |
| Single training run (one seed); the checkpoint was selected on the dev set, never on test. | |
| ## Usage | |
| The architecture is custom (`model_type: openvivqa`), so it does not load with `AutoModel`. Use the code at commit | |
| [`d350d49`](https://github.com/Kussssssss/ViSceT5/tree/d350d49), which is the code this checkpoint was trained with, and follow the notebook | |
| `notebooks/twc_colab.ipynb` (cell 6b, "predict from a Hugging Face repo"). In short: | |
| ```bash | |
| git clone https://github.com/Kussssssss/ViSceT5 && cd ViSceT5 && git checkout d350d49 | |
| bash setup.sh | |
| huggingface-cli download Kus669/ViSceT5-ViTextVQA --local-dir ckpt | |
| MOCK_TEST=false STAGE=predict PREDICT_SPLIT=both PREDICT_DATASET=ViTextVQA \ | |
| PREDICT_CKPT_DIR=ckpt HF_REPO= python3 run_pipeline.py | |
| ``` | |
| The script prepares the dataset and its OCR features, generates answers with beam search (4 beams, up to 56 new tokens) | |
| and writes a submission CSV. | |
| ## Limitations | |
| - Trained and evaluated on ViTextVQA only; no claim is made for other Vietnamese VQA data or other languages. | |
| - The answer depends on the OCR tokens: errors or missing text in the OCR output limit what the model can answer. | |
| - Results are from a single seed. | |
| ## Citation | |
| A paper describing ViSceT5 is in preparation; the citation will be added here. | |