About the Benchmark
PoVisLE is a monocultural vision-language evaluation benchmark centered on Polish cultural and linguistic competence. It is designed for grounded evaluation: answers should depend on the interaction between the image and the question, rather than on text-only associations or surface-level entity recognition.
Tasks
During annotation, each VQA pair is assigned one of three task types. These are multiple-choice, binary yes/no, and open-ended questions. All questions are designed to require image understanding, and the answer should not be obtainable from textual knowledge alone without reference to the image.
- Multiple-choice questions include answer options appended below the question with letter labels.
- Yes/no questions use the raw question as the prompt and are evaluated as binary answers.
- Open-ended questions also use the raw question as the prompt, without appended answer options.
Categories
The taxonomy is adapted from PLCC and refined for the VQA setting. Grammar and vocabulary are merged into a unified Language category, while the hierarchy supports fine-grained diagnostics across cultural, linguistic, and visual reasoning domains. The dataset is organized into the following main categories.
- Art and Entertainment covers Polish and Poland-related artistic, media, and cultural references, including architecture, film, literature, music, paintings, sculpture, sport, and media.
- Culture and Tradition covers shared customs, practices, symbols, cuisine, religion, traditions, pop culture, and regional or ethnic cultural variation.
- Geography and Nature covers Polish physical, natural, urban, infrastructural, and socio-political spaces, including landscapes, landmarks, regions, and administrative entities.
- History and Society covers historical and contemporary Polish social context, including the Middle Ages, World War II, post-war history, and current affairs.
- Language covers visually grounded Polish linguistic phenomena, such as colloquial speech, slang, dialects, regionalisms, grammar, orthography, phraseology, rhetorical figures, and semantics.
- Image Understanding focuses on direct recognition and interpretation of visual content in Polish or Poland-related contexts.
- Visual Reasoning focuses on relationships, context, and inferred information within an image.
Image Understanding and Visual Reasoning form a smaller complementary subset focused on general multimodal skills while remaining embedded in Polish visual and linguistic contexts.
Dataset Creation
PoVisLE was created through manual, template-free annotation. Annotators selected or reviewed images from Wikimedia Commons, other permissively available public sources, and personal collections contributed for research use. Each image was paired with one or more Polish VQA prompts and labeled with a task type, category, and subcategory.
The dataset construction also included a Wikimedia-based augmentation stage to increase visual diversity and reduce selection bias. Candidate images were reviewed by annotators, and visually similar replacements were used only when the original question remained answerable from the new image.
Quality assurance included cross-validation by a second annotator, metadata and license checks, regular team discussion, and supervision by an expert annotator. The annotation guidelines required questions to be visually grounded, unambiguous, linguistically natural, and suitable for deterministic evaluation.
The benchmark is divided into a held-out test split for final evaluation and a public validation split. The validation split is intended mainly to illustrate the range of question types, so it is not sampled from the same distribution as the test set. In particular, it contains fewer open-ended questions, which are largely retained in the held-out test split.
Evaluation Protocol
Models receive the image and a Polish prompt specifying the expected answer format, length, word order, and, where relevant, grammatical form. Macro accuracy is the main metric. Multiple-choice questions use circular evaluation: answer options are cyclically rotated, and a prediction is counted as correct only if the model selects the gold answer under every rotation. This reduces option-position bias while keeping evaluation efficient.
Yes/no and open-ended questions are evaluated in a single pass. Yes/no questions require a binary answer in Polish. Open-ended predictions are compared against the gold answers, with correct diacritics required in all cases and correct capitalization required where relevant. For selected questions, multiple answer variants are accepted through predefined inclusion patterns developed through iterative human validation.