| --- |
| language: |
| - en |
| - cs |
| license: cc-by-4.0 |
| metrics: |
| - bleurt |
| - bleu |
| - bertscore |
| --- |
| # AlignScoreCS |
|
|
| A MultiTask multilingual model is developed to **assess factual consistency in context-claim pairs** across various Natural Language Understanding (NLU) tasks, |
| including **Summarization**, **Question Answering (QA)**, **Semantic Textual Similarity (STS)**, **Paraphrase**, **Fact Verification (FV)**, and **Natural Language Inference (NLI)**. |
| AlignScoreCS is fine-tuned on a vast multi-task dataset consisting of 7 million documents, encompassing these NLU tasks in both **Czech** and **English** languages. |
| Its multilingual pre-training enables its potential utilization in **various other languages**. The architecture is capable of processing tasks using regression, |
| binary classification, or ternary classification, although for evaluation purposes, we recommend employing the AlignScore function. |
|
|
| This work is influenced by its English counterpart [AlignScore: Evaluating Factual Consistency with a Unified Alignment Function](https://arxiv.org/abs/2305.16739). |
| However, we employed homogeneous batches instead of heterogeneous ones during training and utilized three distinct architectures sharing a single encoder. |
| This setup allows for the independent use of each architecture with its classification head. |
|
|
|
|
| ## Evaluation |
| As in the paper AlignScore, we use their AlignScore function which chunk context into roughly 350 tokens and splits claim into sentences |
| each context chunk is evaluated against each claim sentence and aggregated one consistency score |
|
|
| AlignScoreCS model is built on three XLM-RoBERTa architectures sharing one encoder |
|
|
|
|
| MultiTask multilingual model for assessing facticity in various NLU tasks in Czech and English language. We followed the initial paper AlignScore https://arxiv.org/abs/2305.16739. |
| We trained a model using a shared architecture of checkpoint xlm-roberta-large [xlm-roberta](https://huggingface.co/FacebookAI/xlm-roberta-large) with three linear layers for regression, |
| binary classification and ternary classification. |
|
|
|
|
| # Usage |
| ```python |
| # Assuming you copied the attached Files_and_versions/AlignScore.py file for ease of use in transformers. |
| from AlignScoreCS import AlignScoreCS |
| alignScoreCS = AlignScoreCS.from_pretrained("krotima1/AlignScoreCS") |
| # put the model to cuda to accelerate |
| print(alignScoreCS.score(context="This is context", claim="This is claim")) |
| |
| ``` |
|
|
| # Results |
|
|
|
|
|
|
| # Training datasets |
| The following table shows datasets that has been utilized for training the model. We translated these english datasets to Czech using seamLessM4t. |
|
|
| | NLP Task | Dataset | Training Task | Context (n words) | Claim (n words) | Sample Count | |
| |-----------------------|-------------------|---------------|-------------------|-----------------|--------------| |
| | NLI | SNLI | 3-way | 10 | 13 | Cs: 500k | |
| | | | | | | En: 550k | |
| | | MultiNLI | 3-way | 16 | 20 | Cs: 393k | |
| | | | | | | En: 393k | |
| | | Adversarial NLI | 3-way | 48 | 54 | Cs: 163k | |
| | | | | | | En: 163k | |
| | | DocNLI | 2-way | 97 | 285 | Cs: 200k | |
| | | | | | | En: 942k | |
| | Fact Verification | NLI-style FEVER | 3-way | 48 | 50 | Cs: 208k | |
| | | | | | | En: 208k | |
| | | Vitamin C | 3-way | 23 | 25 | Cs: 371k | |
| | | | | | | En: 371k | |
| | Paraphrase | QQP | 2-way | 9 | 11 | Cs: 162k | |
| | | | | | | En: 364k | |
| | | PAWS | 2-way | - | 18 | Cs: - | |
| | | | | | | En: 707k | |
| | | PAWS labeled | 2-way | 18 | - | Cs: 49k | |
| | | | | | | En: - | |
| | | PAWS unlabeled | 2-way | 18 | - | Cs: 487k | |
| | | | | | | En: - | |
| | STS | SICK | reg | - | 10 | Cs: - | |
| | | | | | | En: 4k | |
| | | STS Benchmark | reg | - | 10 | Cs: - | |
| | | | | | | En: 6k | |
| | | Free-N1 | reg | 18 | - | Cs: 20k | |
| | | | | | | En: - | |
| | QA | SQuAD v2 | 2-way | 105 | 119 | Cs: 130k | |
| | | | | | | En: 130k | |
| | | RACE | 2-way | 266 | 273 | Cs: 200k | |
| | | | | | | En: 351k | |
| | Information Retrieval| MS MARCO | 2-way | 49 | 56 | Cs: 200k | |
| | | | | | | En: 5M | |
| | Summarization | WikiHow | 2-way | 434 | 508 | Cs: 157k | |
| | | | | | | En: 157k | |
| | | SumAug | 2-way | - | - | Cs: - | |
| | | | | | | En: - | |
