| --- |
| title: README |
| emoji: 🏃 |
| colorFrom: purple |
| colorTo: green |
| sdk: static |
| pinned: false |
| --- |
| |
| # UPC-HLE Cross-Encoder Models |
|
|
| Welcome to the UPC-HLE organization on HuggingFace! |
| This repository hosts the cross-encoder models developed for our submission to [SemEval-2025 Task 7: Multilingual and Cross-lingual Fact-Checked Claim Retrieval](https://disai.eu/semeval-2025/). Our approach combines dense Text Embedding Models (TEMs) with a neural Cross-Encoder re-ranking stage to retrieve fact-checked claims for social-media posts in a multilingual setting. The dataset is a subset of [MultiClaim Dataset](https://zenodo.org/records/7737983). |
|
|
| --- |
|
|
| ## 🔍 Organization Overview |
|
|
| The SemEval-2025 Task 7 challenge requires retrieving previously fact-checked claims relevant to user queries (social-media posts) across multiple languages. Our solution uses a two-step pipeline: |
|
|
| 1. **Initial Retrieval** |
| Encode the query (post) and all fact-check documents with a bi-encoder TEM to get dense embeddings. Retrieve the top 100 candidates by cosine similarity. |
|
|
| 2. **Cross-Encoder Re-Ranking** |
| Fine-tune a transformer-based Cross-Encoder (Jina Reranker v2) on triplets of (post, candidate, label) using hard-negative sampling. Re-score the 100 candidates to produce the final ranked list. |
|
|
| This yields strong improvements over lexical baselines and standalone embeddings, especially in cross-lingual retrieval. |
|
|
| --- |
|
|
| ## 📦 Models in This Organization |
|
|
| These are the best performing models in the final solution for the shared task. The bi-encoder retrieval is performed using [Snowflake/snowflake-arctic-embed-l-v2.0](https://huggingface.co/Snowflake/snowflake-arctic-embed-l-v2.0) |
| and, afterwards, the re-ranking is done using the models available in this repository, which use Jina V2 as base Cross Encoder. |
|
|
| **Languages Available** |
|
|
| - **English** (eng) |
| - **French** (fra) |
| - **German** (deu) |
| - **Portuguese** (por) |
| - **Thai** (tha) |
| - **Malay** (msa) |
| - **Arabic** (ara) |
| - **Polish** (pol) |
| - **Turkish** (tur) |
| - **Crosslingual** |
|
|
| Thanks to its XLM-RoBERTa backbone, the model can encode sentences in any of these languages into the same 768-dimensional vector space. You can therefore use the **same** model instance to compute embeddings and cosine-similarity scores for queries and fact-check candidates **across** all test‐set languages, without loading separate language-specific checkpoints. |
|
|
| ## 🚀 Usage |
|
|
| Install the 🤗 Sentence Transformers library and then load any model as follows: |
|
|
| ```python |
| from sentence_transformers import CrossEncoder |
| rerank_model = CrossEncoder(reranker_model_name, num_labels=1, max_length=1024, trust_remote_code=True) |
| ranked_vals = rerank_model.rank(str_candidate, ls_factchecks, show_progress_bar=False) |
| |
| ``` |
|
|
| --- |
|
|
| ## 📈 Performance |
|
|
| All scores are reported as success@10 (the fraction of queries where at least one correct fact-check appears in the top 10). |
|
|
|
|
| Model Monolingual Cross-lingual |
| mTEM-CE-Eng-SF 0.91 0.78 |
| mTEM-CE-E5 0.91 0.75 |
| mTEM-CE-Eng-E5 0.90 0.76 |
| mTEM-LF-CE-Eng-E5 0.89 0.75 |
| These results demonstrate the benefit of CE re-ranking over the baseline TEM alone |
|
|
| --- |
|
|
| ## 🛠 Training Details |
| Triplet Mining: |
| – Retrieve 100 candidates per post by cosine similarity. |
| – Label the gold fact-check as positive; sample up to 4 hard negatives where their TEM score ≥ 0.95 × positive score. |
|
|
| Hyperparameters (for mTEM-CE-Eng-SF): |
| – Batch size: 32 |
| – Epochs: 5 |
| – Learning rate: 2 × 10⁻⁵ |
| – Warmup steps: 10% of total |
| – Candidates for re-ranking: 100 |
|
|
| ## 📚 Citation |
| If you use these models or our results, please cite: |
|
|
| @inproceedings{semeval2025task7upchle, |
| title={UPC-HLE at SemEval-2025 Task 7: Multilingual Fact-Checked Claim Retrieval with Text Embedding Models and Cross-Encoder Re-Ranking}, |
| author={Alberto Becerra-Tome, Agustín Conesa}, |
| booktitle = {Proceedings of the 19th International Workshop on Semantic Evaluation}, |
| series = {SemEval 2025}, |
| year = {2025}, |
| address = {Vienna, Austria}, |
| month = {July}, |
| pages = {} |
| doi= {} |
| } |
| |
| ## 📄 License |
| All models and code are released under the MIT License. See LICENSE for details. |
|
|
| Happy fact-checking! |
