| --- |
| language: |
| - it |
| library_name: transformers |
| pipeline_tag: feature-extraction |
| base_model: DT4H/CardioBERTa.it |
| tags: |
| - biomedical |
| - clinical |
| - cardiology |
| - entity-linking |
| - concept-normalization |
| - umls |
| - metric-learning |
| --- |
| |
| # DT4H_CardioBERTa_grandparents_it_enriched |
|
|
| `DT4H_CardioBERTa_grandparents_it_enriched` is a Italian biomedical terminology encoder for **clinical concept normalization and entity linking**. It is initialized from [`DT4H/CardioBERTa.it`] and specialized using CUI-supervised terminology pairs and metric learning. |
|
|
| ## Backbone |
|
|
| The backbone belongs to the **CardioBERTa** family from *CardioLM - a multilingual suite of small language models for the cardiology domain*. CardioBERTa comprises language-specific encoder models adapted to cardiology through continued pretraining on monolingual biomedical and cardiology-related corpora using Masked Language Modeling (MLM). The family covers Czech, Dutch, English, Italian, Romanian, Spanish and Swedish. |
|
|
| ## Training |
|
|
| | | | |
| |---|---| |
| | Language | Italian (`it`) | |
| | Triplet collection | `enriched` | |
| | Strategy | `grandparents` | |
| | Objective | Multi-Similarity Loss | |
| | Mining | All triplets, margin 0.2 | |
| | Pooling | CLS | |
| | Epochs | 1 | |
| | Batch size | 256 | |
| | Learning rate | 2e-5 | |
| | Max. length | 25 | |
|
|
| CUI-supervised terminology pairs enriched with grandparent-level ontology relations. |
|
|
| ### Terminology statistics |
|
|
| | Strategy | Triplets | CUIs | Unique terms | Unique positives | Terms/CUI | Δ terms | |
| |---|---:|---:|---:|---:|---:|---:| |
| | synonyms | 69,631 | 69,631 | 136,720 | 69,113 | 2.00 | 0 | |
| | parents | 1,597,673 | 476,349 | 529,199 | 414,377 | 3.93 | +392,479 | |
| | grandparents | 4,714,271 | 476,970 | 529,487 | 461,831 | 9.83 | +392,767 | |
|
|
| This model uses **4,714,271 triplets**, covering **476,970 CUIs** and **529,487 unique normalized terms**. |
|
|
| The training terminology is **not distributed** with this repository because it contains resources subject to UMLS licensing conditions. Only aggregate statistics are released. |
|
|
| ## Intended use |
|
|
| The model is intended for terminology embedding, biomedical candidate retrieval, concept normalization and entity linking, particularly in cardiology and clinical NLP pipelines. It is not intended for direct clinical decision-making. |
|
|
| ## Usage |
|
|
| ```python |
| import torch |
| import torch.nn.functional as F |
| from transformers import AutoModel, AutoTokenizer |
| |
| model_id = "DT4H/DT4H_CardioBERTa_grandparents_it_enriched" |
| |
| tokenizer = AutoTokenizer.from_pretrained(model_id) |
| model = AutoModel.from_pretrained(model_id) |
| |
| inputs = tokenizer( |
| "clinical concept", |
| return_tensors="pt", |
| truncation=True, |
| max_length=25, |
| ) |
| |
| with torch.no_grad(): |
| output = model(**inputs) |
| |
| embedding = F.normalize( |
| output.last_hidden_state[:, 0, :], |
| p=2, |
| dim=1, |
| ) |
| ``` |
|
|
| ## Reference |
|
|
| **Danu et al.** *CardioLM - a multilingual suite of small language models for the cardiology domain*. |
|
|
| Developed within the **DataTools4Heart (DT4H)** project, Grant Agreement 101057849. |
|
|
