Hugging Face's logo

YLab-Open
/
MultiClinicalBERT

Fill-Mask
Safetensors
modernbert
clinical-nlp
multilingual
bert
Model card Files
xet
 Community
enqiliu commited on
Commit
ccc58c1
·
verified ·
1 Parent(s): abe9bb2

Update README

Browse files
Files changed (1) hide show
  1. README.md +24 -16
README.md CHANGED
@@ -29,7 +29,7 @@ The model captures:
29
  - Cross-lingual representations for medical text
30
  - Robust performance across diverse healthcare datasets
31
 
32
- ![paper_diagram](https://cdn-uploads.huggingface.co/production/uploads/690e53164670480464c0d207/JcNksj4nBefZ87WtxWOTl.png)
33
 
34
  ## Pretraining Data
35
 
@@ -37,27 +37,27 @@ The model is trained on a mixture of three data sources:
37
 
38
  ### 1. Clinical Data (BRIDGE Corpus)
39
  - 87 multilingual clinical datasets
40
- - ~1.42M documents
41
- - ~995M tokens
42
- - Languages: English, Chinese, Spanish, Japanese, Russian
43
 
44
  This dataset reflects real-world clinical practice and is the core contribution of this work.
45
 
46
  ### 2. Biomedical Literature (PubMed)
47
- - ~1.25M documents
48
- - ~194M tokens
49
 
50
  Provides domain knowledge and medical terminology.
51
 
52
  ### 3. General-Domain Text (Wikipedia)
53
- - ~5.8K documents
54
- - ~43M tokens
55
- - Languages: Spanish, Japanese, Russian
56
 
57
  Improves general linguistic coverage.
58
 
59
  ### Total
60
- - ~2.7M documents >1.2B tokens
61
 
62
 
63
  ## Pretraining Strategy
@@ -80,15 +80,23 @@ We adopt a **two-stage domain-adaptive pretraining approach**:
80
 
81
  ## Evaluation
82
 
83
- ![main_results](https://cdn-uploads.huggingface.co/production/uploads/690e53164670480464c0d207/in7j8ifSGRl35mL2KrLlK.png)
 
84
 
85
- We evaluate MultiClinicalBERT on **10 clinical NLP tasks across 5 languages**:
86
 
87
  - English: MIMIC-III Mortality, MIMIC-IV CDM
88
- - Chinese: CEMR, IMCS-V2 NER
89
  - Japanese: IFMIR Incident Type, IFMIR NER
90
- - Russian: RuMedNLI, RuCCoN NER
91
- - Spanish: EHR De-identification, PPTS
 
 
 
 
 
 
 
92
 
93
  ### Key Results
94
  - Consistently outperforms Modern multilingual BERT (mmBERT)
@@ -98,7 +106,7 @@ We evaluate MultiClinicalBERT on **10 clinical NLP tasks across 5 languages**:
98
  Example:
99
  - MIMIC-III Mortality: **89.40% accuracy**
100
  - MIMIC-IV CDM: **92.18% accuracy**
101
- - CEMR: **94.24% accuracy**
102
  - IFMIR NER: **86.21 F1**
103
  - EHR De-identification: **88.64 F1**
104
 
 
29
  - Cross-lingual representations for medical text
30
  - Robust performance across diverse healthcare datasets
31
 
32
+ ![main_workflow](https://cdn-uploads.huggingface.co/production/uploads/690e53164670480464c0d207/o78VXfg5xxjJXnKiirNYd.png)
33
 
34
  ## Pretraining Data
35
 
 
37
 
38
  ### 1. Clinical Data (BRIDGE Corpus)
39
  - 87 multilingual clinical datasets
40
+ - 1.42M documents
41
+ - 860.08M tokens
42
+ - Languages: English, Chinese, Spanish, Japanese
43
 
44
  This dataset reflects real-world clinical practice and is the core contribution of this work.
45
 
46
  ### 2. Biomedical Literature (PubMed)
47
+ - 1.25M documents
48
+ - 161.47M tokens
49
 
50
  Provides domain knowledge and medical terminology.
51
 
52
  ### 3. General-Domain Text (Wikipedia)
53
+ - 5.5K documents
54
+ - 14.73M tokens
55
+ - Languages: Spanish, Japanese
56
 
57
  Improves general linguistic coverage.
58
 
59
  ### Total
60
+ - ~2.7M documents >1B tokens
61
 
62
 
63
  ## Pretraining Strategy
 
80
 
81
  ## Evaluation
82
 
83
+ ### Clinical NLP Tasks
84
+ ![main_nlp_results](https://cdn-uploads.huggingface.co/production/uploads/690e53164670480464c0d207/xtFDoJDAcicub6K-0ZleT.png)
85
 
86
+ We evaluateD MultiClinicalBERT on **8 clinical NLP tasks across 4 languages**:
87
 
88
  - English: MIMIC-III Mortality, MIMIC-IV CDM
89
+ - Chinese: C-EMR, IMCS-V2 NER
90
  - Japanese: IFMIR Incident Type, IFMIR NER
91
+ - Spanish: EHR De-identification, PPTS
92
+
93
+ ### Retrieval Augmented Generation (RAG)
94
+
95
+ ![rag_results_figure](https://cdn-uploads.huggingface.co/production/uploads/690e53164670480464c0d207/I-r8vU_CVlSVCrLiyezft.png)
96
+
97
+ We further evaluated whether MultiClinicalBERT could enhance downstream clinical classification by retrieving external, unlabeled knowledge from language-matched Wikipedia corpora.
98
+
99
+ Across four clinical classification benchmarks and three Qwen3.5 model sizes (i.e., 2B, 4B, 9B), MultiClinicalBERT achieved the highest overall average performance among the seven retrieval approaches.
100
 
101
  ### Key Results
102
  - Consistently outperforms Modern multilingual BERT (mmBERT)
 
106
  Example:
107
  - MIMIC-III Mortality: **89.40% accuracy**
108
  - MIMIC-IV CDM: **92.18% accuracy**
109
+ - C-EMR: **94.24% accuracy**
110
  - IFMIR NER: **86.21 F1**
111
  - EHR De-identification: **88.64 F1**
112