anocha-mcomp commited on
Commit
2793a15
·
verified ·
1 Parent(s): 9aca1da

Update model card from Confluence: Model Card: SEA-LION-ModernBERT-300M

Browse files
model_card_sea-lion-modernbert-300m.md ADDED
@@ -0,0 +1,215 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Model Card for SEA-LION-ModernBERT-300M
2
+
3
+ *[Last update: 2026-03-16]*
4
+
5
+ SEA-LION is a collection of Large Language Models (LLMs) which have been pretrained and fine-tuned for the Southeast Asia (SEA) region.
6
+
7
+ This encoder-only model leverages the advanced **ModernBERT** architecture combined with the Gemma 3 SentencePiece tokenizer. The adoption of the **Gemma 3 tokenizer** with ModernBERT allows the model to achieve highly efficient and culturally nuanced text processing. This combination significantly improves the tokenization fertility and compression rates for complex regional scripts and diverse Southeast Asian languages, enabling the model to handle longer context windows and cross-lingual tasks with greater computational efficiency.
8
+
9
+ To achieve this level of performance, the model was developed through a rigorous, multi-stage training pipeline. The foundation was established through extensive **pre-training on 2 Trillion (2T) tokens**, followed by a mid-training phase on an additional 1 Trillion (1T) tokens. Both of these massive training phases comprehensively covered code alongside 13 specific languages: Burmese, Chinese, English, Filipino, Indonesian, Javanese, Khmer, Lao, Malay, Sundanese, Tamil, Thai, and Vietnamese.
10
+
11
+ ## Model Details
12
+
13
+ ### Model Description
14
+
15
+ The SEA-LION-ModernBERT-300M models are built on the [ModernBERT-base](https://huggingface.co/answerdotai/ModernBERT-base) architecture and has a vocabulary size of 262K.
16
+
17
+ For tokenization, the model employs our custom [Gemma3](https://storage.googleapis.com/deepmind-media/gemma/Gemma3Report.pdf) tokenizer, which has excellent performance for SEA languages, ensuring optimal model performance.
18
+
19
+ - **Developed by:** AI Products Pillar, AI Singapore
20
+ - **Funded by:** Singapore NRF
21
+ - **Shared by:** AI Products Pillar, AI Singapore
22
+ - **Model type:** Encoder
23
+ - **Context length:** 8k
24
+ - **Languages:** Burmese, Chinese, English, Filipino, Indonesian, Javanese, Khmer, Lao, Malay, Sundanese, Tamil, Thai, and Vietnamese
25
+ - **License:** [MIT](https://tlo.mit.edu/understand-ip/exploring-mit-open-source-license-comprehensive-guide)
26
+
27
+ ### Model Sources
28
+
29
+ - **Repository:** The weights for this model and its various training stages are being released to support transparency, research, and diverse downstream applications. **[**[**Link to HF Repo**](https://huggingface.co/aisingapore/SEA-LION-ModernBERT-300M-checkpoints)**]**
30
+
31
+ ## Uses
32
+
33
+ This model card details one of the variants available within this collection.
34
+
35
+ | Model Variant | Model Repository | Suggesting Applications & Use Cases |
36
+ | --- | --- | --- |
37
+ | **Fine-tuned Embedding Models** | - [aisingapore/SEA-LION-E5-Embedding-600M](https://huggingface.co/aisingapore/SEA-LION-E5-Embedding-600M)
38
+ - [aisingapore/SEA-LION-ModernBERT-Embedding-300M](https://huggingface.co/aisingapore/SEA-LION-ModernBERT-Embedding-300M)
39
+ - [aisingapore/SEA-LION-ModernBERT-Embedding-600M](https://huggingface.co/aisingapore/SEA-LION-ModernBERT-Embedding-600M)
40
+ | - Retrieval-Augmented Generation (RAG)
41
+ - Information retrieval, and search
42
+ - Similarity comparisons
43
+ |
44
+ | **Pre-trained Encoder Models** | - [aisingapore/SEA-LION-ModernBERT-300M](https://huggingface.co/aisingapore/SEA-LION-ModernBERT-300M)
45
+ - [aisingapore/SEA-LION-ModernBERT-600M](https://huggingface.co/aisingapore/SEA-LION-ModernBERT-600M)
46
+ | - Fill mask
47
+ - Text classification
48
+ - Fine-tuning for downstream tasks (e.g., sentiment analysis, classification).
49
+ |
50
+ | **Pre-trained Model Checkpoints** | - [aisingapore/SEA-LION-ModernBERT-300M-checkpoints](https://huggingface.co/aisingapore/SEA-LION-ModernBERT-300M-checkpoints)
51
+ - [aisingapore/SEA-LION-ModernBERT-600M-checkpoints](https://huggingface.co/aisingapore/SEA-LION-ModernBERT-600M-checkpoints)
52
+ | - Continued Pre-Training (CPT)
53
+ - Fine-tuning for downstream tasks (e.g., sentiment analysis, classification).
54
+ |
55
+
56
+ *Note: If you are deploying our models for your specific use case, we would love to hear from you! Please feel free to* [*contact us*](mailto:sealion@aisingapore.org) *to share your experience or explore potential collaborations.*
57
+
58
+ ### Bias, Risks, and Limitations
59
+
60
+ The model was not tested for robustness against adversarial usage. It is important for users to be aware that our model exhibits certain limitations that warrant consideration. Users should also exercise caution in continue-implementing and validating the model's responses due to the potential inconsistencies.
61
+
62
+ ### Recommendations
63
+
64
+ Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model.
65
+
66
+ ## How to Get Started with the Model
67
+
68
+ Use the code below to download the model locally.
69
+
70
+ ```
71
+ pip install -U transformers>=4.48.0
72
+ ```
73
+
74
+ ```
75
+ import torch
76
+ from transformers import pipeline
77
+
78
+ pipeline = pipeline(
79
+ task="fill-mask",
80
+ model="aisingapore/SEA-LION-ModernBERT-300M",
81
+ dtype=torch.float16,
82
+ device=0
83
+ )
84
+ pipeline("Plants create through a process known as photosynthesis.")
85
+ ```
86
+
87
+ ---
88
+
89
+ ## Training Details
90
+
91
+ The models are pre-trained from scratch through a two-phase pipeline, beginning with an extensive initial stage on 2 trillion tokens, followed by a mid-training phase on an additional 1 trillion tokens. Both phases incorporated a diverse dataset covering programming code and 13 languages: Burmese, Chinese, English, Filipino, Indonesian, Javanese, Khmer, Lao, Malay, Sundanese, Tamil, Thai, and Vietnamese.
92
+
93
+ ### Training Data
94
+
95
+ **SEA-LION-ModernBERT-300M-checkpoints** was pre-trained from scratch on a number of trillion tokens corpus with the following linguistic and thematic distribution:
96
+
97
+ | Data Source | Percentage |
98
+ | --- | --- |
99
+ | code | 10% |
100
+ | EN - English | 35% |
101
+ | ID - Indonesian | 8% |
102
+ | JV - Javanese | 0.5% |
103
+ | KM - Khmer | 1.5% |
104
+ | LO - Lao | 0.5% |
105
+ | MS - Malay | 4.75% |
106
+ | MY - Burmese | 1.75% |
107
+ | SU - Sundanese | 0.5% |
108
+ | TA - Tamil | 4.5% |
109
+ | TH - Thai | 8% |
110
+ | TL - Filipino | 2.5% |
111
+ | VI - Vietnamese | 8.5% |
112
+ | ZH - Chinese | 14% |
113
+
114
+ ## Evaluation
115
+
116
+ ### Testing Data, Factors & Metrics
117
+
118
+ ### Testing Data, Factors & Metrics
119
+
120
+ #### Testing Data
121
+
122
+ The model is evaluated across three primary benchmark suites to provide a comprehensive assessment of embedding quality across Southeast Asian, Chinese, and English contexts:
123
+
124
+ - **[SEA-BED (Southeast Asia Embedding Benchmark)]**(<https://arxiv.org/pdf/2508.12243>): The primary testing suite, consisting of 169 datasets across 10 Southeast Asian languages (Burmese, Filipino, Indonesian, Khmer, Malay, Lao, Tamil, Tetum, Thai, and Vietnamese). Notably, 71% of these datasets are native-authored or human-curated to preserve regional linguistic properties.
125
+ - **CMTEB (Chinese Massive Text Embedding Benchmark)**: A specialised subset of MTEB focused on Chinese language tasks, used to evaluate performance in one of the region's most prominent scripts.
126
+ - **MTEB (Massive Text Embedding Benchmark)**: The industry-standard global benchmark used to gauge general-purpose English embedding performance across a wide array of tasks.
127
+
128
+ ### Factors
129
+
130
+ Evaluation factors are categorised by task type and linguistic diversity to ensure the model's "fertility" and "nuance" are captured accurately:
131
+
132
+ - **Linguistic Coverage**: Evaluation spans across 10+ languages, including complex Brahmic scripts (Burmese, Khmer, Lao, Tamil, Thai) and Latin-based SEA scripts (Indonesian, Filipino, Malay, Tetum, Vietnamese).
133
+ - **Task Modality**:
134
+
135
+ - **Retrieval/Reranking:** Efficiency in finding relevant documents within a large corpus.
136
+ - **Semantic Textual Similarity (STS):** Precision in sentence-level semantic alignment.
137
+ - **Clustering & Classification:** Ability to group or categorize text based on latent semantic meaning.
138
+ - **Summarisation & Bitext Mining:** High-level semantic matching and cross-lingual alignment.
139
+ - **Architecture Efficiency:** Performance is measured in the context of the ModernBERT architecture and Gemma 3 tokenizer to assess computational efficiency versus embedding quality.
140
+
141
+ #### Metrics
142
+
143
+ To provide a standardized view of performance, we report the following metrics across the benchmark suites:
144
+
145
+ - **Classification:** F1-score.
146
+ - **Multi-label Classification:** F1-score
147
+ - **Pair Classification:** Average Precision (AP).
148
+ - **Semantic Textual Similarity (STS):** Cosine similarity scores.
149
+ - **Clustering:** V-Measure Score.
150
+ - **Bitext Mining:** F1-score.
151
+ - **Retrieval & Reranking:** NDCG@10 (Primary) and MAP.
152
+ - **Instruction Retrieval:** NNDCG@5
153
+
154
+ ### Results
155
+
156
+ [Attachment: image-20260313-064345.png]Performance comparison of embedding models on `SEA-BED` (<https://leaderboard.sea-lion.ai/embedding/SEA> ) captured on 13/03/2026 02:50pm.
157
+
158
+ ## Environmental Impact
159
+
160
+ Carbon emission was estimated using the fact sheet from TRG [Datacenters](https://www.trgdatacenters.com/resource/h200-power-consumption/).
161
+
162
+ - **Hardware Type:** Nvidia H200 140GB GPUs
163
+ - **Hours used:** 1,825 GPU hours
164
+ - **Cloud Provider:** SMC H200
165
+ - **Compute Region:** Singapore
166
+ - **Carbon Emitted:** appx. 513.27 kg CO2 e
167
+
168
+ ## Technical Specifications
169
+
170
+ ### Model Architecture and Objective
171
+
172
+ SEA-LION-ModernBERT-300M is an encoder model using the ModernBERT architecture.
173
+
174
+ | Parameter | SEA-LION-ModernBERT |
175
+ | --- | --- |
176
+ | Layers | 22 |
177
+ | d_model | 768 |
178
+ | head_dim | 12 |
179
+ | Vocabulary | 262144 |
180
+ | Sequence Length | 8k |
181
+
182
+ ### Compute Infrastructure
183
+
184
+ #### Hardware
185
+
186
+ - **Hardware Type:** Nvidia H200 140GB GPUs
187
+ - **Cloud Provider:** SMC H200
188
+
189
+ #### Software
190
+
191
+ SEA-LION was trained using the [ModernBERT code base](https://github.com/AnswerDotAI/ModernBERT) which is powered by the [Composer](https://github.com/mosaicml/composer) training framework from MosaicML.
192
+
193
+ ## Glossary
194
+
195
+ - **SEA-BED:** Southeast Asia Embedding Benchmark – a comprehensive evaluation suite for embedding models on SEA languages.
196
+ - **Asymmetric Retrieval:** Retrieval tasks where query and document formulations differ.
197
+ - **Mean Pooling:** Aggregating token embeddings by averaging (weighted by attention mask) to produce a fixed-size sentence representation.
198
+
199
+ ## More Information
200
+
201
+ This is the repository for the commercial fine-tuned model. The model has *not* been aligned for safety. Developers and users should perform their own safety fine-tuning and related security measures. In no event shall the authors be held liable for any claims, damages, or other liabilities arising from the use of the released weights and codes.
202
+
203
+ For more info, please contact us at [sealion@aisingapore.org](mailto:sealion@aisingapore.org)
204
+
205
+ ## Team
206
+
207
+ Ahmed Dabeer, Ahn Jeongmi, Antonyrex Sajeban, Chan Hok Teng Adwin, Cheng Zi Yi Nicholas, Choa Hsueh Mei Esther, Heng Jonathan, Huang Yuli, Jann Railey Estrada Montalan, Lee Chwan Ren, Leong Wai Yi, Leong Wei Qi, Liew Rachel, **Limkonchotiwat Peerat**, Muhammad Ridzuan Bin Mokhtar, Nagarajan Karthik, **Ng Boon Cheong Raymond**, Ngee Chia Tai, Ngui Jian Gang, Nguyen Thanh Ngan, Ong Tat-Wee David, Ong Zhi Hao, Pereira Mark, Poon Joseph, Rengarajan Hamsawardhini, Siow Wei Kang Bryan, Susanto Yosephine, Sutaveephamochanon Anocha, Tan Choon Meng, Tan Chor Phin Evelyn, Tan Siao Wei Jessica, Tan Yixian, Tee Jun Yun, Teng Kok Wai Walter, Teo Eng Sipp Leslie, Tjhi William, Wu Donghang, Yeo Yeow Tong, Yong Xianbin, Zhang Haoyang, Zhang Zhou
208
+
209
+ ## **Acknowledgement**
210
+
211
+ This project is supported by the National Research Foundation Singapore and Infocomm Media Development Authority (IMDA), Singapore under its National Large Language Model Funding Initiative.
212
+
213
+ ## Contact
214
+
215
+ [sealion@aisingapore.org](mailto:sealion@aisingapore.org)