anocha-mcomp commited on
Commit
c58251b
·
verified ·
1 Parent(s): e2b5098

Update model card from Confluence: Model Card: SEA-LION-cptr-600M

Browse files
Files changed (1) hide show
  1. model_card_sea-lion-cptr-600m.md +210 -0
model_card_sea-lion-cptr-600m.md ADDED
@@ -0,0 +1,210 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Model Card for SEA-LION-cptr-600M
2
+
3
+ *[Last update: 2026-03-06]*
4
+
5
+ SEA-LION is a collection of Large Language Models (LLMs) which have been pretrained and fine-tuned for the Southeast Asia (SEA) region.
6
+
7
+ This encoder-only model leverages the advanced ModernBERT architecture combined with the Gemma 3 SentencePiece tokenizer. The adoption of the Gemma 3 tokenizer on top of ModernBERT allows the model to achieve highly efficient and culturally nuanced text processing. This combination significantly improves the tokenization fertility and compression rates for complex regional scripts and diverse Southeast Asian languages, enabling the model to handle longer context windows and cross-lingual tasks with greater computational efficiency.
8
+
9
+ To achieve this level of performance, the model was developed through a rigorous, multi-stage training pipeline. The foundation was established through extensive pre-training on 2 Trillion (2T) tokens, followed by a mid-training phase on an additional 1 Trillion (1T) tokens. Both of these massive training phases comprehensively covered code alongside 13 specific languages: Burmese, Chinese, English, Filipino, Indonesian, Javanese, Khmer, Lao, Malay, Sundanese, Tamil, Thai, and Vietnamese.
10
+
11
+ ## Model Details
12
+
13
+ ### Model Description
14
+
15
+ The SEA-LION-cptr-600M model is built on the [ModernBERT-large](https://huggingface.co/answerdotai/ModernBERT-large) architecture and has a vocabulary size of 262K.
16
+
17
+ For tokenization, the model employs our custom [Gemma3](https://storage.googleapis.com/deepmind-media/gemma/Gemma3Report.pdf) tokenizer, which is specially tailored for SEA languages, ensuring optimal model performance.
18
+
19
+ - **Developed by:** AI Products Pillar, AI Singapore
20
+ - **Funded by:** Singapore NRF
21
+ - **Shared by:** AI Products Pillar, AI Singapore
22
+ - **Model type:** Encoder
23
+ - **Context length:** 8K
24
+ - **Languages:** Burmese, Chinese, English, Filipino, Indonesian, Javanese, Khmer, Lao, Malay, Sundanese, Tamil, Thai, and Vietnamese
25
+ - **License:** [Apache-2.0](https://choosealicense.com/licenses/apache-2.0/)
26
+ - **Continued pretrained from model:** [answerdotai/ModernBERT-large](<https://huggingface.co/answerdotai/ModernBERT-large>)
27
+
28
+ ### Model Sources
29
+
30
+ - **Repository:** The weights for this model and its various training stages are being released to support transparency, research, and diverse downstream applications.
31
+
32
+ | Weight / Model Variant | Description | Repository Link |
33
+ | --- | --- | --- |
34
+ | **Instruction-Tuned (Final)** | The final aligned model optimized for instruction-following and conversational tasks. | `[Link to HF Repo]` |
35
+ | **cptr Model (No Decay)** | The foundational model without learning rate decay applied. Ideal for continued pre-training (CPT) on custom domain data. | `[Link to HF Repo]` |
36
+ | **cptr Model (Decay)** | The foundational model with learning rate decay applied. Optimized for direct fine-tuning on downstream tasks (e.g., classification, sentiment analysis). | `[Link to HF Repo]` |
37
+ | **Intermediate Checkpoints** | Weight snapshots saved during the pre-training and mid-training phases. Released for research, transparency, and training dynamics analysis. | `[Link to HF Repo]` |
38
+
39
+ *Note: If you are deploying our models for your specific use case, we would love to hear from you! Please feel free to* [*contact us*](mailto:sealion@aisingapore.org) *to share your experience or explore potential collaborations.*
40
+
41
+ ## Uses
42
+
43
+ This model card details one of the variants available within this collection.
44
+
45
+ | Model Variant | Suggesting Applications & Use Cases |
46
+ | --- | --- |
47
+ | **Embedding Models** | Retrieval-Augmented Generation (RAG), retrieval, and similarity comparisons. |
48
+ | **cptr Model (No Decay)** | Continued Pre-Training (CPT) and fine-tuning for downstream tasks (e.g., sentiment analysis, classification). |
49
+ | **cptr Model (Decay)** | Fine-tuning for downstream tasks (e.g., sentiment analysis, classification). |
50
+
51
+ ### Out-of-Scope Use
52
+
53
+ The model has not been aligned for safety. Developers and users should perform their own safety fine-tuning and related security measures. In no event shall the authors be held liable for any claims, damages, or other liabilities arising from the use of the released weights and codes.
54
+
55
+ ## Bias, Risks, and Limitations
56
+
57
+ *The model was not tested for robustness against adversarial prompting.* It is important for users to be aware that our model exhibits certain limitations that warrant consideration. Like many LLMs, the model can hallucinate and occasionally generates irrelevant content, introducing fictional elements that are not grounded in the provided context. Users should also exercise caution in interpreting and validating the model's responses due to the potential inconsistencies.
58
+
59
+ ## How to Get Started with the Model
60
+
61
+ Use the code below to get started with the model. ?????
62
+
63
+ ```
64
+ pip install -U transformers>=4.48.0
65
+ ```
66
+
67
+ ```
68
+ #########################
69
+ ## Need revision !!!
70
+ ## Example code adopted from https://huggingface.co/docs/transformers/main/en/model_doc/modernbert#modernbert
71
+ #########################
72
+ import torch
73
+ from transformers import pipeline
74
+
75
+ pipeline = pipeline(
76
+ task="fill-mask",
77
+ model="aisingapore/SEA-LION-ModernBERT-base",
78
+ dtype=torch.float16,
79
+ device=0
80
+ )
81
+ pipeline("The term [MASK] is an Indonesian word commonly utilized within Changi Airport.")
82
+ ```
83
+ ## Training Details
84
+
85
+ The model underwent continued pre-training through a two-phase pipeline, beginning with an extensive initial stage on 2 trillion tokens, followed by a mid-training phase on an additional 1 trillion tokens. Both phases incorporated a diverse dataset covering programming code and 13 languages: Burmese, Chinese, English, Filipino, Indonesian, Javanese, Khmer, Lao, Malay, Sundanese, Tamil, Thai, and Vietnamese.
86
+
87
+ ### Training Data
88
+
89
+ **SEA-LION-ModernBERT** was continuing pre-trained on a number of trillion token corpus with the following linguistic and thematic distribution:
90
+
91
+ | Data Source | Percentage |
92
+ | --- | --- |
93
+ | code | 10% |
94
+ | EN - English | 35% |
95
+ | ID - Indonesian | 8% |
96
+ | JV - Javanese | 0.5% |
97
+ | KM - Khmer | 1.5% |
98
+ | LO - Lao | 0.5% |
99
+ | MS - Malay | 4.75% |
100
+ | MY - Burmese | 1.75% |
101
+ | SU - Sundanese | 0.5% |
102
+ | TA - Tamil | 4.5% |
103
+ | TH - Thai | 8% |
104
+ | TL - Filipino | 2.5% |
105
+ | VI - Vietnamese | 8.5% |
106
+ | ZH - Chinese | 14% |
107
+
108
+ ### Training Procedure
109
+
110
+ #### Training Hyperparameters
111
+
112
+ - **Training regime:**
113
+
114
+ | Hyper Parameter | cptr model (no decay) | cptr model (decay) |
115
+ | --- | --- | --- |
116
+ | Precision | bfloat16??? | bfloat16??? |
117
+ | Optimizer | decoupled_adamw??? | decoupled_adamw??? |
118
+ | Scheduler | ??? | linear_decay_with_warmup??? |
119
+
120
+ ## Evaluation
121
+
122
+ ### Contents are auto-generated. Need revision !!!
123
+
124
+ ### Testing Data, Factors & Metrics
125
+
126
+ ### Testing Data
127
+
128
+ The model is evaluated across three primary benchmark suites to provide a comprehensive assessment of embedding quality across Southeast Asian, Chinese, and Global contexts:
129
+
130
+ - **SEA-BED (Southeast Asia Embedding Benchmark):** The primary testing suite, consisting of 169 datasets across 10 Southeast Asian languages (Indonesian, Thai, Vietnamese, Burmese, Filipino, Tamil, Khmer, Malay, Lao, and Tetum). Notably, 71% of these datasets are native-authored or human-curated to preserve regional linguistic properties.
131
+ - **CMTEB (Chinese Massive Text Embedding Benchmark):** A specialized subset of MTEB focused on Chinese language tasks, used to evaluate performance in one of the region's most prominent scripts.
132
+ - **MTEB (Massive Text Embedding Benchmark):** The industry-standard global benchmark used to gauge general-purpose multilingual and English embedding performance.
133
+
134
+ ### Factors
135
+
136
+ Evaluation factors are categorized by task type and linguistic diversity to ensure the model's "fertility" and "nuance" are captured accurately:
137
+
138
+ - **Linguistic Coverage:** Testing spans across 13+ languages, including complex Brahmic scripts (Thai, Lao, Khmer, Burmese) and Latin-based SEA scripts (Indonesian, Vietnamese, Malay).
139
+ - **Task Modality:** * **Retrieval/Reranking:** Efficiency in finding relevant documents within a large corpus.
140
+
141
+ - **Semantic Textual Similarity (STS):** Precision in sentence-level semantic alignment.
142
+ - **Clustering & Classification:** Ability to group or categorize text based on latent semantic meaning.
143
+ - **Summarization & Bitext Mining:** High-level semantic matching and cross-lingual alignment.
144
+ - **Architecture Efficiency:** Performance is measured in the context of the ModernBERT architecture and Gemma 3 tokenizer to assess computational efficiency versus embedding quality.
145
+
146
+ ### Metrics
147
+
148
+ To provide a standardized view of performance, we report the following metrics across the benchmark suites:
149
+
150
+ - **Retrieval & Reranking:** NDCG@10 (Primary), Mean Reciprocal Rank (MRR), and MAP.
151
+ - **Semantic Textual Similarity (STS):** Spearman Correlation based on cosine similarity scores.
152
+ - **Classification:** Accuracy and F1-score.
153
+ - **Clustering:** V-Measure and Silhouette Score.
154
+ - **Pair Classification:** Average Precision (AP).
155
+ - **Bitext Mining:** F1 (Cosine Similarity).Results
156
+
157
+ {{ results | default("[More Information Needed]", true)}}
158
+
159
+ ## Environmental Impact
160
+
161
+ Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
162
+
163
+ - **Hardware Type:** Nvidia H200 140GB GPUs
164
+ - **Hours used:** ????????? hrs
165
+ - **Cloud Provider:** SMC H200
166
+ - **Compute Region:** Singapore
167
+ - **Carbon Emitted:** appx. 27 kg CO2 e
168
+
169
+ ## Technical Specifications
170
+
171
+ ### Model Architecture and Objective
172
+
173
+ SEA-LION-cptr-600M is an encoder model using the ModernBERT architecture.
174
+
175
+ | Parameter | SEA-LION-ModernBERT |
176
+ | --- | --- |
177
+ | Layers | 22 in 300M / 28 in 600M |
178
+ | d_model | 768 in 300M / 1024 in 600M |
179
+ | head_dim | 12 in 300M / 16 for 600M |
180
+ | Vocabulary | 262144 |
181
+ | Sequence Length | 128/ 8k |
182
+
183
+ ### Compute Infrastructure
184
+
185
+ #### Hardware
186
+
187
+ - **Hardware Type:** Nvidia H200 140GB GPUs
188
+ - **Cloud Provider:** SMC H200
189
+
190
+ #### Software
191
+
192
+ SEA-LION was trained using ???
193
+
194
+ ## More Information
195
+
196
+ This is the repository for the commercial fine-tuned model. The model has *not* been aligned for safety. Developers and users should perform their own safety fine-tuning and related security measures. In no event shall the authors be held liable for any claims, damages, or other liabilities arising from the use of the released weights and codes.
197
+
198
+ For more info, please contact us at [sealion@aisingapore.org](mailto:sealion@aisingapore.org)
199
+
200
+ ## Team
201
+
202
+ Ahmed Dabeer, Ahn Jeongmi, Antonyrex Sajeban, Chan Hok Teng Adwin, Cheng Zi Yi Nicholas, Choa Hsueh Mei Esther, Heng Jonathan, Huang Yuli, Jann Railey Estrada Montalan, Lee Chwan Ren, Leong Wai Yi, Leong Wei Qi, Liew Rachel, Limkonchotiwat Peerat, Muhammad Ridzuan Bin Mokhtar, Nagarajan Karthik, Ng Boon Cheong Raymond, Ngee Chia Tai, Ngui Jian Gang, Nguyen Thanh Ngan, Ong Tat-Wee David, Ong Zhi Hao, Pereira Mark, Poon Joseph, Rengarajan Hamsawardhini, Siow Wei Kang Bryan, Susanto Yosephine, Sutaveephamochanon Anocha, Tan Choon Meng, Tan Chor Phin Evelyn, Tan Siao Wei Jessica, Tan Yixian, Tee Jun Yun, Teng Kok Wai Walter, Teo Eng Sipp Leslie, Tjhi William, Wu Donghang, Yeo Yeow Tong, Yong Xianbin, Zhang Zhou
203
+
204
+ ## **Acknowledgement**
205
+
206
+ This project is supported by the National Research Foundation Singapore and Infocomm Media Development Authority (IMDA), Singapore under its National Large Language Model Funding Initiative.
207
+
208
+ ## Contact
209
+
210
+ [sealion@aisingapore.org](mailto:sealion@aisingapore.org)