Instructions to use anocha-mcomp/lora_model with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use anocha-mcomp/lora_model with Transformers:
# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("anocha-mcomp/lora_model", device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- Unsloth Desktop
Update model card from Confluence: Model Card: SEA-LION-ModernBERT-Embedding-300M
Browse files
model_card_sea-lion-modernbert-embedding-300m.md
ADDED
|
@@ -0,0 +1,227 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Model Card for SEA-LION-ModernBERT-Embedding-300M
|
| 2 |
+
|
| 3 |
+
*[Last update: 2026-03-16]*
|
| 4 |
+
|
| 5 |
+
SEA-LION is a collection of Large Language Models (LLMs) which have been pretrained and fine-tuned for the Southeast Asia (SEA) region.
|
| 6 |
+
|
| 7 |
+
Leveraging the advanced encoder-only **ModernBERT** architecture combined with the **Gemma 3 SentencePiece tokenizer**, this specific model achieves highly efficient and culturally nuanced text processing with improved tokenization fertility and compression rates for complex regional scripts, enabling it to handle longer context windows and cross-lingual tasks with greater computational efficiency. Using the [**SEA-LION-ModernBERT-300M-checkpoints**](https://huggingface.co/aisingapore/SEA-LION-ModernBERT-300M-checkpoints) as the base model, this model is contrastively pre-trained on 245 million text pairs (EN-EN and EN-SEA) and finally instruction-tuned using a diverse dataset of 13 million text pairs (spanning EN-EN, CN-CN, EN-SEA, and SEA-SEA) to create the final instruction-tuned model.
|
| 8 |
+
|
| 9 |
+
## Model Details
|
| 10 |
+
|
| 11 |
+
### Model Description
|
| 12 |
+
|
| 13 |
+
The SEA-LION-ModernBERT-Embedding-300M model is built on the [ModernBERT-base](https://huggingface.co/answerdotai/ModernBERT-base) architecture and has a vocabulary size of 262K.
|
| 14 |
+
|
| 15 |
+
For tokenization, the model employs our custom [Gemma 3](https://storage.googleapis.com/deepmind-media/gemma/Gemma3Report.pdf) tokenizer, which has excellent performance for SEA languages, ensuring optimal model performance.
|
| 16 |
+
|
| 17 |
+
- **Developed by:** AI Products Pillar, AI Singapore
|
| 18 |
+
- **Funded by:** Singapore NRF
|
| 19 |
+
- **Shared by:** AI Products Pillar, AI Singapore
|
| 20 |
+
- **Model type:** Encoder
|
| 21 |
+
- **Context length:** 8k
|
| 22 |
+
- **Languages:** Burmese, Chinese, English, Filipino, Indonesian, Khmer, Lao, Malay, Tamil, Thai, and Vietnamese
|
| 23 |
+
- **License:** [MIT](https://tlo.mit.edu/understand-ip/exploring-mit-open-source-license-comprehensive-guide)
|
| 24 |
+
- **Finetuned from model:** [aisingapore/SEA-LION-ModernBERT-checkpoints](https://huggingface.co/aisingapore/SEA-LION-ModernBERT-300M-checkpoints)
|
| 25 |
+
|
| 26 |
+
### Model Sources
|
| 27 |
+
|
| 28 |
+
- **Repository:** We are releasing the weights for every training stage to promote transparency and research and to support a wide range of downstream applications. The SEA-LION-ModernBERT-Embedding-300M model can be accessed at the **[**[**Link to HF Repo**](https://huggingface.co/aisingapore/SEA-LION-ModernBERT-Embedding-300M)**]**.
|
| 29 |
+
|
| 30 |
+
## Uses
|
| 31 |
+
|
| 32 |
+
This model card details one of the variants available within this [collection](https://huggingface.co/collections/aisingapore/sea-lion-modernbert-and-sea-lion-embedding).
|
| 33 |
+
|
| 34 |
+
| Model Variant | Model Repository | Suggesting Applications & Use Cases |
|
| 35 |
+
| --- | --- | --- |
|
| 36 |
+
| **Fine-tuned Embedding Models** | - [aisingapore/SEA-LION-E5-Embedding-600M](https://huggingface.co/aisingapore/SEA-LION-E5-Embedding-600M)
|
| 37 |
+
- [aisingapore/SEA-LION-ModernBERT-Embedding-300M](https://huggingface.co/aisingapore/SEA-LION-ModernBERT-Embedding-300M)
|
| 38 |
+
- [aisingapore/SEA-LION-ModernBERT-Embedding-600M](https://huggingface.co/aisingapore/SEA-LION-ModernBERT-Embedding-600M)
|
| 39 |
+
| - Retrieval-Augmented Generation (RAG)
|
| 40 |
+
- Information retrieval, and search
|
| 41 |
+
- Similarity comparisons
|
| 42 |
+
|
|
| 43 |
+
| **Pre-trained Encoder Models** | - [aisingapore/SEA-LION-ModernBERT-300M](https://huggingface.co/aisingapore/SEA-LION-ModernBERT-300M)
|
| 44 |
+
- [aisingapore/SEA-LION-ModernBERT-600M](https://huggingface.co/aisingapore/SEA-LION-ModernBERT-600M)
|
| 45 |
+
| - Fill mask
|
| 46 |
+
- Text classification
|
| 47 |
+
- Fine-tuning for downstream tasks (e.g., sentiment analysis, classification).
|
| 48 |
+
|
|
| 49 |
+
| **Pre-trained Model Checkpoints** | - [aisingapore/SEA-LION-ModernBERT-300M-checkpoints](https://huggingface.co/aisingapore/SEA-LION-ModernBERT-300M-checkpoints)
|
| 50 |
+
- [aisingapore/SEA-LION-ModernBERT-600M-checkpoints](https://huggingface.co/aisingapore/SEA-LION-ModernBERT-600M-checkpoints)
|
| 51 |
+
| - Continued Pre-Training (CPT)
|
| 52 |
+
- Fine-tuning for downstream tasks (e.g., sentiment analysis, classification).
|
| 53 |
+
|
|
| 54 |
+
|
| 55 |
+
### Bias, Risks, and Limitations
|
| 56 |
+
|
| 57 |
+
The model was not tested for robustness against adversarial usage. It is important for users to be aware that our model exhibits certain limitations that warrant consideration. Users should also exercise caution in continue-implementing and validating the model's responses due to the potential inconsistencies.
|
| 58 |
+
|
| 59 |
+
### Recommendations
|
| 60 |
+
|
| 61 |
+
Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model.
|
| 62 |
+
|
| 63 |
+
## How to Get Started with the Model
|
| 64 |
+
|
| 65 |
+
Use the code below to get started with the model.
|
| 66 |
+
|
| 67 |
+
```
|
| 68 |
+
pip install -U sentence-transformers
|
| 69 |
+
```
|
| 70 |
+
|
| 71 |
+
```
|
| 72 |
+
from sentence_transformers import SentenceTransformer
|
| 73 |
+
|
| 74 |
+
model = SentenceTransformer(
|
| 75 |
+
"aisingapore/SEA-LION-ModernBERT-Embedding-300M",
|
| 76 |
+
prompts={
|
| 77 |
+
"STS": "Instruct: Retrieve semantically similar text.\nQuery: ",
|
| 78 |
+
"Clustering": "Instruct: Classify text into its appropriate category\nQuery: ",
|
| 79 |
+
"Classification": "Instruct: Classify text into its appropriate category\nQuery: ",
|
| 80 |
+
"Retrieval": "Instruct: Given a passage that is guaranteed to contain the answer, retrieve relevant passages that answer the query.\nQuery: ",
|
| 81 |
+
"BitextMining": "Instruct: Retrieve parallel sentences.\nQuery: ",
|
| 82 |
+
"PairClassification": "Instruct: Retrieve semantically similar text.\nQuery: ",
|
| 83 |
+
"Reranking": "Instruct: Retrieve semantically similar text.\nQuery: ",
|
| 84 |
+
"InstructionRetrieval": "Instruct: Given a instruction and a output, retrieve the most relevant output that answer the instruction.\nQuery: ",
|
| 85 |
+
"MultiLabelTextClassification": "Instruct: Classify the given text into its appropriate classes\nQuery: ",
|
| 86 |
+
"QARetrieval": "Instruct: Given a passage that is guaranteed to contain the answer, retrieve relevant passages that answer the query.\nQuery: ",
|
| 87 |
+
"Summarization": "Instruct: Summarize the given text.\nQuery: "
|
| 88 |
+
},
|
| 89 |
+
)
|
| 90 |
+
|
| 91 |
+
sentences = [
|
| 92 |
+
"The weather is lovely today.",
|
| 93 |
+
"อากาศวันนี้ดีมาก",
|
| 94 |
+
"Dia berkendara ke stadion.",
|
| 95 |
+
]
|
| 96 |
+
embeddings = model.encode(sentences, prompt_name="STS")
|
| 97 |
+
print(embeddings.shape)
|
| 98 |
+
# [3, 1024]
|
| 99 |
+
|
| 100 |
+
similarities = model.similarity(embeddings, embeddings)
|
| 101 |
+
print(similarities)
|
| 102 |
+
#tensor([[1.0000, 0.8594, 0.5534],
|
| 103 |
+
# [0.8594, 1.0000, 0.6343],
|
| 104 |
+
# [0.5534, 0.6343, 1.0000]])
|
| 105 |
+
```
|
| 106 |
+
## Training Details
|
| 107 |
+
|
| 108 |
+
### Training Data
|
| 109 |
+
|
| 110 |
+
This model was tuned using a multi-stage training pipeline with the following datasets:
|
| 111 |
+
|
| 112 |
+
- **Contrastive Pre-training:** 245 million text pairs (EN-EN and EN-SEA) to enhance cross-lingual alignment.
|
| 113 |
+
- **Fine-tuning:** 13 million diverse text pairs (spanning EN-EN, CN-CN, EN-SEA, and SEA-SEA) to create the final fine-tuned model.
|
| 114 |
+
|
| 115 |
+
| **Language** | Percentage |
|
| 116 |
+
| --- | --- |
|
| 117 |
+
| EN-EN | 20% |
|
| 118 |
+
| CN-CN | 20% |
|
| 119 |
+
| EN-SEA | 10% |
|
| 120 |
+
| SEA-SEA | 50% |
|
| 121 |
+
|
| 122 |
+
### Training Procedure
|
| 123 |
+
|
| 124 |
+
#### Preprocessing
|
| 125 |
+
|
| 126 |
+
Following the foundational training, the model's cross-lingual alignment was substantially enhanced by undergoing contrastive pre-training utilising 245 million text pairs, specifically focusing on English-to-English and English-to-Southeast Asian language mappings (EN-EN and EN-SEA). Finally, to ensure the model could effectively follow user instructions and handle complex interactions, it was instruction-tuned using a diverse dataset of 13 million text pairs spanning EN-EN, CN-CN, EN-SEA, and SEA-SEA, culminating in the final highly capable instruction-tuned model.
|
| 127 |
+
|
| 128 |
+
## Evaluation
|
| 129 |
+
|
| 130 |
+
### Testing Data, Factors & Metrics
|
| 131 |
+
|
| 132 |
+
### Testing Data
|
| 133 |
+
|
| 134 |
+
The model is evaluated across three primary benchmark suites to provide a comprehensive assessment of embedding quality across Southeast Asian, Chinese, and English contexts:
|
| 135 |
+
|
| 136 |
+
- **SEA-BED (Southeast Asia Embedding Benchmark) (**<https://arxiv.org/pdf/2508.12243>**):** The primary testing suite, consisting of 169 datasets across 10 Southeast Asian languages (Burmese, Filipino, Indonesian, Khmer, Malay, Lao, Tamil, Tetum, Thai, and Vietnamese). Notably, 71% of these datasets are native-authored or human-curated to preserve regional linguistic properties.
|
| 137 |
+
- **CMTEB (Chinese Massive Text Embedding Benchmark):** A specialised subset of MTEB focused on Chinese language tasks, used to evaluate performance in one of the region's most prominent scripts.
|
| 138 |
+
- **MTEB (Massive Text Embedding Benchmark):** The industry-standard global benchmark used to gauge general-purpose English embedding performance across a wide array of tasks.
|
| 139 |
+
|
| 140 |
+
### Factors
|
| 141 |
+
|
| 142 |
+
Evaluation factors are categorised by task type and linguistic diversity to ensure the model's "fertility" and "nuance" are captured accurately:
|
| 143 |
+
|
| 144 |
+
- **Linguistic Coverage:** Evaluation spans across 12 languages, including complex Brahmic scripts (Burmese, Khmer, Lao, Tamil, Thai), Latin-based SEA scripts (Indonesian, Filipino, Malay, Tetum, Vietnamese), English, and Mandarin.
|
| 145 |
+
- **Task Modality:**
|
| 146 |
+
|
| 147 |
+
- **Retrieval/Reranking:** Efficiency in finding relevant documents within a large corpus.
|
| 148 |
+
- **Semantic Textual Similarity (STS):** Precision in sentence-level semantic alignment.
|
| 149 |
+
- **Clustering & Classification:** Ability to group or categorise text based on latent semantic meaning.
|
| 150 |
+
- **Summarisation & Bitext Mining:** High-level semantic matching and cross-lingual alignment.
|
| 151 |
+
- **Architecture Efficiency:** Performance is measured in the context of the ModernBERT architecture and Gemma 3 tokenizer to assess computational efficiency versus embedding quality.
|
| 152 |
+
|
| 153 |
+
#### Metrics
|
| 154 |
+
|
| 155 |
+
To provide a standardised view of performance, we report the following metrics across the benchmark suites:
|
| 156 |
+
|
| 157 |
+
- **Classification:** F1-score.
|
| 158 |
+
- **Multi-label Classification:** F1-score
|
| 159 |
+
- **Pair Classification:** Average Precision (AP).
|
| 160 |
+
- **Semantic Textual Similarity (STS):** Cosine similarity scores.
|
| 161 |
+
- **Clustering:** V-Measure Score.
|
| 162 |
+
- **Bitext Mining:** F1-score.
|
| 163 |
+
- **Retrieval & Reranking:** NDCG@10 (Primary) and MAP.
|
| 164 |
+
- **Instruction Retrieval:** NNDCG@5.
|
| 165 |
+
|
| 166 |
+
## Results
|
| 167 |
+
|
| 168 |
+
[Attachment: image-20260313-064345.png]Performance comparison of embedding models on SEA-BED (<https://leaderboard.sea-lion.ai/embedding/SEA>). Captured on 13/03/2026 02:50pm.
|
| 169 |
+
|
| 170 |
+
## Environmental Impact
|
| 171 |
+
|
| 172 |
+
Carbon emission was estimated using the fact sheet from TRG [Datacenters](https://www.trgdatacenters.com/resource/h200-power-consumption/).
|
| 173 |
+
|
| 174 |
+
- **Hardware Type:** Nvidia H200 140GB GPUs
|
| 175 |
+
- **Hours used:** 384 hrs
|
| 176 |
+
- **Cloud Provider:** SMC H200
|
| 177 |
+
- **Compute Region:** Singapore
|
| 178 |
+
- **Carbon Emitted:** appx. 108.05 kg CO2 e
|
| 179 |
+
|
| 180 |
+
## Technical Specifications
|
| 181 |
+
|
| 182 |
+
### Model Architecture and Objective
|
| 183 |
+
|
| 184 |
+
SEA-LION-ModernBERT-Embedding is an embedding model using the ModernBERT architecture.
|
| 185 |
+
|
| 186 |
+
| Parameter | SEA-LION-ModernBERT-Embedding |
|
| 187 |
+
| --- | --- |
|
| 188 |
+
| Layers | 22 in 300M / 28 in 600M |
|
| 189 |
+
| d_model | 768 in 300M / 1024 in 600M |
|
| 190 |
+
| head_dim | 12 in 300M / 16 in 600M |
|
| 191 |
+
| Vocabulary | 262144 |
|
| 192 |
+
| Sequence Length | 8k |
|
| 193 |
+
|
| 194 |
+
### Compute Infrastructure
|
| 195 |
+
|
| 196 |
+
#### Hardware
|
| 197 |
+
|
| 198 |
+
- **Hardware Type:** Nvidia H200 140GB GPUs
|
| 199 |
+
- **Cloud Provider:** SMC H200
|
| 200 |
+
|
| 201 |
+
#### Software
|
| 202 |
+
|
| 203 |
+
SEA-LION-ModernBERT-Embedding was trained using Sentence-Transformers.
|
| 204 |
+
|
| 205 |
+
## Glossary
|
| 206 |
+
|
| 207 |
+
- **SEA-BED:** Southeast Asia Embedding Benchmark – a comprehensive evaluation suite for embedding models on SEA languages.
|
| 208 |
+
- **Asymmetric Retrieval:** Retrieval tasks where query and document formulations differ.
|
| 209 |
+
- **Mean Pooling:** Aggregating token embeddings by averaging (weighted by attention mask) to produce a fixed-size sentence representation.
|
| 210 |
+
|
| 211 |
+
## More Information
|
| 212 |
+
|
| 213 |
+
While this model supports masked language modeling, it is primarily optimised via contrastive fine-tuning for downstream tasks such as sequence classification, token classification, or question answering. Please note that these weights have not been specifically aligned for safety; therefore, developers should implement their own safety evaluations and security measures. The authors disclaim all liability for any claims, damages, or other liabilities arising from the use of the released code or weights.
|
| 214 |
+
|
| 215 |
+
For more info, please contact us at [sealion@aisingapore.org](mailto:sealion@aisingapore.org)
|
| 216 |
+
|
| 217 |
+
## Team
|
| 218 |
+
|
| 219 |
+
Ahmed Dabeer, Ahn Jeongmi, Antonyrex Sajeban, Chan Hok Teng Adwin, Cheng Zi Yi Nicholas, Choa Hsueh Mei Esther, Heng Jonathan, Huang Yuli, Jann Railey Estrada Montalan, Lee Chwan Ren, Leong Wai Yi, Leong Wei Qi, Liew Rachel, **Limkonchotiwat Peerat**, Muhammad Ridzuan Bin Mokhtar, Nagarajan Karthik, **Ng Boon Cheong Raymond**, Ngee Chia Tai, Ngui Jian Gang, Nguyen Thanh Ngan, Ong Tat-Wee David, Ong Zhi Hao, Pereira Mark, Poon Joseph, Rengarajan Hamsawardhini, Siow Wei Kang Bryan, Susanto Yosephine, Sutaveephamochanon Anocha, Tan Choon Meng, Tan Chor Phin Evelyn, Tan Siao Wei Jessica, Tan Yixian, Tee Jun Yun, Teng Kok Wai Walter, Teo Eng Sipp Leslie, Tjhi William, Wu Donghang, Yeo Yeow Tong, Yong Xianbin, Zhang Haoyang, Zhang Zhou
|
| 220 |
+
|
| 221 |
+
## **Acknowledgement**
|
| 222 |
+
|
| 223 |
+
This project is supported by the National Research Foundation Singapore and Infocomm Media Development Authority (IMDA), Singapore under its National Large Language Model Funding Initiative.
|
| 224 |
+
|
| 225 |
+
## Contact
|
| 226 |
+
|
| 227 |
+
[sealion@aisingapore.org](mailto:sealion@aisingapore.org)
|