anocha-mcomp commited on
Commit
a00cd84
·
verified ·
1 Parent(s): 3d5e690

Update model card from Confluence: Working Gemma-SEA-LION-v4-27B-IT Model Card - Draft1

Browse files
working_gemma-sea-lion-v4-27b-it_model_card_-_draft1.md CHANGED
@@ -1,7 +1,215 @@
1
- # Working Gemma-SEA-LION-v4-27B-IT Model Card - Draft1
2
 
3
- **Error converting content:** You may specify either tags to strip or tags to convert, but not both.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
4
 
5
  ```
6
- <h1>Model Card for Gemma-SEA-LION-v4-27B-IT (Instruct model)</h1><p><u>SEA-LION</u> is a collection of Large Language Models (LLMs) which have been pretrained and instruct-tuned for the Southeast Asia (SEA) region.</p><h2>Model Details</h2><h3>Model Description</h3><p>SEA-LION stands for Southeast Asian Languages In One Network. </p><p>We continue pre-train Gemma 3 27B Instruct in English, Chinese, Filipino, Indonesian, Malay, Tamil, Thai, Vietnamese, Burmese, Khmer and Lao using 500B tokens fro...
7
- ```
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Model Card for Gemma-SEA-LION-v4-27B-IT (Instruct model)
2
 
3
+ SEA-LION is a collection of Large Language Models (LLMs) which have been pretrained and instruct-tuned for the Southeast Asia (SEA) region.
4
+
5
+ ## Model Details
6
+
7
+ ### Model Description
8
+
9
+ SEA-LION stands for Southeast Asian Languages In One Network.
10
+
11
+ We continue pre-train Gemma 3 27B Instruct in English, Chinese, Filipino, Indonesian, Malay, Tamil, Thai, Vietnamese, Burmese, Khmer and Lao using 500B tokens from mix sources of web-data, code, opensource dataset and synthetically generated dataset, to create Gemma-SEA-LION-v4-27B-IT.
12
+
13
+ **(Add more Post-training part from Tai or Wei Yi)**
14
+ We conduct post-training using a variety of Reinforcement Learning algorithm. The post-training data for instruction fine-tuning consists of our own SEA-Instruct, Infinity-Instruct, OpenMath-Instruct 2, and other datasets. The post-training process includes multiple stages of instruction fine-tuning, an alignment stage, and several merge stages (need Tai).
15
+
16
+ For tokenization, the model employs the default tokenizer used in Gemma 3 27B Instruct. ~~The model has a context length of 128k tokens.~~
17
+
18
+ - **Developed by:** Products Pillar, AI Singapore
19
+ - **Funded by:** Singapore NRF
20
+ - **Shared by:** Products Pillar, AI Singapore
21
+ - **Model type:** Decoder
22
+ - **Context Length:** 128k tokens
23
+ - **Language(s) (NLP):** Burmese, Chinese, English, Filipino, Indonesian, Malay, Tamil, Thai, Vietnamese, Khmer and Lao,
24
+ - **License:** [Gemma Terms of Use](https://ai.google.dev/gemma/terms)
25
+ - **Finetuned from model [optional]:** [Gemma-3-27B-IT](https://huggingface.co/google/gemma-3-27b-it)
26
+
27
+ ### Model Sources [optional]
28
+
29
+ - **Repository:** {{ repo | default("[More Information Needed]", true)}}
30
+ - **Paper [optional]:** {{ paper | default("[More Information Needed]", true)}}
31
+ - **Demo [optional]:** {{ demo | default("[More Information Needed]", true)}}
32
+
33
+ ## Uses
34
+
35
+ ### Direct Use
36
+
37
+ {{ direct_use | default("[More Information Needed]", true)}}
38
+
39
+ ### Downstream Use [optional]
40
+
41
+ {{ downstream_use | default("[More Information Needed]", true)}}
42
+
43
+ ### Out-of-Scope Use
44
+
45
+ The model has *not* been aligned for safety. Developers and users should perform their own safety fine-tuning and related security measures. In no event shall the authors be held liable for any claims, damages, or other liabilities arising from the use of the released weights and codes.
46
+
47
+ ## Bias, Risks, and Limitations
48
+
49
+ ### Caveats || Risks
50
+
51
+ It is important for users to be aware that our model exhibits certain limitations that warrant consideration. Like many LLMs, the model can hallucinate and occasionally generates irrelevant content, introducing fictional elements that are not grounded in the provided context. Users should also exercise caution in interpreting and validating the model's responses due to the potential inconsistencies.
52
+
53
+ ## Limitations
54
+
55
+ In terms of vision capability, Gemma-SEA-LION-v4-27B-IT has been trained and fine-tuned exclusively on the text back-end. As a result, its vision capabilities are expected to be comparable to those of Gemma 3 IT 27B, and may not exhibit significant improvements or differences in this area. (<https://huggingface.co/google/gemma-3-27b-it> )
56
+
57
+ ### Safety
58
+
59
+ Current SEA-LION models, including this commercially permissive release, have not been aligned for safety. Developers and users should perform their own safety fine-tuning and related security measures. In no event shall the authors be held liable for any claim, damages, or other liability arising from the use of the released weights and codes.
60
+
61
+ ### Recommendations
62
+
63
+ {{ bias_recommendations | default("Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.", true)}}
64
+
65
+ ## How to Get Started with the Model
66
+
67
+ But not ready to use yet! Use the code below to get started with the model using the 🤗 Transformers library.
68
 
69
  ```
70
+ from transformers import pipeline
71
+ import torch
72
+
73
+ pipe = pipeline(
74
+ "text-generation",
75
+ # model="aisingapore/Gemma-SEA-LION-v4-27B-IT",
76
+ device="cuda",
77
+ torch_dtype=torch.bfloat16
78
+ )
79
+
80
+ messages = [
81
+ {
82
+ "role": "system",
83
+ "content": [{"type": "text", "text": "You are a helpful assistant."}]
84
+ },
85
+ {
86
+ "role": "user",
87
+ "content": [
88
+ {"type": "text", "text": "Write a poem on southeast asian countries in Thai."}
89
+ ]
90
+ }
91
+ ]
92
+
93
+ output = pipe(text=messages, max_new_tokens=200)
94
+ print(output[0]["generated_text"][-1]["content"])
95
+ ```
96
+ Later, additional things like image-text-to-text example
97
+
98
+ ## Training Details
99
+
100
+ ### Training Data
101
+
102
+ The dataset comprises English, Chinese, Tagalog, Indonesian, Malay, Tamil, Thai, Vietnamese, Burmese, Khmer, and Lao languages, collected from a mixture of sources including web data, code, open-source datasets, and synthetically generated datasets, amounting to a total of 500 billion tokens.
103
+
104
+ ### Training Procedure
105
+
106
+ #### Preprocessing [optional]
107
+
108
+ {{ preprocessing | default("[More Information Needed]", true)}}
109
+
110
+ #### Training Hyperparameters
111
+
112
+ - **Training regime:** {{ training_regime | default("[More Information Needed]", true)}}
113
+
114
+ #### Speeds, Sizes, Times [optional]
115
+
116
+ {{ speeds_sizes_times | default("[More Information Needed]", true)}}
117
+
118
+ ## Evaluation
119
+
120
+ ### Testing Data, Factors & Metrics
121
+
122
+ We evaluated Gemma-SEA-LION-v4-27B-IT on both general language capabilities and instruction-following capabilities.
123
+
124
+ #### Testing Data
125
+
126
+ General
127
+
128
+ For the evaluation of general language capabilities, we employed the [SEA-HELM evaluation benchmark](https://arxiv.org/abs/2502.14301) across a variety of tasks. These tasks include Question Answering (QA), Sentiment Analysis (Sentiment), Toxicity Detection (Toxicity), Translation in both directions (Eng>Lang & Lang>Eng), Abstractive Summarisation (Abssum), Causal Reasoning (Causal), Natural Language Inference (NLI), and linguistic diagnostics (LINDSEA).
129
+
130
+ Instruction-following
131
+
132
+ We evaluated the models on instruction-following capabilities with two datasets, SEA-IFEval (based on [IFEval](https://arxiv.org/abs/2311.07911)) and SEA-MTBench (based on [MT-Bench](https://arxiv.org/abs/2306.05685)). The two datasets were originally in English, the linguists and native speakers in the team worked together to filter, localise and translate the datasets into the respective target languages to ensure that the examples remained reasonable, meaningful and natural.
133
+
134
+ Counts for Selected Tasks in SEA-HELM
135
+
136
+ | **Competency** | **Task** | **Count** |
137
+ | NLP Classic - NLG | Translation and Abstractive Summarisation | 11467 |
138
+ | NLP Classic - NLR | Natural Language Inference and Causal Reasoning | 7050 |
139
+ | NLP Classic - NLU | Metaphor Understanding, Question Answering, Paraphrase and Sentiment Analysis | 10810 |
140
+ | LLM-Specific: Instruction Following | Instruction/Format Following (IF-Eval | 630 |
141
+ | LLM-Specific: Chat | Multi-Turn Chat (MT-Bench) | 381 |
142
+
143
+ #### Metrics
144
+
145
+ The evaluation was done **zero-shot** with native prompts on a sample of 100-1000 instances for each dataset.
146
+
147
+ ### Results
148
+
149
+ For details on Gemma-SEA-LION-v4-27B-IT performance, please refer to the SEA-HELM leaderboard, <https://leaderboard.sea-lion.ai/>.
150
+
151
+ #### Summary
152
+
153
+ {{ results_summary | default("", true) }}
154
+
155
+ ## Model Examination [optional]
156
+
157
+ {{ model_examination | default("[More Information Needed]", true)}}
158
+
159
+ ## Environmental Impact
160
+
161
+ Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://www.google.com/url?q=https%3A%2F%2Fmlco2.github.io%2Fimpact%23compute) presented in [Lacoste et al. (2019)](https://www.google.com/url?q=https%3A%2F%2Farxiv.org%2Fabs%2F1910.09700).
162
+
163
+ - **Hardware Type:** H256 from 32 nodes
164
+ - **Hours used:** {{ hours_used | default("[More Information Needed]", true)}}
165
+ - **Cloud Provider:** {{ cloud_provider | default("[More Information Needed]", true)}}
166
+ - **Compute Region:** {{ cloud_region | default("[More Information Needed]", true)}}
167
+ - **Carbon Emitted:** {{ co2_emitted | default("[More Information Needed]", true)}}
168
+
169
+ ## Technical Specifications [optional]
170
+
171
+ ### Model Architecture and Objective
172
+
173
+ H256 from 32 nodes
174
+
175
+ ### Compute Infrastructure
176
+
177
+ {{ compute_infrastructure | default("[More Information Needed]", true)}}
178
+
179
+ #### Hardware
180
+
181
+ {{ hardware_requirements | default("[More Information Needed]", true)}}
182
+
183
+ #### Software
184
+
185
+ {{ software | default("[More Information Needed]", true)}}
186
+
187
+ ## ~~Citation [optional]~~
188
+
189
+ **~~BibTeX:~~**
190
+
191
+ ~~{{ citation_bibtex | default("[More Information Needed]", true)}}~~
192
+
193
+ **~~APA:~~**
194
+
195
+ ~~{{ citation_apa | default("[More Information Needed]", true)}}~~
196
+
197
+ ## ~~Glossary [optional]~~
198
+
199
+ ~~{{ glossary | default("[More Information Needed]", true)}}~~
200
+
201
+ ## More Information [optional]
202
+
203
+ This is the repository for the commercial instruction-tuned model. The model has *not* been aligned for safety. Developers and users should perform their own safety fine-tuning and related security measures. In no event shall the authors be held liable for any claims, damages, or other liabilities arising from the use of the released weights and codes.
204
+
205
+ For more info, please contact us using this [SEA-LION Inquiry Form](https://forms.gle/sLCUVb95wmGf43hi6)
206
+
207
+ [Link to SEA-LION's GitHub repository](https://github.com/aisingapore/sealion)
208
+
209
+ ## Model Card Authors [optional]
210
+
211
+ Adithya Venkatadri Hulagadri, Adwin Chan Hok Teng, Anocha Sutaveephamochanon, Brandon Ong Jin Jie, Bryan Siow Wei Kang, David Ong Tat-Wee, Esther Choa Hsueh Mei, Evelyn Tan Chor Phin, Hamsawardhini Rengarajan, Huang Yuli, Jann Railey Estrada Montalan, Jessica Tan Siao Wei, Jonathan Heng, Karthik Nagarajan, Lee Chwan Ren, Leong Wai Yi, Leong Wei Qi, Leslie Teo, Mark Pereira, Muhammad Ridzuan Bin Mokhtar, Ngee Chia Tai, Ngui Jian Gang, Nguyen Thanh Ngan, Nicholas Cheng Zi Yi, Ong Zhi Hao, Peerat Limkonchotiwat, Raymond Ng Boon Cheong, Sajeban Antonyrex, Susanto Yosephine, Tan Choon Meng, Walter Teng Kok Wai, Wayne Lau, William Tjhi Chandra, Yeo Yeow Tong, Yong Xianbin, Liew Rachel, Liu Bing Jie Darius, Teo Wei Yi
212
+
213
+ ## Model Card Contact
214
+
215
+ {{ model_card_contact | default("[More Information Needed]", true)}}