anocha-mcomp commited on
Commit
e12b260
·
verified ·
1 Parent(s): 32c008e

Update model card from Confluence: Model Card: Gemma-SEA-LION-v4-4B-VL

Browse files
Files changed (1) hide show
  1. model_card_gemma-sea-lion-v4-4b-vl.md +20 -36
model_card_gemma-sea-lion-v4-4b-vl.md CHANGED
@@ -6,11 +6,10 @@ SEA-LION is a collection of Large Language Models (LLMs) which have been pretrai
6
 
7
  **Gemma-SEA-LION-v4-4B-VL** is a 4-billion parameter Vision-Language Model (VLM) built upon the gemma-3-4b-it architecture. To ensure **domain adaptation** for the region, the model underwent rigorous post-training on a curated dataset of approximately **8.54 million** instruction-text pairs. This curated dataset included various SEA Languages such as Burmese, Malay, Tagalog and Tamil. This extensive post-training instills **multilingual** and **multicultural** fluency for both SEA languages and English.
8
 
9
- Gemma-SEA-LION-v4-4B-VL inherits the following features from gemma-3-4b-it:
10
 
11
- - Image and text understanding capabilities, including document comprehension, visual Q&A, and image-grounded reasoning,
12
- - Advanced function calling and structured outputs to allow for seamless integration into larger systems, and
13
- - Large 128K context length
14
 
15
  ## Model Details
16
 
@@ -100,19 +99,10 @@ The instruction fine-tuning text dataset comprises of a collection of OSS & synt
100
  - <https://huggingface.co/datasets/allenai/Dolci-Instruct-SFT-Tool-Use>
101
  - <https://huggingface.co/datasets/Agent-Ark/Toucan-1.5M>
102
 
103
- **Datasets for OCR:**
104
-
105
- - <https://huggingface.co/datasets/Goekdeniz-Guelmez/unsloth-LaTeX_OCR-for-mlx-vlm> (68,700)
106
- - <https://huggingface.co/datasets/sushanthr/fineweb_ocr/viewer/default/train?row=0> (150,000)
107
- - <https://huggingface.co/datasets/sushanthr/wiki_ocr> (50,000)
108
- - <https://huggingface.co/datasets/Chanrith123333/khmer_english_ocr_image_line> (11.3 million)
109
- - <https://huggingface.co/datasets/chuuhtetnaing/burmese_ocr_data_hf> (9070)
110
- - <https://huggingface.co/datasets/SoyVitou/KhmerST-Dataset-OCR-Cropped> (3560)
111
-
112
  **Datasets for Reinforcement Learning:**
113
 
114
- - Instruction following: <https://huggingface.co/datasets/nvidia/Nemotron-Cascade-RL-Instruction-Following>
115
- - Tool: <https://huggingface.co/datasets/openai/gsm8k>
116
 
117
  ### Training Procedure
118
 
@@ -179,36 +169,32 @@ For details on Gemma-SEA-LION-v4-4B-VL performance, please refer to the SEA-HELM
179
 
180
  ### Tool calling
181
 
182
- We evaluated the tool calling capabilities of our model
183
-
184
- #### Results
185
-
186
- ### OCR
187
 
188
- #### Results
189
 
190
- ### Retaining VL Capabilities
191
 
192
- We also evaluated the model on two types of tasks using datasets specifically focused on Southeast Asian examples to benchmark against the original base models (Gemma-4b-it).
193
 
194
- - **Visual Question Answering (VQA):** We utilized Multiple Choice Question (MCQ) style tasks, including MARVL, CVQA, and WorldCuisines.
195
- - **Image Captioning:** We employed the XM3600 dataset, evaluating strictly on examples relevant to the SEA region.
196
- - ~~OCR~~
197
 
198
- Key Insight: Despite our fine-tuning process focusing primarily on text data (approximately 8 million regional Q&A and instruction pairs), our evaluation confirms that Gemma-SEA-LION-v4-4B-VL successfully retains the high-performance vision-language capabilities of the original base models.
199
 
200
- #### Factors
201
 
202
- The evaluation was done **zero-shot** with native prompts.
203
 
204
- #### Metrics
 
 
 
205
 
206
- The following metrics were used to measure performance:
207
 
208
- - **Normalized accuracy** was the primary metric for the VQA tasks (CVQA, MARVL, and WorldCuisines).
209
- - **RefCLIP Score** was used for the XM3600 image captioning task.
210
 
211
- ### Results
212
 
213
  ## More Information
214
 
@@ -220,8 +206,6 @@ For more info, please contact us at [sealion@aisingapore.org](mailto:sealion@ais
220
 
221
  Ahmed Dabeer, Ahn Jeongmi, Antonyrex Sajeban, Chan Hok Teng Adwin, Cheng Zi Yi Nicholas, Choa Hsueh Mei Esther, Heng Jonathan, Huang Yuli, Jann Railey Estrada Montalan, Kang Siow Wei Bryan, Lee Chwan Ren, Leong Wai Yi, Leong Wei Qi, Liew Rachel, Limkonchotiwat Peerat, Muhammad Ridzuan Bin Mokhtar, Nagarajan Karthik, Ng Boon Cheong Raymond, Ngee Chia Tai, Ngui Jian Gang, Nguyen Thanh Ngan, Ong Tat-Wee David, Ong Zhi Hao, Pereira Mark, Poon Joseph, Rengarajan Hamsawardhini, Susanto Yosephine, Sutaveephamochanon Anocha, Tan Choon Meng, Tan Chor Phin Evelyn, Tan Siao Wei Jessica, Tan Yixian, Tee Jun Yun, Teng Kok Wai Walter, Teo Eng Sipp Leslie, Tjhi William, Yeo Yeow Tong, Yong Xianbin, Zhang Zhou, Elliott Chris (Google), Mohseni Mohammadreza (Google), Sharan Mayank (Google), Wei Fanny (Google), Tang Jiuqiang (Google), Xu Xiang (Google), Yu Ting (Google), Loh Michelle (Google), Mangal Saurabh (Google), Mukherjee Pratyusha (Google), Sim Stephanie (Google)
222
 
223
- ~~Gemma team??? Elliott Chris (Google), Mohseni Mohammadreza (Google), Sharan Mayank (Google), Wei Fanny (Google), Tang Jiuqiang (Google), Xu Xiang (Google), Yu Ting (Google), Loh Michelle (Google), Mangal Saurabh (Google), Mukherjee Pratyusha (Google), Sim Stephanie (Google)~~
224
-
225
  ## Acknowledgement
226
 
227
  This project is supported by the National Research Foundation Singapore and Infocomm Media Development Authority (IMDA), Singapore under its National Large Language Model Funding Initiative.
 
6
 
7
  **Gemma-SEA-LION-v4-4B-VL** is a 4-billion parameter Vision-Language Model (VLM) built upon the gemma-3-4b-it architecture. To ensure **domain adaptation** for the region, the model underwent rigorous post-training on a curated dataset of approximately **8.54 million** instruction-text pairs. This curated dataset included various SEA Languages such as Burmese, Malay, Tagalog and Tamil. This extensive post-training instills **multilingual** and **multicultural** fluency for both SEA languages and English.
8
 
9
+ Gemma-SEA-LION-v4-4B-VL inherits the image and text capabilities from gemma-3-4b-it alongside its large context length of 128K tokens. Additionally, beyond extending the multilingual capabilities of the original gemma model for SEA languages, we experimented with:
10
 
11
+ 1. Adding function calling to the model to allow for this model to be used in tool calling applications
12
+ 2. The OCR capabilities in Thai, Chinese and English
 
13
 
14
  ## Model Details
15
 
 
99
  - <https://huggingface.co/datasets/allenai/Dolci-Instruct-SFT-Tool-Use>
100
  - <https://huggingface.co/datasets/Agent-Ark/Toucan-1.5M>
101
 
 
 
 
 
 
 
 
 
 
102
  **Datasets for Reinforcement Learning:**
103
 
104
+ - <https://huggingface.co/datasets/nvidia/Nemotron-Cascade-RL-Instruction-Following>
105
+ - <https://huggingface.co/datasets/openai/gsm8k>
106
 
107
  ### Training Procedure
108
 
 
169
 
170
  ### Tool calling
171
 
172
+ We evaluated the tool calling capabilities of our model using the Berkeley Function Calling Leaderboard (BFCL) V4 evaluation.
 
 
 
 
173
 
174
+ #### Factors
175
 
176
+ The evaluation was done using the codes from the BFCL v4 repository. Modifications were made to the agentic web search task:
177
 
178
+ - The Brave Search API was used instead of the DuckDuckGo Search API. The Brave Search API is also a privacy focused search engine and is similar to the DuckDuckGo Search API.
179
 
180
+ **Note:** Gemma-3-4b-it was run in prompt mode as it does not support function calling.
 
 
181
 
182
+ #### Results
183
 
184
+ ### OCR
185
 
186
+ We also evaluated the OCR capabilities using the components of the OCRBenchv2 and ThaiOCRBench that focused primarily on text extraction. These included the following tasks:
187
 
188
+ - Table/Chart/Document parsing
189
+ - Fine grained OCR
190
+ - Full page OCR
191
+ - Text recognition
192
 
193
+ #### Factors
194
 
195
+ The evaluation was done zero-shot using native prompts.
 
196
 
197
+ #### Results
198
 
199
  ## More Information
200
 
 
206
 
207
  Ahmed Dabeer, Ahn Jeongmi, Antonyrex Sajeban, Chan Hok Teng Adwin, Cheng Zi Yi Nicholas, Choa Hsueh Mei Esther, Heng Jonathan, Huang Yuli, Jann Railey Estrada Montalan, Kang Siow Wei Bryan, Lee Chwan Ren, Leong Wai Yi, Leong Wei Qi, Liew Rachel, Limkonchotiwat Peerat, Muhammad Ridzuan Bin Mokhtar, Nagarajan Karthik, Ng Boon Cheong Raymond, Ngee Chia Tai, Ngui Jian Gang, Nguyen Thanh Ngan, Ong Tat-Wee David, Ong Zhi Hao, Pereira Mark, Poon Joseph, Rengarajan Hamsawardhini, Susanto Yosephine, Sutaveephamochanon Anocha, Tan Choon Meng, Tan Chor Phin Evelyn, Tan Siao Wei Jessica, Tan Yixian, Tee Jun Yun, Teng Kok Wai Walter, Teo Eng Sipp Leslie, Tjhi William, Yeo Yeow Tong, Yong Xianbin, Zhang Zhou, Elliott Chris (Google), Mohseni Mohammadreza (Google), Sharan Mayank (Google), Wei Fanny (Google), Tang Jiuqiang (Google), Xu Xiang (Google), Yu Ting (Google), Loh Michelle (Google), Mangal Saurabh (Google), Mukherjee Pratyusha (Google), Sim Stephanie (Google)
208
 
 
 
209
  ## Acknowledgement
210
 
211
  This project is supported by the National Research Foundation Singapore and Infocomm Media Development Authority (IMDA), Singapore under its National Large Language Model Funding Initiative.