lora_model / linkedin_post_draft.md
anocha-mcomp's picture
Update model card from Confluence: Linkedin post draft
959b64d verified
|
Raw History Blame Contribute Delete
1.69 kB
# Linkedin post draft
**The Best Embedding Models for Southeast Asian Use (and Open Too!)**
Excited to share something we've been building for a while β€” open embedding models, purpose-built for Southeast Asia.
Three models, two checkpoints:
- **SEA-LION-E5-Embedding-600M** β€” fine-tuned on multilingual-E5, 512 token context
- **SEA-LION-ModernBERT-300M** β€” ModernBERT + Gemma 3 tokenizer, 8k context
- **SEA-LION-ModernBERT-600M** β€” ModernBERT + Gemma 3 tokenizer, 8k context
Trained on 1–2T tokens from the region. We chose ModernBERT as the base for its strong multilingual architecture, and paired it with the Gemma 3 tokenizer β€” which handles SEA scripts and languages significantly better than most tokenizers out there. Less token fragmentation, better semantic coverage across Thai, Bahasa, Tagalog, Vietnamese and the rest.
On performance: all three sit at the top of SEA-BED β€” the benchmark specifically designed for Southeast Asian language understanding β€” comparable to or beating multilingual-E5-large, BGE-M3, Qwen, and text-embedding-ada-002 depending on the language. The 600M ModernBERT in particular punches hard across the board.
Why care? Embeddings are how AI searches and retrieves β€” foundational to RAG, search, any agentic system. Most models treat SEA languages as a byproduct. Get that wrong and you get a memory system with blind spots, whole chunks of information that just don't surface when they should. [data point]. We built these so SEA is the default, not the afterthought β€” while keeping strong English and Chinese performance too (both very much SEA languages πŸ™‚).
Models: [HuggingFace link] Blog: [link]
[Acknowledgements]