SEA-LION v3: 128K Context Length and 70B Models

We are excited to announce the release of two new variants for SEA-LION v3, our latest large language models tailored specifically for Southeast Asian languages. Building upon Meta’s Llama and SEA-LION’s data, these variants have strong capabilities in handling diverse linguistic and cultural nuances inherent to Southeast Asian region.

New SEA-LION v3 Variants

1. SEA-LION v3 8B

2. SEA-LION v3 70B

Technical Enhancements in SEA-LION v3

Continued Pre-Training

Both variants underwent continued pre-training on Llama 3.1 using an additional 200 billion tokens of Southeast Asian data. This extensive training enhances the models’ understanding of regional languages and cultural contexts, resulting in significant performance boosts in languages such as Thai, Vietnamese, Tamil, and Indonesian.

Post-Training

Both variants undergo supervised fine-tuning (SFT) in two stages:

1. Stage 1: Focuses on math and reasoning instructions using approximately 9.5 million instructions, predominantly in English.
2. Stage 2: Emphasizes chat and instruction-following tasks with around 7.3 million instructions, including a substantial portion in Southeast Asian languages.

This fine-tuning process, combined with model merging techniques, ensures that SEA-LION v3 maintains its superior performance while mitigating issues like catastrophic forgetting.

Multilingual Proficiency

SEA-LION v3 supports up to 13 languages, including newly added languages like Javanese and Sundanese. This multilingual capability ensures that our models can cater to a wide array of Southeast Asian languages, fostering greater accessibility and usability across the region. In addition, our experiments show some modest cross-lingual transfer which help languages that are not as well representated in digital data.

Training Infrastructure

• Hardware: Utilized MosaicML Composer on AWS p5e.48xlarge and SingTel HGX-100 instances equipped with Nvidia H200 and H100 GPUs.
• Training Duration: The 8B variant was trained for approximately 136 hours, while the 70B variant underwent 495 hours.
• Configuration: Both models employ bfloat16 precision, decoupled_adamw optimizer, and a global batch size of 512.

Evaluation Metrics

SEA-LION v3 variants have been rigorously evaluated using both English and Southeast Asian benchmarks:

• English Evaluation: Utilizes tasks from the Open LLM Leaderboard v2, including MMLU-PRO, MUSR, and others.
• Southeast Asian Evaluation: Employs SEA-HELM metrics covering sentiment analysis, toxicity detection, causal reasoning, and more, tailored to regional languages.

Based on these narrow benchmarks, SEA-LION v3 outperforms many open source and even larger models like Llama 3.3 70B Instruct in several benchmarks, establishing new standards for regional AI capabilities. See our Leaderboard for details.

Accessibility and Availability

All SEA-LION v3 variants are open-source and freely available for research and commercial use. Developers and enterprises can immediately access the models on platforms such as Hugging Face, Kaggle, and Ollama.

Hugging Face

Gemma-SEA-LION-v3-9B
Gemma-SEA-LION-v3-9B-IT
Gemma-SEA-LION-v3-9B-IT-GGUF

Llama-SEA-LION-v3-8B
Llama-SEA-LION-v3-8B-IT
Llama-SEA-LION-v3-8B-IT-GGUF

Llama-SEA-LION-v3-70B
Llama-SEA-LION-v3-70B-IT
Llama-SEA-LION-v3-70B-IT-GGUF

Ollama

Gemma-SEA-LION-v3-9B-IT
Llama-SEA-LION-v3-8B-IT
Llama-SEA-LION-v3-70B-IT

Kaggle

Gemma-SEA-LION-v3-9B
Gemma-SEA-LION-v3-9B-IT

Llama-SEA-LION-v3-8B
Llama-SEA-LION-v3-8B-IT

Partner Models

We are also happy to share a few models built by our partners and collaborators in the region:

Acknowledgments

We extend our gratitude to our partners and collaborators across Southeast Asia.

We are also grateful for the support of the Infocomm Media Development Authority (IMDA) of Singapore.