SEA-LION v3: Advancing Southeast Asian AI in An Open, Inclusive, and Responsible Manner
The latest evolution in Southeast Asian AI, built with assistance from regional collaborators, and global partners such as Google and Nvidia.
Introducing SEA-LION v3
Continued Pre-training
The latest in our family of open-source models, SEA-LION v3, was built by continued pre-training of Gemma2 9B on 200 billion high quality Southeast Asian tokens from 11 official Southeast Asian languages.
Custom Post-training
Further custom post-training combining instruction tuning, model merging and basic alignment; and leveraging the ongoing high-quality regional fine-tuning data collected under Project SEALD, helped SEA-LION v3 achieve an overall State-of-the-Art performance of SEA-LION v3 on Southeast Asia-oriented benchmark.
Strategic Partnerships
We benefited from Project SEALD, a strategic partnership initiated by AI Singapore and Google. Google and our Project SEALD partners supported us in post-training data collection, technical advisory and compute resources.
Powerful Infrastructure
SEA-LION v3 was pre-trained using 64+8 H100 GPUs on 8+1 instances of SingTel HGX-100, over a span of 10 days.
Lessons from Our Model Building Experience
1. Leveraging Open LLMs
This year, we learned to take advantage of the strong performance of open LLMs like Gemma and Llama.
2. Fine-tuning Challenges
Our experiments showed that directly fine-tuning these LLMs was helpful but insufficient to close some gaps, especially for the more underrepresented languages in our region.
3. Continued Pre-training
Continued pre-training was our method of choice, but our challenge there was to minimize forgetting. We learned ways to overcome this challenge when we built our Llama-based SEA-LION v2.
4. Post-training Discoveries
In the post-training phase, besides instruction tuning and alignment, we discovered that model merging was also valuable to improve model performance.
The Importance of NVIDIA and Our Cloud Partners
Critical Support from Partners
None of our accomplishments is possible without the critical support of NVIDIA and our cloud infrastructure partners, namely GCP, AWS, Singtel and Alicloud.
Evolution of SEA-LION
The following table shows the evolution of SEA-LION from v1 to v3, and the corresponding numbers and types of GPUs utilized for the pre-training.
| Version | Model Size | Tokens | GPUs | Compute | Duration |
|---|---|---|---|---|---|
| SEA-LION v3 | 9B | 200B | 64+8 H100 | Singtel | 10 days |
| SEA-LION v2 | 8B | 48B | 64 H100 | AWS | 2 days |
| SEA-LION v1 | 7B | 1T | 256 A100 | AWS | 22 days |
Powered by NVIDIA GPUs
Throughout the development of these versions, NVIDIA GPUs on our GCP and Alicloud instances powered our supervised fine-tuning (SFT) and experiments, ensuring robust and efficient model training and validation processes.
Expanding Horizons with NVIDIA
Advancing the Model
As we look ahead, SEA-LION aims to further advance the model. We are committed to extending the range of model sizes, both larger and smaller, and to this end, AI Singapore will deepen its partnership with NVIDIA by adopting Nemo 2.0 for our modeling framework.
Enhancing Data Pipeline
Additionally, we will integrate Nemo Curator to ensure scalable and efficient data pipeline operations, further enhancing the richness and accuracy of our models, as we push the frontier of regional data collection by covering lower-resource languages and incorporating modalities beyond text.
Looking Ahead: The Future of AI in Southeast Asia
Continued Innovation and Collaboration: Together with NVIDIA and our partners, both regional and global, SEA-LION will continue to push the boundaries of what’s possible, ensuring that Southeast Asia remains at the forefront of AI innovation. This partnership represents a commitment to ongoing research, development, and implementation of cutting-edge AI technologies tailored to the unique needs of the region.
Stay tuned for more updates on our progress and exciting developments. The future of AI in Southeast Asia is bright, and we are thrilled to be on this journey with you. As SEA-LION continues to evolve, it promises to unlock new possibilities for innovation, economic growth and societal advancement across the region.