Introducing SEA-HELM Audio: Measuring the Speech Gap in Southeast Asian Voice Agents

ByAISG | AI Products

When you speak to an AI agent instead of typing, how much intelligence is actually lost along the way?

While Large Language Models (LLMs) continue to post impressive scores on text benchmarks, real-world deployment is increasingly shifting towards voice. No matter how speech reaches the model, the task it solves is the one it heard, not the one the user spoke. Every mis-heard accent, dropped word or mangled instruction changes the task the model is given, and a text benchmark never sees it.

For Southeast Asia (SEA), this blind spot is only now being addressed. Established suites such as VoiceBench are almost entirely English, and the current regional speech benchmarks measure how well audio models hear: transcription accuracy, translation, speaker and emotion recognition. What they do not measure is the question a voice-agent builder actually has: after transcription, does the agent still do what the user asked, and how much worse than if they had typed?

Today we are glad to introduce SEA-HELM Audio, together with three open resources built on it: the SEA-HELM Audio benchmark corpus on Hugging Face, the SEA Audio Leaderboard, and the SEA Voice-Agent Playground. Together they let anyone measure, compare and experience the speech gap in SEA voice pipelines. The full methodology is in our technical report on arXiv.

What is SEA-HELM Audio?

SEA-HELM Audio extends SEA-HELM, AI Singapore’s Southeast Asian Holistic Evaluation of Language Models adapted from Stanford’s HELM, from text into speech. SEA-HELM already provides regionally grounded prompts and rule-based verifiers, but it is natively text-only. We synthesised its prompts into speech and ran the same tasks through full ASR-LLM pipelines, scoring the audio branch and the text branch with identical verifiers.

The headline metric is the speech gap: the difference between text-input accuracy and speech-input accuracy for the same model on the same prompts. A large gap means the pipeline understands the task but cannot hear it.

The first public release of the corpus covers:

  • Five languages: Chinese (zh), English (en), Indonesian (id), Malay (ms) and Tamil (ta).
  • Two task subsets with deterministic rule-based scoring:
    • seahelm_ifeval (instruction following with strict, checkable constraints such as word counts and formatting), and
    • seahelm_global_mmlu_lite (multiple-choice reasoningNeither needs an LLM-as-a-judge, so results are reproducible and cheap to run.
A table displaying configuration names and their corresponding data file paths for language splits including English, Indonesian, Tamil, and Chinese.
  • The “Modality Gap” Metric: every item carries both the ground-truth text and the synthesised audio, so transcription loss can be isolated exactly. The scorer also reports Word Error Rate (WER) and Character Error Rate (CER), separating how much the ASR changed the prompt from how much that change hurt the task.
Diagram illustrating the SEA-HELM process, depicting two paths: the speech path with components including TTS, ASR, LLM, and a verifier, alongside a baseline text path.
  • A unified acoustic baseline: all prompts are synthesised with a consistent voice chosen independently of the systems under test, so every pipeline hears the same input.

This consistency is also the first version’s main limitation. Synthetic speech does not capture real regional accents, background noise or code-switching, so the gaps reported here are likely a floor rather than a ceiling. Closing that distance is where we want the community’s help (see below).

The SEA Audio Leaderboard: a first version

The SEA Audio Leaderboard evaluates end-to-end conversational audio stacks, pairing ASR models with downstream LLMs across SEA languages. Rather than ranking ASR on WER alone, it reports task-level accuracy: after transcribing your speech, can the full pipeline still follow your instructions or answer correctly? Each entry shows text accuracy, speech accuracy and the speech gap between them, filterable by language, task and pipeline component.

This is a deliberately early release. The first board covers five languages, two task subsets and synthetic speech, with a regional open-weight stack (SEA-LION-v4.5-27B behind Qwen3-ASR-1.7B and MERaLiON3-ASR-3B) and a commercial stack (Gemini 3.5 Flash behind ElevenLabs Scribe Realtime). We will extend it to more SEA languages and update it as community submissions come in. Treat today’s numbers as a baseline to beat, not a verdict.

Key Insights from Our First Benchmark Runs

Our initial runs on the leaderboard revealed stark differences in how languages and task structures handle the voice shift:

  • The Tamil Bottleneck: SEA-LION-v4.5-27B follows Tamil instructions well on text but loses more than 40 points once the prompt is spoken (falling 32–33 points in Indonesian and Malay), and the commercial stack loses exactly as much. That the gap is identical across a regionally adapted ASR model and a commercial one points to data scarcity in Tamil acoustic modelling rather than a weak LLM.
  • Chinese Modality Resilience: Chinese multiple-choice tasks proved remarkably resilient to the voice shift, incurring a modality gap of under 11 points across evaluated models.
  • Task format matters as much as language. On multiple choice, the LLM can usually recover the right option from a slightly garbled transcript. On instruction following, a single dropped or substituted word can remove a constraint, so the model answers a different instruction, and the verifier marks it wrong.
  • The LLM is not the weak link. On text, SEA-LION-v4.5-27B was competitive with Gemini 3.5 Flash in SEA languages and beat it on Indonesian instruction following (94.29% vs 89.52%). Through speech, the same model lost 33 points in Indonesian. The limiting factor in the open-weight voice agent stack is the ASR layer’s handling of regional accents and code-switching, not the LLM’s reasoning.

Try It Live: The SEA Voice-Agent Playground

Benchmarks tell half the story. Hearing the latency, listening to the synthesised reply and testing your own accent is where abstract metrics turn into real user experience.

Alongside the leaderboard, we are launching the Voice-Agent Playground, a real-time testing arena where you can directly interact with open and commercial voice agent stacks.

Playground Features:

  • Mix-and-Match Stacks: Swap out ASR engines (e.g., Whisper, SeamlessM4T, regional fine-tunes) and LLM backends (including SEA-LION-v4.5) on the fly to experience performance differences in real time.
  • Live Accent & Dialect Testing: Speak in English, Indonesian, Malay, Tamil, Chinese, Thai, or Vietnamese and hear how well each voice agent stack handle natural regional accents and code-switching.
  • Real-Time Latency & Modality Breakdown: Live telemetry shows Time to First Token (TTFT), ASR transcription speed, LLM generation rate and Text-to-Speech (TTS) synthesis latency as you talk.
Screenshot of a voice agent application interface, displaying listening status and configuration options including model selection, language, voice type, and speech speed settings.

Benchmark Locally with Our Open-Source Evaluation Repo

Along with the dataset and playground, we are releasing the official evaluation repository so teams can benchmark their own custom voice stacks locally or in private cloud environments.

What You Can Do with the Eval Codebase

  • Run the Core Voice Tasks: Evaluates custom ASR and LLM configurations on both seahelm_ifeval and seahelm_global_mmlu_lite.
  • Automated Dual-Branch Execution: Runs both the text baseline branch and the speech-to-text cascade branch in a single pass, automatically reporting overall accuracy and calculating the ASR penalty (Delta).
  • Fast, Local Rule-Based Verifiers: Built-in programmatic checkers execute completely in-process, no external LLM judge API calls required.
  • One-Command Evaluation: Run a benchmark suite across regional languages with a single command:
Python
python eval_speech.py --asr_model whisper-large-v3 --llm_model aisingapore/tron-SEA-LION-v4.8-Instruct --languages id,ms,ta,zh,en

Help us build open voice AI for Southeast Asia

SEA-LION has always been a community effort, and the speech gap will not close from us alone:

  • Real speech data, above all. The results say the bottleneck is acoustic: open-weight ASR models have not heard enough Southeast Asian speech. Recordings with natural accents, code-switching and everyday background noise are what will let the region’s ASR and TTS models improve, and what will let the next version of this corpus measure the real world rather than a synthetic baseline.
  • Native-speaker review. Speakers who can check synthesised prompts and transcripts in Tamil, Malay, Indonesian and the languages we add next.
  • More languages. Extending the corpus and leaderboard beyond the current five is our next priority.
  • Models and evaluation. If you train regional ASR or TTS, or want your voice stack on the leaderboard, the GitHub repo is open for submissions and issues.

If you hold or can collect speech in a Southeast Asian language, that is the contribution that moves the numbers. Write to us at sealion@aisingapore.org and we will work out how to bring it in.

Links


Have questions? Open an issue on our GitHub repo, reach out via our Hugging Face Community or write to sealion@aisingapore.org!