Bahasa Malaysia Evals: Why English Benchmarks Lie
A copilot that scores 90% on English benchmarks can fail half its Malay queries: code-switching, formal versus colloquial forms, Jawi loanwords, and translated policy terms it never saw in training.
The problem in prod
- Mixed Malay and English queries (Manglish) break intent classification trained on clean English.
- Translated banking terms differ from the Malay customers actually type.
- No public benchmark covers Malaysian banking Malay, so vendors report English scores and hope.
Loop engineering
Mine real support chats and branch queries for 300 Malay, 200 English and 100 mixed questions with verified answers. This set is the product spec.
Every model or prompt change runs the mixed-language slice separately. Regressions here block release even when English scores rise.
Native-speaking reviewers grade a weekly sample for tone and correctness. Formal Malay for official letters, natural mix for chat. Wrong register counts as a defect.
Production hardening checklist
- Detect language per query and route to the right prompt template and glossary.
- Lock banking term translations in a glossary the model must cite, not invent.
- Log language distribution in prod to catch demographic drift early.
- Re-recruit reviewer panels yearly so phrasing stays current.
Deploying this in MY
This is the single highest-leverage MY differentiator. Every global vendor demos in English. The team that ships verified Malay quality wins bank RFPs, because procurement teams test in Malay.
Malaysia FDE guide →