TukaBench: Testing LLM Safety in African Languages
Why safety evaluation of large language models cannot remain English-only, and what we found across seven African languages
Safety evaluation of large language models remains heavily English-centric. The benchmarks that decide whether a model is safe to deploy are written, tested, and judged in English, which leaves low-resource languages, particularly African ones, critically underexplored. If a model refuses a harmful request in English but complies with the same request in Yoruba or Swahili, is it safe? Until now, we have not had a rigorous way to ask that question.
TukaBench, our new benchmark accepted to the main conference of EMNLP 2026, is an attempt to answer it. It covers seven African languages and extends JailbreakBench beyond direct translation through four settings: human translation of the original prompts, English adaptation to African contexts followed by human translation, human-curated prompts validated through model interactions, and code-switched prompts that mix English with African languages. This design lets us isolate the effect of language, cultural grounding, and prompt evasiveness on model safety.
The headline finding is uncomfortable: across both closed and open models, prompting in African languages reduces refusal relative to English, and culturally adapted prompts lead to the least refusal of all. In other words, the safety training that holds up in English weakens precisely where cultural context and lower-resource languages meet.
The evaluation also surfaced two structural problems with how the field measures safety in low-resource languages. First, models sometimes fail to comprehend the prompt at all, which looks like safety but is not. To capture this we introduce Deflection as a category alongside Refused and Jailbroken. Second, the common practice of using an LLM as the judge becomes less reliable in these languages: when we validated outputs with human annotations, judge-human agreement dropped in lower-resource languages and in less commonly supported scripts.
For anyone building or deploying multilingual systems, the takeaway is that safety claims established in English do not automatically transfer. Evaluation has to happen in the languages your users actually speak, with cultural grounding taken seriously and with humans in the loop where automated judges are weakest.
The paper was work with Senyu Li, Wassim Hamidouche, Waqas Zamir, Inbal Becker-Reshef, and David Ifeoluwa Adelani. The preprint is available on arXiv (2606.01322), and you can find it alongside my other publications on the research page.