PolyAEGIS: Benchmarking Large Language Model Safety Across 100 Languages

Authors: Md Jahangir Alam¹, Tanzim Ahad¹, Ismail Hossain², Sai Puppala³, Syed Bahauddin Alam⁴, Sajedul Talukder²

¹ University of Texas at El Paso  ·  ² University of Texas at El Paso, Department of Computer Science  ·  ³ New Mexico Institute of Mining & Technology  ·  ⁴ University of Illinois at Urbana-Champaign

Partial leakage — a response that attempts to refuse yet still discloses actionable harmful content — accounts for nearly half of all unsafe outcomes, and binary attack-success-rate reporting makes it invisible.

Abstract

We present PolyAEGIS, a massively multilingual LLM safety benchmark spanning 100 languages across 17 language families and 26 scripts, built on 10,362 Gold-quality translated prompts drawn from two successive AEGIS-derived sourcing pipelines (2,894 and 11,788 unique unsafe root prompts respectively) and gated through a dual-metric translation-quality filter (BERTScore F1 and COMETKiwi). Each translated prompt is sent to six generation models — three open-weight (Aya-Expanse-32B, Mistral-Large-123B, GPT-OSS-120B) and three closed-source (Claude Haiku 4.5, Gemini 2.5 Flash, GPT-5.4 Mini) — and every response is scored by three independent LLM judges (Mistral-Large, Qwen2.5, and Claude Sonnet 4.6) under a four-class response taxonomy — Full Refusal, Partial Leakage, Full Compliance, and Irrelevant/Unintelligible — whose per-judge verdicts we reconcile into a majority-vote consensus label, yielding 61,536 consensus-scored responses.

At a Glance

Method

1. Sourcing. Two successive AEGIS-derived pipelines yield 2,894 (v1, null-filtered) and 11,788 (v2, hierarchical structural/redaction/label-conflict filtering) unique unsafe root prompts; only v2’s new-only prompts are translated to avoid duplicate work.
2. Translation and quality gating. Prompts are translated with NLLB-200 and back-translated, then scored with BERTScore F1 and COMETKiwi. Only samples clearing all three Gold thresholds (F1 ≥ 0.95, COMET ≥ 0.92, combined ≥ 0.94) enter the benchmark — 2.31% of the v1 pool.
3. Leakage-aware taxonomy. Responses are labeled Full Refusal, Partial Leakage, Full Compliance, or Irrelevant/Unintelligible, giving two ASR variants: ASR_full (IU treated as safe) and ASR_interp (conditioned on comprehension).
4. Multi-judge reconciliation. Because the v1 and v2 judge outputs did not share a schema, a four-step backtracking reconciliation (root-identity normalization, metadata backfill, quality-bucket recomputation, language-metadata backfill) restores all five aggregation fields to 100% coverage — without it, one judge’s votes were recoverable for only 29.6% of samples.

Main Findings

• Partial leakage is roughly half the problem. Full refusal dominates at 77.02%, but unsafe outcomes total 13.78% (full compliance 7.45%, partial leakage 6.32%), with ASR_interp at 15.17%. Leakage carries a mean severity of 0.393, and leakage is the dominant failure mode in 45 of 100 languages.
• A 3.96× spread across languages. Mean harm runs from 0.059 (Ukrainian) to 0.234 (Burmese) on identical prompts. Several languages reach the maximum observed harm score of 1.0 despite moderate means.
• Low-resource languages are less safe on every metric — IU rate 9.53% vs 8.02%, ASR_full 14.18% vs 12.37%, mean harm 0.1353 vs 0.1043 — so this is not merely an artifact of comprehension failure masking unsafe behavior.
• Script matters independently of resource level. Right-to-left scripts show higher harm (0.1528 vs 0.1269) and ASR_full (16.21% vs 13.63%) than left-to-right, despite including several high-resource languages.
• Open-weight models are unsafe 1.6× as often as closed-source (16.90% vs 10.65% ASR_full, a 6.25-point gap that holds within both resource tiers) — but not uniformly: GPT-OSS-120B (7.16%) is safer than two of the three closed models.
• Measurement is itself unreliable. Only 36.8% of prompts receive a single consistent label across all six generation models, and three-judge parseable verdicts cover 97.06% of responses — with a ten-fold higher fallback rate for one generation model (Mistral-Large-123B, 10.5%) than any other.

Limitations

• Translation relies on a single system (NLLB-200), and the counter-intuitive BERTScore F1 advantage observed for low-resource languages is itself evidence that round-trip quality metrics are an imperfect proxy for fidelity.
• Per-language counts range from 46 to 734 by construction, since the benchmark is assembled by quality gating rather than stratified sampling.
• Evaluation depends on LLM judges, which are a disagreement-prone instrument; three-judge consensus mitigates but does not eliminate this.
• Aggregate and per-language metrics are reported without formal significance testing.
• Six models do not cover the space of deployed multilingual LLMs, and the dataset contains harmful prompts that carry misuse risk out of context.

BibTeX

@misc{alam2026polyaegis,
  title  = {PolyAEGIS: Benchmarking Large Language Model Safety Across 100 Languages},
  author = {Alam, Md Jahangir and Ahad, Tanzim and Hossain, Ismail and Puppala, Sai and Alam, Syed Bahauddin and Talukder, Sajedul},
  year   = {2026},
  note   = {Under review at ICLR 2027}
}