Md Jahangir Alam

Md Jahangir Alam

LLM & Agent Security, Privacy, Applied ML

When Guardrails Fail at Scale: Guardrail Evaluation Across 200 Languages

Published in In preparation, 2026

Authors: Md Jahangir Alam1, Ismail Hossain1, Tanzim Ahad1, Sai Puppala2, Sajedul Talukder1

1 Department of Computer Science, University of Texas at El Paso  ·  2 New Mexico Institute of Mining & Technology

This is an extended version of When Guardrails Fail: Multilingual Long-Context Safety in Tool-Using Agents (NeurIPS 2026, LCFM), scaling the guardrail evaluation from 98 languages to 200.

Overview

When Guardrails Fail showed that a runtime guard’s verdict does not determine deployment safety: after a harmful request is admitted, the downstream agent may still refuse, escalate, or reach for a restricted tool, and which of those happens depends jointly on the guard, the language, the context length, the prompt position, and the agent. It also found that low-resource languages carry roughly twice the downstream risk of high- and medium-resource ones.

That finding was measured over 98 languages. This extension asks whether it survives at 200 languages — twice the coverage, and far deeper into the long tail where guard training data, tokenizer support, and translation quality all thin out at once. The question is not only whether bypass rates rise, but whether the structure of the earlier result holds: whether guards continue to disagree about which prompts to admit, and whether downstream agents keep recovering from upstream classification errors at the same rate once the language is one the stack has barely seen.

What This Extension Adds

  • 200 languages rather than 98, extending the evaluation well past the point where most guard models document support.
  • A deeper low-resource tail, where the earlier study’s resource-tier gap is most likely to widen — and where translation quality becomes hardest to distinguish from genuine safety behavior.
  • A larger base for guard comparison, since the earlier work found that the two most permissive guards agreed on only 65.5% of their admitted sets by Jaccard, and the most and least permissive on just 0.5% — a disagreement that more languages should sharpen rather than average away.

Design Carried Over

The evaluation keeps the guard-to-action pipeline established in the parent study, so results remain comparable:

  1. Unsafe roots from AEGIS2.0, translated across the language set, with each instance retaining its root identifier, harm category, language-resource tier, and translation-quality metadata.
  2. Runtime guardrails screening every prompt through adapters that normalize heterogeneous outputs, with unknown or malformed verdicts handled fail-closed. Since every prompt is gold-labeled unsafe, a safe verdict is a bypass.
  3. Tool-using agents receiving guard-admitted requests alongside an inert, category-relevant restricted tool. Agents may refuse, escalate, or call the tool; parsing and inference failures are recorded separately, never as safe outcomes.
  4. Matched comparisons via deterministic sample identifiers, so the same harmful request can be followed across languages, guards, context conditions, and agents.

Results

Placeholder — the 200-language run is in progress. Guard bypass rates, downstream restricted-tool selection, and the resource-tier breakdown will be added here once the evaluation completes.

BibTeX

@misc{alam2026guardrails200,
  title  = {When Guardrails Fail at Scale: Guardrail Evaluation Across 200 Languages},
  author = {Alam, Md Jahangir and Hossain, Ismail and Ahad, Tanzim and Puppala, Sai and Talukder, Sajedul},
  year   = {2026},
  note   = {In preparation; extended version of \emph{When Guardrails Fail} (NeurIPS 2026 LCFM)}
}