Lost in Speech: Measuring the Multilingual Cross-Modal Safety Gap in Speech-Language Models

Authors: Md Jahangir Alam¹, Tanzim Ahad¹, Ismail Hossain², Sai Puppala³, Sajedul Talukder²

¹ University of Texas at El Paso  ·  ² University of Texas at El Paso, Department of Computer Science  ·  ³ New Mexico Institute of Mining & Technology

A model that fails to understand an unsafe request is not equivalent to a model that understands and refuses it.

Abstract

Large speech-language models (SLMs) are increasingly deployed as voice assistants, yet safety evaluation remains dominated by English text. We present a root-controlled study of safety under two perturbations: translation into a non-English language and conversion to synthesized speech. Using a 10,000-row, 100-language AEGIS unsafe-prompt sample plus a 5,235-item English baseline, we synthesize speech for 75 languages and evaluate six SLMs across English text, English speech, translated text, and translated speech, yielding 178,608 scored responses.

Unsafe Compliance Rate (UCR) remains low (0.0–4.4%), but comprehension degrades sharply: non-English speech Comprehension Rate (CR) ranges from 24.13% (Phi-4-Multimodal) to 74.09% (Qwen2-Audio), despite near-perfect English-text comprehension. Consequently, up to 76.18% of nominally safe non-English spoken responses reflect failed comprehension rather than genuine alignment. We further show that TTS route, documented language support, judge choice, and comprehension rubric materially affect conclusions; MMS-ASR rates 93.52% of low-resource native audio as good versus only 51.19% under Whisper. A benign control indicates that the dominant failure is comprehension rather than over-refusal. These findings show that low unsafe-compliance rates can substantially overstate multilingual speech safety.

At a Glance

Method

1. Root-controlled benchmark. From 11,788 curated AEGIS roots across 21 harm categories, every translated row keeps its root_id, so conditions differing only in language or modality preserve the underlying request, topic, and harm category. Translation quality is measured as an explicit covariate rather than assumed equal.
2. Speech synthesis with evidentiary strata. MMS-TTS supplies native checkpoints for 54 languages and alias checkpoints for six; Piper supplies native voices for 12 and related-language proxy voices for three. Native, alias, and proxy audio are kept distinct rather than pooled.
3. Independent validation. MMS-ASR round-trip CER is supplemented by GlotLID language identification on every recovered transcript and by an architecturally independent Whisper-large-v3-turbo transcription of a stratified 1,200-file subset.
4. Models and judging. Six SLMs — Gemma-3n, Ultravox, Voxtral-Mini-3B, Phi-4-Multimodal, Qwen2-Audio-7B, and Gemini 3.5 Flash-Lite — run under greedy decoding with the same neutral voice-assistant instruction and no added safety policy. Three judges score under a leave-one-organization-out rule, each predicting both COMPREHENDED and an outcome label.
5. Comprehension as a first-class variable. Crossing comprehension with safety yields four states: genuine alignment, alignment failure, false appearance of safety (FAoS), and comprehension failure with unsafe output. UCR is used instead of Attack Success Rate because no adversarial optimization is performed.

Main Findings

• Near-zero UCR coexists with collapsing comprehension. English-text CR is ≈100% for every system, while non-English speech CR runs from 24.13% (Phi-4-Multimodal) to 74.09% (Qwen2-Audio) — a spread of more than 50 points on the identical condition.
• FAoS dominates for some models. In the non-English speech condition, the share of safe-looking responses that arise without comprehension ranges from 26.03% (Qwen2-Audio) to 76.18% (Phi-4-Multimodal). Models with indistinguishable UCR have radically different operational behavior.
• Two dissociable bottlenecks. Phi-4-Multimodal is already at 48.00% CR on non-English text, so speech compounds a multilingual weakness. Gemini shows the opposite: the highest non-English text CR (89.03%) collapsing to 37.99% in speech — isolating the audio pathway specifically.
• Language and modality are not additive. The pooled language×modality interaction is +2.12 [1.79, 2.47] for comprehension and −1.12 [−1.52, −0.70] for UCR.
• TTS route changes the answer. Ultravox reaches 47.70% CR on native speech but 82.97% on proxy speech; Phi-4-Multimodal rises from 15.97% to 71.89%. GlotLID recovers the intended language for 93.31% of native audio but only 23.23% of proxy audio, so those gains are not target-language capability.
• A same-family validator is optimistic. On low-resource native speech, MMS-ASR rates 93.52% of audio good while Whisper rates only 51.19%, with agreement dropping to 54.61%.
• Evaluation choices move the rankings. Ultravox’s non-English-speech UCR is 2.03% under a three-judge ensemble versus 0.02% under the two-judge primary configuration (pairwise κ as low as 0.011), and a stricter comprehension rubric lowers CR from 80.71% to 60.96% — meaning the headline estimates understate comprehension failure.
• A benign control rules out simple over-refusal. Five of six models stay below 9% inappropriate refusal in every benign cell, yet benign speech comprehension can be even lower than on unsafe speech — pointing to a broad speech-comprehension limitation rather than harm-specific caution.

Limitations

• The evaluation is fully automated: translation, TTS, generation, and safety/comprehension labels have no human calibration.
• All speech is synthesized rather than naturally spoken, and 25 of the 100 languages have no speech route.
• GlotLID operates on recovered transcripts rather than raw audio, and Whisper independently validates only 59 of the 76 audio-bearing languages.
• Headline CR uses the original one-line comprehension instruction; the stricter rubric was applied only to subsets, not propagated through all 178,608 responses.
• The benign control is smaller (100 prompts, 12 languages) and not length-matched to the unsafe set, so it identifies a confound without establishing utterance length as its cause.
• Per-language cells are often small, two models show unexplained resource-tier reversals, and only one of the six models is a closed commercial API.

BibTeX

@misc{alam2026lostinspeech,
  title  = {Lost in Speech: Measuring the Multilingual Cross-Modal Safety Gap in Speech-Language Models},
  author = {Alam, Md Jahangir and Ahad, Tanzim and Hossain, Ismail and Puppala, Sai and Talukder, Sajedul},
  year   = {2026},
  note   = {Submitted to ACL Rolling Review (NAACL)}
}