跳转至

视觉语言模型中基于方向的推理时防御机制的跨架构审计

文章背景与核心概要

视觉语言模型（VLM）的越狱攻击给人工智能系统的安全部署带来了严峻挑战。当前主流的推理时防御方法通常通过在特定解码器层的残差流中减去一个经过校准的方向来抵御攻击。然而，这些防御策略在不同架构之间的有效性和可迁移性尚不明确。

本文对四种主流架构家族中的 15 个模型与层组合进行了深入研究，系统评估了五种不同的防御候选方案。研究采用了一种严格的幅度控制协议，将每种方向与相同范数的随机控制进行配对。核心发现表明：没有任何单一的防御方案能够在拒绝恢复（refusal recovery）和效用保持（utility preservation）指标上全面领先；图像条件平移在 LLaVA 1.5 和 Pixtral 12B 上表现最佳；且基于方向的防御具有极强的架构特异性，无法直接跨语言解码器家族进行迁移。

📋 Summary

This research paper investigates inference-time defences against vision-language model (VLM) jailbreaks, which typically involve subtracting a calibrated direction from the residual stream at a specific decoder layer.

The authors evaluate five distinct defence candidates across 15 model and layer cells from four architectural families. Using a magnitude-controlled protocol that pairs every direction with a random control of the same norm, the study yields the following key insights: * No single defense dominates all metrics for both refusal recovery and utility preservation. * Image conditioning shifts perform best on LLaVA 1.5 and Pixtral 12B, maintaining utility loss at the measurement noise floor across every family. * Prompt instructions lead on Qwen2.5 VL, while attack-specific residuals excel on Qwen2 VL 2B. * Direction-based defences are highly architecture-specific and non-transferable, meaning they must be calibrated independently for each language decoder family.

📄 Metadata

  • arXiv ID: arXiv:2607.27910 [cs.AI]
  • Primary Subject: Artificial Intelligence (cs.AI)
  • Authors: Xiangyu Yin, Tora Bodin, Rohan Menon, Chih-Hong Cheng
  • Submission Dates:
  • Submitted: 30 July 2026 (v1)
  • Last Revised: 23 August 2026 (v2)
  • DOI: 10.48550/arXiv.2607.27910

🔍 Abstract

针对视觉语言模型越狱攻击的推理时防御，通常会在选定的解码器层从残差流中减去一个经过校准的方向。我们在四个架构家族的 15 个模型与层单元中，在幅度控制协议下比较了五个防御候选方案，该协议匹配了每个提示词的干预大小，并将每个方向与相同范数的随机对照配对。这些候选方案包括：平均图像条件平移、CMRM 风格的拒绝方向、ShiftDC 风格的攻击特定残差、忽略图像的提示词指令以及随机对照。没有单一的候选方案能够在拒绝恢复和效用保持上同时占优。图像条件平移在 LLaVA 1.5 和 Pixtral 12B 上处于领先地位，并且是唯一一个在每个家族中效用损失都保持在测量噪声底线上的候选方案。提示词指令在 Qwen2.5 VL 上领先，而攻击特定残差在 Qwen2 VL 2B 上领先。图像条件方向在 15 个单元中有 13 个表现出方向特异性，但在唯一维度兼容的配对（LLaVA 1.5 13B 和 Pixtral 12B）之间表现出强烈的架构特异性且不可迁移。我们还将纯文本和多模态拒绝几何结构联系起来。CMRM 方向在所有 15 个单元中与图像条件平移具有正的余弦对齐，平均值为 0.35，范围从 0.17 到 0.65，是随机向量零值的 15 到 25 倍，符号检验 p 值约为 3e-5。这些结果表明，这两种方法恢复了部分重叠的几何结构，并且基于方向的防御应该针对每个语言解码器家族分别进行校准。

Inference time defences against vision language model jailbreaks often subtract a calibrated direction from the residual stream at a chosen decoder layer. We compare five defence candidates across 15 model and layer cells from four architectural families under a magnitude controlled protocol that matches the intervention size for each prompt and pairs every direction with a random control of the same norm. The candidates are the mean image conditioning shift, a CMRM style refusal direction, a ShiftDC style attack specific residual, a prompt instruction to ignore the image, and a random control. No single candidate dominates on both refusal recovery and utility preservation. The image conditioning shift leads on LLaVA 1.5 and Pixtral 12B and is the only candidate whose utility loss remains at the measurement noise floor in every family. The prompt instruction leads on Qwen2.5 VL, while the attack specific residual leads on Qwen2 VL 2B. The image conditioning direction is direction specific in 13 of 15 cells, but strongly architecture specific and nontransferable across the only dimension compatible pair, LLaVA 1.5 13B and Pixtral 12B. We also connect text only and multimodal refusal geometry. The CMRM direction has positive cosine alignment with the image conditioning shift in all 15 cells, with mean 0.35, range 0.17 to 0.65, 15 to 25 times the random vector null, and a sign test p value of about 3e-5. These results show that the two recipes recover partially overlapping geometry and that direction based defences should be calibrated separately for each language decoder family.