跳转至

学习不该学的内容：用于鲁棒视觉-语言模型的对抗解耦提示微调

文章背景与核心概要

对抗提示微调能够高效增强视觉-语言模型（VLMs）的鲁棒性，但现有方法在面对已见类别时，往往会遭遇鲁棒泛化过拟合（robust generalization overfitting）问题。随着训练的进行，这会导致模型在抵御未见类别的对抗样本时性能急剧下降。

为了解决这一现象背后的核心根源——即模型倾向于学习伪鲁棒特征（即无法泛化的捷径），本文作者提出了 ADAPT（Adversarial Disentangled Prompt Tuning，对抗解耦提示微调）框架，其核心哲学是“学习不该学的内容”。ADAPT 创新性地采用了一种双提示机制，包含一个目标提示和一个诱饵提示池。在训练过程中，诱饵提示负责捕获各种各样的伪鲁棒特征，而目标提示则在嵌入空间中被约束为与诱饵提示正交，从而剥离并隔离出真正的鲁棒特征。该研究不仅有效防止了鲁棒泛化过拟合，还通过理论分析证明了正交损失对未见类别伪鲁棒特征偏移的有界控制，并在实验中显著提升了模型在未见类别上的对抗鲁棒性。

📋 Summary

Adversarial prompt tuning efficiently enhances the robustness of vision-language models (VLMs), but existing methods often suffer from robust generalization overfitting on seen classes. As training progresses, this causes a rapid performance drop against adversarial examples in unseen classes.

Empirical findings show this degradation is driven by the model's tendency to learn pseudo-robust features (non-generalizable shortcuts). To solve this, the authors introduce ADAPT (Adversarial Disentangled Prompt Tuning), a framework built on the philosophy of "Learning What Not to Learn." ADAPT utilizes a dual-prompt mechanism featuring a target prompt and a pool of decoy prompts. While the decoys capture diverse pseudo-robust features, the target prompt is constrained to be orthogonal to the decoys in the embedding space to isolate genuine robust features.

Adversarial prompt tuning efficiently enhances the robustness of vision-language models (VLMs), but existing methods often suffer from robust generalization overfitting on seen classes. As training progresses, this causes a rapid performance drop against adversarial examples in unseen classes.

Empirical findings show this degradation is driven by the model's tendency to learn pseudo-robust features (non-generalizable shortcuts). To solve this, the authors introduce ADAPT (Adversarial Disentangled Prompt Tuning), a framework built on the philosophy of "Learning What Not to Learn." ADAPT utilizes a dual-prompt mechanism featuring a target prompt and a pool of decoy prompts. While the decoys capture diverse pseudo-robust features, the target prompt is constrained to be orthogonal to the decoys in the embedding space to isolate genuine robust features.

📌 Metadata

  • arXiv ID: arXiv:2608.17306 [cs.CV]
  • Subjects: Computer Vision and Pattern Recognition (cs.CV), Artificial Intelligence (cs.AI)
  • Submission Date: August 18, 2026
  • DOI: 10.48550/arXiv.2608.17306
  • Authors:
  • Yang Chen
  • Zhan Zhuang
  • Yanbin Wei
  • Zebin Chen
  • Hua Liu
  • Yu Zhang
  • arXiv ID: arXiv:2608.17306 [cs.CV]
  • Subjects: Computer Vision and Pattern Recognition (cs.CV), Artificial Intelligence (cs.AI)
  • Submission Date: August 18, 2026
  • DOI: 10.48550/arXiv.2608.17306
  • Authors:
  • Yang Chen
  • Zhan Zhuang
  • Yanbin Wei
  • Zebin Chen
  • Hua Liu
  • Yu Zhang

🚀 Abstract

虽然对抗提示微调能够高效增强视觉-语言模型的鲁棒性，但我们发现现有方法会加剧已见类别上的鲁棒泛化过拟合，导致随着训练的进行，模型在应对未见类别的对抗样本时性能出现急剧下降。我们通过实验发现，这种性能退化源于模型倾向于学习伪鲁棒特征（即无法泛化的捷径）。

为了缓解这一问题，我们提出了 ADAPT（Adversarial Disentangled Prompt Tuning，对抗解耦提示微调），这是一个遵循“学习不该学的内容”哲学的鲁棒提示微调框架。具体而言，ADAPT 采用了一种双提示机制，包含一个目标提示和一个诱饵提示池。在训练过程中，诱饵提示被引导去捕获各种伪鲁棒特征，同时在嵌入空间中约束目标提示与诱饵提示正交，从而学习到真正的鲁棒特征。

通过将鲁棒特征与伪鲁棒特征解耦，ADAPT 有效地防止了鲁棒泛化过拟合。我们进一步提供了理论分析，表明正交损失能够限制伪鲁棒特征在未见类别上的偏移影响，从而提供测试误差保证。从实验来看，广泛的实验证明 ADAPT 显著提升了目标提示在未见类别上的鲁棒性。

While adversarial prompt tuning can enhance robustness of vision-language models efficiently, we find that existing methods aggravate robust generalization overfitting on seen classes, leading to a rapid degradation in performance against adversarial examples of unseen classes as training progresses. We empirically identify that this degradation stems from the tendency of the model to learn pseudo-robust features (i.e., non-generalizable shortcuts).

To mitigate this, we propose ADAPT (Adversarial Disentangled Prompt Tuning), a robust prompt tuning framework following the philosophy of "Learning What Not to Learn". Specifically, ADAPT uses a dual-prompt mechanism with a target prompt and a pool of decoy prompts. During training, the decoy prompts are guided to entrap diverse pseudo-robust features, while the target prompt is constrained to be orthogonal to the decoys in the embedding space to learn robust features.

By disentangling the robust features from the pseudo-robust features, ADAPT effectively prevents robust generalization overfitting. We further provide an analysis showing that the orthogonal loss bounds the effect of shifts in pseudo-robust features on unseen classes, yielding a testing error guarantee. Empirically, extensive experiments demonstrate that ADAPT substantially improves the robustness of the target prompt on unseen classes.