跳转至

语义叠加：利用超越Token和引导向量的注解技术缓解提示词注入

文章背景与核心概要

大语言模型在处理输入时完全依赖Token，依靠模型自身来追踪片段的属性（例如用户输入、工具输出和指令）。这种架构上的脆弱性使得提示词注入攻击能够利用并混淆对文本片段的解析。为了解决这一问题，“语义叠加”（Semantic Overlays）引入了一种非文本的、带外（out-of-band）的注解通道。通过在预填充（prefill）阶段将小型学习适配器应用到冻结模型的残差流中，叠加层能够安全地编码复杂的语义（例如将某个文本片段标记为“不可执行”），而无需替换底层文本或依赖传统的引导向量。

这项由 Joshua Penman 提出的研究在多个标准提示词注入基准测试中展现出了卓越的性能提升，在完全不改变模型原有通用能力的前提下，大幅压低了注入攻击的成功率，为大模型安全性提供了一种创新且高效的防御范式。

Executive Summary

Language models process input exclusively as tokens, relying on the model itself to track span identities (such as user inputs, tool outputs, and instructions). This vulnerability allows prompt injection attacks to exploit and scramble span interpretation. To address this, Semantic Overlays introduce a non-textual, out-of-band annotation channel. By applying small learned adapters to a frozen model's residual stream during prefill, overlays safely encode complex semantics—such as marking a span as "non-executable"—without replacing underlying text or relying on traditional steering vectors.

Paper Metadata

  • Title: Semantic Overlays: Mitigating Prompt Injection with Annotations Beyond Tokens and Steering Vectors
  • Author: Joshua Penman
  • Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Cryptography and Security (cs.CR); Machine Learning (cs.LG)
  • arXiv ID: arXiv:2608.23873 [cs.AI]
  • Dates: Submitted August 24, 2026; Last revised September 3, 2026 (v3)
  • Links:
  • View PDF
  • Interactive Demo
  • Code & Adapters Repository

Abstract & Core Concept

大语言模型将所有内容都处理为Token，这使得片段属性容易受到操纵，攻击者可以将恶意指令伪装成常规文本。语义叠加通过以下方式解决了这一问题： * 带外通信（Out-of-band communication）： 在预填充阶段的特定位置，将小型学习适配器应用到冻结模型的残差流中。 * 复杂的语义编码： 叠加层重塑了模型对被标记片段的感知方式。例如，应用一个断言不同编程语言的叠加层，会使模型将复制的代码片段重写为断言的目标语言。 * 强大的防御机制： 将某个片段标记为“不可执行”，可以有效中和嵌入在不受信任上下文中的大类提示词注入攻击。

Language models process everything as tokens, leaving span identity vulnerable to manipulation where attackers can disguise malicious instructions as routine text. Semantic Overlays solve this by offering: * Out-of-band communication: Small learned adapters applied at specific prefill positions to a frozen model's residual stream. * Complex semantic encoding: Overlays reshape how a model perceives marked spans. For example, applying an overlay that asserts a different programming language causes the model to rewrite copied code snippets into the asserted language. * Robust defense mechanisms: Marking a span as "non-executable" effectively neutralizes broad classes of prompt injections embedded within untrusted contexts.

Key Performance Results

语义叠加在几个标准的提示词注入基准测试中展现出了显著的性能提升：

  • SEP 分离度： 从 24.3% 剧烈提升至 99.0%，同时模型的实用性保持完全不变。
  • TensorTrust： 攻击成功率从 34.8% 显著下降至 6.2%。
  • AlpacaFarm： 成功率从 99.0% 下降至 0%。
  • PIArena： 在所有能够保持模型功能完好的已发布基准防御中，叠加层的表现超越了它们，同时确保了被标记片段的可读性，其字符相似度达到原文本的 >95%。

Semantic Overlays demonstrate remarkable improvements across several standard prompt injection benchmarks:

  • SEP Separation: Rises dramatically from 24.3% to 99.0% with model utility remaining entirely unchanged.
  • TensorTrust: Attack success rate drops significantly from 34.8% to 6.2%.
  • AlpacaFarm: Success rate falls from 99.0% to 0%.
  • PIArena: Overlays outperform every published baseline defense that keeps the model functional, while ensuring marked spans remain readable at >95% character similarity to the original text.