跳转至

效用遭受攻击：智能体记忆投毒与内容筛选及溯源排名的局限性

文章背景与核心概要

随着具备持久化记忆能力的AI智能体广泛应用，其记忆系统的安全性面临着严峻挑战。本文深入研究了持久化智能体记忆对“记忆投毒（memory poisoning）”的脆弱性，揭示了微量虚假信息注入即可大幅降低AI系统准确率的现象。作者对当前主流的防御机制（如写入时内容筛选和溯源加权检索）进行了严格评估，指出了它们在区分真伪信息和平衡安全性与效用方面的根本局限性。

研究表明，仅污染1.2%的语料库就会导致系统准确率从0.850暴跌至0.300，且常规的内容筛选流水线无法有效拦截语义伪造的记忆。针对这一困境，论文进一步探讨了溯源加权排名的失效边界，并创新性地倡导在检索阶段采用“有界占用约束（bounded occupancy constraints）”来代替加性溯源惩罚。该研究为构建更安全的AI长期记忆架构提供了重要的理论参考与开源实验资源。

📌 执行摘要 (Executive Summary)

本文调查了持久化智能体记忆对记忆投毒（memory poisoning）的脆弱性，展示了微小的虚假信息注入如何严重降低AI系统的准确率。作者评估了当前的防御机制——具体为写入时内容筛选（write-time content screening）与溯源加权检索（provenance-weighted retrieval）——并指出了它们的根本局限性：

This paper investigates the vulnerability of persistent agent memories to memory poisoning, demonstrating how minor injections of false information can severely degrade AI system accuracy. The author evaluates current defense mechanisms—specifically write-time content screening and provenance-weighted retrieval—and highlights their fundamental limitations:

  • 内容筛选失效： 一个四阶段的写入时筛选流水线（对间接提示词注入有效）未能拦截哪怕一个措辞平淡的虚假记忆，这表明区分虚假断言与真实断言需要超出纯文本分析的外部基础支撑。
    • Content Screening Failure: A four-stage write-time screening pipeline (effective against indirect prompt injections) failed to block a single plainly worded false memory, revealing that distinguishing false assertions from true ones requires external grounding beyond textual analysis alone.
  • 溯源排名困境： 加性溯源惩罚被证明在很大程度上无效。在典型的相似性机制下，任何足以阻挡查询形状投毒的加权强度，同时也足以抑制合法的非受信证据。
    • Provenance Ranking Dilemma: Additive provenance penalties proved largely ineffective. Under typical similarity regimes, any weighting strong enough to block query-shaped poison is simultaneously strong enough to suppress legitimate untrusted evidence.
  • 提出的替代方案： 论文主张在检索时采用有界占用约束（bounded occupancy constraints）而非加性溯源惩罚，并发布了所有相关的实验测试套件、语料库和运行报告。
    • Proposed Alternative: The paper advocates for bounded occupancy constraints at retrieval time rather than additive provenance penalties, and releases all associated experimental harnesses, corpora, and run reports.

📄 摘要 (Abstract)

持久化记忆使虚假信息具有持久性：一旦存储了虚假陈述，它就可以被检索到未来与之匹配的会话中。我们通过单次生成的、无指令、无触发器或无检索器优化的措辞平淡的虚假断言，来衡量这种失效模式的代价。

Persistent memory makes false information durable: once a false statement is stored, it can be retrieved into future sessions that match it. We measure the cost of this failure mode using plainly worded false assertions generated in a single pass, with no instruction, trigger, or retriever optimization.

  • 投毒的影响： 仅污染 LongMemEval 语料库的 1.2% 就会将系统准确率从 0.850 降低到 0.300。
    • Impact of Poisoning: Poisoning just 1.2% of a LongMemEval corpus reduces system accuracy from 0.850 to 0.300.
  • 内容筛选的局限性： 一个在间接提示词注入上达到 0.832 召回率、同时标记 1.5% 带有触发词的良性文本的四阶段写入时筛选流水线，拒绝了 360 个被投毒记忆中的 0 个。我们认为这暴露了纯内容筛选的边界：将虚假断言与真实断言区分开来，通常需要超出文本本身的外部基础支撑。
    • Limits of Content Screening: A four-stage write-time screening pipeline that reaches 0.832 recall on indirect prompt injection while flagging 1.5% of trigger-word-laden benign text rejects 0 of 360 poisoned memories. We argue this exposes a boundary of content-only screening: distinguishing a false assertion from a true one generally requires external grounding beyond the text itself.
  • 溯源加权检索的评估： 发布的权重在统计学上与无防御无异 (\(p=0.80\))，而更强的权重只有在排除非受信内容的情况下才能恢复效用。
    • Evaluation of Provenance-Weighted Retrieval: The shipped weight is statistically indistinguishable from no defense (\(p=0.80\)), while a stronger weight recovers utility only by excluding untrusted content.
  • 在非受信内容大多为良性的混合溯源语料库中，准确率从 0.3167 上升到 0.7000。 > * In a mixed-provenance corpus where untrusted content is mostly benign, accuracy rises from 0.3167 to 0.7000.
  • 当承载答案的证据本身到达时是不受信任的，证据召回率下降至零，准确率下降至 0.0417。 > * When the answer-bearing evidence itself arrives untrusted, evidence recall falls to zero and accuracy to 0.0417.
  • 结论： 在所测量的相似性机制下，加性溯源项没有可用的设置：足以抵御查询形状投毒的权重也足以抑制合法的非受信证据。因此，我们主张在检索时使用有界占用约束而不是加性溯源惩罚，并发布了测试套件、语料库和聚合运行报告。
    • Conclusion: Under the measured similarity regime, the additive provenance term has no usable setting: a weight strong enough to resist query-shaped poison is also strong enough to suppress legitimate untrusted evidence. We therefore argue for bounded occupancy constraints at retrieval rather than additive provenance penalties, and release the harnesses, corpora, and aggregate run reports.

🔗 获取与资源 (Access & Resources)