跳转至

文章背景与核心概要

当前针对大语言模型（LLM）智能体长期记忆的评估，高度依赖于对话式回忆基准（如 LoCoMo 和 LongMemEval）。这类基准通常仅测量对话历史上的简单问答，而未能评估被记住的事实是否能从根本上改善工具调用智能体在动态环境中的行为。

为了填补这一空白，作者推出了 MERIT（面向真实受控任务的记忆评估，Memory Evaluation for Realistic Instrumented Tasks）。MERIT 是一个综合基准与测试平台，旨在严格且明确的成本核算下，测量记忆对执行任务智能体的边际效用。该研究通过跨三个领域的 23,440 个评分回合测试发现，虽然记忆能够显著提升任务成功率，但不同的记忆架构（如嵌入检索与结构化存储）在面对事实更新时表现出巨大差异，且全上下文重放（Full context replay）在经济上永远不具备可行性。作者已公开了该基准测试、评估平台及所有执行轨迹。

When Does Memory Help? A Cost-Aware Evaluation of Long-Term Memory in Tool-Using LLM Agents

Authors: Shweta Mishra, Shashank Mishra
ArXiv ID: arXiv:2609.05441 [cs.AI]
Submitted: July 26, 2026
Subjects: Artificial Intelligence (cs.AI)

📋 Summary

当前针对大语言模型（LLM）智能体长期记忆的评估，高度依赖于对话式回忆基准（如 LoCoMo 和 LongMemEval）。这些基准通常仅测量对话历史上的简单问答，而不是评估被记住的事实是否能从根本上改善工具调用智能体在动态环境中的行为。

Current evaluations of long-term memory for Large Language Model (LLM) agents rely heavily on conversational recall benchmarks (such as LoCoMo and LongMemEval). These benchmarks typically measure simple question-answering over a dialogue history rather than evaluating whether remembered facts fundamentally improve the behavior of a tool-using agent in dynamic environments.

为了解决这一空白，作者引入了 MERIT（面向真实受控任务的记忆评估，Memory Evaluation for Realistic Instrumented Tasks）。MERIT 是一个综合基准与测试平台，旨在严格且明确的成本核算下，测量记忆对任务执行智能体的边际效用。

To address this gap, the authors introduce MERIT (Memory Evaluation for Realistic Instrumented Tasks). MERIT is a comprehensive benchmark and test harness designed to measure the marginal utility of memory for task-executing agents under strict, explicit cost accounting.

Key Findings & Methodology:

  • MERIT 基准： 包含三个领域的片断式工具调用任务。通过自动化泄漏检查验证任务对先前回合的依赖性，并设置了难度阶梯，最终涵盖更新事实的回忆、受控记忆损坏，以及对每项记忆操作的完整 Token/美元计量。
  • 性能影响： 在涉及多种模型配置的 23,440 个评分回合（总计算成本为 42.57 美元）中，记忆成功将依赖任务的成功率从经泄漏验证的基线 0.00 提升至 0.55–1.00。
  • 嵌入检索与结构化存储的对比： 在处理更新后的事实时，嵌入检索会出现不可预测的崩溃（不同模型间的成功率从 0.30 到 0.95 不等，最大种子差距达 0.45），并且智能体仅在 55% 的情况下依据正确检索到的数值采取行动。相反，写入时更新存储（如结构化事实存储和 LLM 摘要）始终保持 0.70–1.00 的成功率。有趣的是，混合方法的表现甚至不如单一的结构化事实存储。
  • 成本效益： 更换记忆实现可以将任务成功率改变多达 60 个百分点。此外，全上下文重放（Full context replay）在经济上永远不具备可行性；与全上下文重放相比，每个领域中表现最好的记忆条件能提供 2.7–3.9倍 的每美元边际效用。

Key Findings & Methodology:

  • The MERIT Benchmark: Features episodic tool-use tasks across three domains. Task dependence on prior episodes is verified using automated leak checks, featuring a difficulty ladder ending in updated-fact recall, controlled memory corruption, and full token/dollar metering for every memory operation.
  • Performance Impact: Across 23,440 scored episodes ($42.57 total compute cost) involving multiple model configurations, memory successfully lifts dependent-task success rates from a leak-verified floor of 0.00 up to 0.55–1.00.
  • Embedding vs. Structured Stores: When handling updated facts, embedding retrieval collapses unpredictably (success rates ranging from 0.30 to 0.95 across models with a maximum seed gap of 0.45), and agents act on correctly retrieved values only 55% of the time. Conversely, update-on-write stores (like structured fact stores and LLM summarization) consistently maintain a 0.70–1.00 success rate. Interestingly, hybrid approaches perform worse than structured fact stores alone.
  • Cost Efficiency: Swapping a memory implementation can shift task success by up to 60 percentage points. Furthermore, full context replay is never economical; the best-performing memory condition per domain delivers 2.7–3.9x its marginal utility per dollar compared to full replay.

作者已公开了该基准测试、评估平台及所有执行轨迹。

The authors have released the benchmark, evaluation harness, and all execution traces publicly.