适得其反的反馈：为什么小型语言模型智能体总是重复刚目睹失败的函数调用

文章背景与核心概要

在构建小型语言模型（SLM）智能体时，标准的控制框架（harness design）通常会将失败的工具调用连同错误信息一同记录在对话记录中，寄希望于模型能将其用作纠正性反馈。然而，这项研究通过对多个指令微调模型、不同环境以及检查点进行测试，揭示了一个令人惊讶的现象：这种反馈普遍会产生“适得其反”的效果。记录失败不仅没有阻止重复，反而显著增加了模型逐字重复完全相同错误调用的可能性。

该研究进一步指出，问题并非出在模型对错误信息的理解上，而是源于控制框架本身的结构。研究通过反事实实验发现，失败调用本身的表层形式（surface form）是导致这一负面效应的主要原因。为了解决这一痛点，文章评估并提出了多种有效的改进方案，例如运行时生成的描述以及解码器层面的约束，为优化小型语言模型智能体的交互框架提供了有力的实证支持和解决方案。

Summary

When building Small Language Model (SLM) agents, standard harness design records a failed tool call alongside its error message into the conversation transcript, assuming the model will use it as corrective feedback.

This paper investigates whether that assumption holds true. By measuring the "corrective gain" (the change in log-probability of re-emitting a failed action) across multiple instruction-tuned models, environments (simulated tool calling and MBPP program repair), and checkpoints, the author discovers a surprising phenomenon: the feedback universally backfires.

Instead of deterring repetition, recording the failure significantly increases the likelihood that the model will blindly repeat the exact same failed call token-for-token. The study pinpoints the issue not in the models' comprehension of error messages, but in the harness structure itself, offering evidence-backed solutions on how to effectively remedy the problem.

核心发现

Key Findings

• 负向纠正增益： 在测试的所有指令微调模型中（涵盖4个模型家族、参数量从135M到1.7B不等的6个检查点），失败记录的纠正增益均表现为负值。经过动作长度归一化后，其负面效应约为每个动作Token -1.03奈特（每个Token的发生几率恶化了2.8倍），且该结论在90%至100%的独立测试样本中均成立。
• 重复率飙升： 在固定的候选集上，重复失败调用的概率从 0.06飙升至0.54。贪婪解码（Greedy decoding）在失败后会有 19%的样本 出现逐字重复错误调用的情况，而在此之前这一比例为 0%。
• 表层形式 vs. 语义含义： 反事实实验表明，失败调用本身的表层形式（surface form）占据了 83%的负面影响，而将其显式标记为“失败”的语义贡献则非常微弱且在不同环境中表现不稳定。

• Negative Corrective Gain: For every instruction-tuned model tested (6 checkpoints ranging from 135M to 1.7B parameters across 4 families), the corrective gain of a failure record is negative. Normalized by action length, the effect is approximately -1.03 nats per action token (a factor of 2.8 in the odds of each token), holding true across 90%–100% of individual items.
• Soaring Repetition Rates: Across a fixed candidate set, the probability of repeating a failed call rises sharply from 0.06 to 0.54. Greedy decoding reproduces the failed call token-for-token on 19% of items post-failure, compared to 0% before.
• Surface Form vs. Semantic Meaning: Counterfactual experiments reveal that the surface form of the failed call itself accounts for 83% of the damage, whereas the semantic contribution of marking it explicitly as "failed" is minor and inconsistent across environments.

控制框架改进方案评估

研究评估了多种常见及创新的缓解策略：

Evaluation of Harness Remedies

❌ 无效或适得其反的方案

• 显式“请勿重复”指令： 添加严格避免重复失败动作的指令，发现测得的重复指标几乎没有变化。
• 上下文删除（清除历史记录）： 清除失败的尝试以便从干净的上下文中重试——这是应对上下文污染的标准处方——实际上被证明是测试中表现最糟糕的框架配置，因为它无意中恢复了导致最初失败的原始上下文。

❌ Ineffective or Counterproductive Remedies

• Explicit "Do Not Repeat" Instructions: Adding a strict instruction to avoid repeating failed actions leaves the measured repetition metrics practically unchanged.
• Context Deletion (Clearing History): Wiping the failed attempt to retry from a clean context—a standard prescription for context contamination—actually proved to be the worst harness configuration tested, as it inadvertently restores the original context that caused the failure in the first place.

✅ 有效的方案

• 运行时生成的描述： 用动态、运行时生成的失败描述替代逐字记录的失败调用，消除了 76%的逆转现象，且没有任何Token成本的开销。
• 解码器端约束： 直接在解码器层面使先前失败的字符串完全不可达，能够精准定位并有效缓解这一漏洞。

✅ Effective Remedies

• Runtime-Generated Descriptions: Replacing the verbatim failed call with a dynamic, runtime-generated description of the failure eliminates 76% of the inversion without any token cost overhead.
• Decoder-Side Constraints: Making previously-failed strings completely unreachable directly at the decoder level successfully targets and mitigates the same vulnerability.

注：本研究完全在CPU上端到端执行。所有底层人工制品，包括代码、数据、探测项以及运行日志，均已公开发布。

Note: This study was executed end-to-end on a CPU. All underlying artifacts, including code, data, probe items, and rollout logs, have been publicly released.