跳转至

文章背景与核心概要

在多智能体强化学习（MARL）中，智能体不仅需要决定“分享什么”，更需要高效地确定“何时”进行通信。传统方法往往强制在每个时间步进行持续通信，或者依赖高方差的 REINFORCE 策略梯度来学习二元门控机制，这会导致行为不稳定且难以解释。

为了解决这一问题，Teoman Kaman 提出了一种有原则的替代方案：只有当智能体所学习的信念分布之间的库尔贝克-莱布勒（KL）散度超过预定义阈值（\(\varepsilon\)）时，智能体才进行通信。该研究在 Predator-Prey 与 MPE simple_spread 基准测试中验证了该方法的有效性，不仅在复杂环境下超越了传统 IC3Net 方法，还证明了信念头（belief head）在改善潜在表征和整体协作方面的正交优势。

When to Communicate: Belief Distributions and KL Divergence for Principled Gating in Multi-Agent RL

  • Authors: Teoman Kaman
  • Submitted on: 21 May 2026
  • Primary Subject: Artificial Intelligence (cs.AI)
  • Secondary Subject: Machine Learning (cs.LG)
  • arXiv ID: 2608.14559 [cs.AI]
  • DOI: 10.48550/arXiv.2608.14559

When to Communicate: Belief Distributions and KL Divergence for Principled Gating in Multi-Agent RL

  • Authors: Teoman Kaman
  • Submitted on: 21 May 2026
  • Primary Subject: Artificial Intelligence (cs.AI)
  • Secondary Subject: Machine Learning (cs.LG)
  • arXiv ID: 2608.14559 [cs.AI]
  • DOI: 10.48550/arXiv.2608.14559

📋 摘要

多智能体强化学习（MARL）要求智能体高效确定何时通信，而不仅仅是共享什么。传统方法要么强制在每个时间步进行持续通信，要么依赖高方差的 REINFORCE 策略梯度来学习二元门控机制，从而导致不稳定且难以解释的行为。

为了解决这一问题，Teoman Kaman 提出了一种有原则的替代方案：智能体仅在它们所学习的信念分布之间的库尔贝克-莱布勒（KL）散度超过预定义阈值（\(\varepsilon\)）时才进行通信。

核心机制：

  • 信念表征： 每个智能体通过对其 LSTM 隐藏状态应用 softmax 函数，计算出关于潜在世界状态的信念分布。
  • 有原则的门控： 仅当智能体之间的信念分歧超过阈值时才交换信息，从而合理化通信成本。

评估与发现：

  • 捕食者-猎物（Predator-Prey, PP）基准测试：
  • 在较小的 \(10 \times 10\) 网格上，IC3Net 在所有阈值下均优于 KL-belief。
  • 在较难的 \(20 \times 20\) 网格上，阈值消融实验（\(\varepsilon \in \{0.1, 0.3, 0.5, 1.0\}\)）展示了倒 U 形的性能曲线。具体而言，\(\varepsilon = 0.5\) 时平均步数为 73.84 步，成功率为 42%，表现优于 IC3Net（75.31 步和 31%），且随机种子方差更小。
  • MPE simple_spread 基准测试：
  • 即使在门控处于非活动状态时，集成信念头也能将平均奖励提升 12 分，并将方差降低 \(26\times\)。这表明了两个正交的好处：信念收敛期间的有原则门控，以及显著改善增强整体协调性的潜在表征。

📋 Summary

Multi-Agent Reinforcement Learning (MARL) requires agents to efficiently determine when to communicate, rather than just what to share. Traditional approaches either force constant communication at every timestep or rely on high-variance REINFORCE policy gradients to learn binary gating mechanisms, leading to unstable and uninterpretable behaviors.

To address this, Teoman Kaman proposes a principled alternative: agents communicate only when the Kullback-Leibler (KL) divergence between their learned belief distributions exceeds a predefined threshold (\(\varepsilon\)).

Key Mechanisms:

  • Belief Representation: Each agent computes a belief distribution over a latent world state using a softmax function applied over its LSTM hidden state.
  • Principled Gating: Information is exchanged strictly when belief disagreement between agents surpasses the threshold, justifying the communication cost.

Evaluation & Findings:

  • Predator-Prey (PP) Benchmark:
  • On a smaller \(10 \times 10\) grid, IC3Net outperforms KL-belief across all thresholds.
  • On a harder \(20 \times 20\) grid, a threshold ablation (\(\varepsilon \in \{0.1, 0.3, 0.5, 1.0\}\)) shows an inverted U-shape performance. Specifically, \(\varepsilon = 0.5\) achieves an average of 73.84 steps and a 42% success rate, outperforming IC3Net (75.31 steps and 31%) with tighter seed variance.
  • MPE simple_spread Benchmark:
  • Integrating the belief head improves mean reward by 12 points and reduces variance by \(26\times\), even when gating is inactive. This indicates two orthogonal benefits: principled gating during belief convergence, and significantly improved latent representations that bolster overall coordination.

🔗 全文链接与资源

🔗 Full-Text Links & Resources

🗂️ 提交历史

  • [v1] 2026年5月21日 周四，23:31:37 UTC (573 KB)

🗂️ Submission History

  • [v1] Thu, 21 May 2026, 23:31:37 UTC (573 KB)