跳转至

Dream-MPC：基于潜在想象的梯度模型预测控制

文章背景与核心概要

在基于模型的强化学习（MBRL）领域中，如何在大规模连续控制任务中高效地进行动作规划与策略优化一直是核心挑战。传统的基于总体的无梯度规划方法虽然鲁棒，但在高维控制任务中往往面临巨大的计算开销；而现有的基于梯度的规划器虽然理论上更具效率，但在实际应用中的性能却常常落后于无梯度方法。

为了弥补这一技术鸿沟，来自波恩大学的研究团队在 2026 年国际机器学习大会（ICML 2026）上发表了 Dream-MPC。该方法巧妙地结合了策略前瞻生成、基于学到的世界模型的梯度上升优化、不确定性正则化以及动作平摊（Amortization）机制。在涵盖 24 个复杂连续控制任务的大量基准测试中，Dream-MPC 展现出了卓越的性能，显著超越了现有的无梯度 MPC 以及各类最先进的基线方法。

Executive Summary (执行摘要)

Dream-MPC is a novel model-based reinforcement learning approach designed for high-dimensional continuous control tasks. Accepted at the International Conference on Machine Learning (ICML) 2026, the paper introduces a gradient-based Model Predictive Control (MPC) framework that addresses the limitations of both traditional gradient-free methods and existing gradient-based planners. By utilizing a rolled-out policy to generate initial candidate trajectories, optimizing them via gradient ascent with a learned world model, applying uncertainty regularization, and amortizing optimization over time, Dream-MPC significantly improves policy performance and outperforms current state-of-the-art baselines across 24 continuous control tasks.

Dream-MPC 是一种专为高维连续控制任务设计的新型基于模型的强化学习方法。该论文已被 2026年国际机器学习大会（ICML 2026） 接收，引入了一种基于梯度的模型预测控制（MPC）框架，旨在解决传统无梯度方法与现有基于梯度规划器的局限性。通过利用前瞻策略生成初始候选轨迹、借助学到的世界模型通过梯度上升对其进行优化、应用不确定性正则化以及随时间平摊优化过程，Dream-MPC 显著提升了策略性能，并在 24 个连续控制任务中全面超越了当前最先进的基线方法。

Paper Metadata (论文元数据)

  • arXiv ID: arXiv:2605.04568 [cs.LG]
  • Authors: Jonathan Spieler, Sven Behnke (University of Bonn)
  • Primary Subject: Machine Learning (cs.LG)
  • Secondary Subjects: Artificial Intelligence (cs.AI), Robotics (cs.RO)
  • Conference: Accepted for International Conference on Machine Learning (ICML) 2026
  • Submission History:
  • v1: May 6, 2026
  • v3: August 6, 2026 (Latest revision)
  • Resources & Links:
  • View PDF
  • Project Website, Code, and Videos
  • arXiv ID: arXiv:2605.04568 [cs.LG]
  • Authors: Jonathan Spieler, Sven Behnke (University of Bonn)
  • Primary Subject: Machine Learning (cs.LG)
  • Secondary Subjects: Artificial Intelligence (cs.AI), Robotics (cs.RO)
  • Conference: Accepted for International Conference on Machine Learning (ICML) 2026
  • Submission History:
  • v1: May 6, 2026
  • v3: August 6, 2026 (Latest revision)
  • Resources & Links:
  • View PDF
  • Project Website, Code, and Videos

Abstract (摘要)

State-of-the-art model-based Reinforcement Learning (RL) approaches either use gradient-free, population-based methods for planning, learned policy networks, or a combination of policy networks and planning. Hybrid approaches that combine Model Predictive Control (MPC) with a learned model and a policy prior to leverage the advantages of both paradigms have shown promising results.

当前最先进的基于模型的强化学习（RL）方法主要采用基于无梯度、基于总体的规划方法、学到的策略网络，或者是策略网络与规划的结合。将模型预测控制（MPC）与学到的模型及策略先验相结合的混合方法，能够汲取两种范式的优势，并已展现出良好的应用前景。

However, these approaches typically rely on gradient-free optimization methods, which can be computationally expensive for high-dimensional control tasks. While gradient-based methods are a promising alternative, recent works have empirically shown that gradient-based methods often perform worse than their gradient-free counterparts.

然而，这些方法通常依赖于无梯度的优化方法，这在高维控制任务中可能会带来高昂的计算成本。尽管基于梯度的方法是一种很有前途的替代方案，但近期的研究在实证上表明，基于梯度的方法其性能往往逊色于无梯度的同类方法。

To overcome this, Dream-MPC introduces: 1. Candidate Generation: Generates a few candidate trajectories from a rolled-out policy. 2. Gradient-Based Optimization: Optimizes each trajectory using gradient ascent with a learned world model. 3. Uncertainty Regularization: Ensures stability and reliability during imagination rollouts. 4. Action Amortization: Reuses previously optimized actions to amortize optimization iterations over time.

为了克服这一难题，Dream-MPC 引入了以下四大核心改进： 1. 候选生成（Candidate Generation）： 从前瞻策略中生成少量候选轨迹。 2. 基于梯度的优化（Gradient-Based Optimization）： 结合学到的世界模型，利用梯度上升法对每条轨迹进行优化。 3. 不确定性正则化（Uncertainty Regularization）： 确保在想象前瞻（imagination rollouts）过程中的稳定性和可靠性。 4. 动作平摊（Action Amortization）： 复用先前优化过的动作，将优化迭代的开销平摊到不同的时间步长中。

Evaluated on 24 continuous control tasks, Dream-MPC significantly improves the performance of the underlying policy, consistently outperforming gradient-free MPC and state-of-the-art baselines.

在 24 个连续控制任务上的评估表明，Dream-MPC 显着提升了底层策略的性能，并在持续性表现上超越了无梯度 MPC 以及当前最先进的基线方法。