文章背景与核心概要

现有的离线上下文强化学习（ICRL）方法主要依赖于监督学习训练目标，这在离线强化学习场景中天生具有局限性。本文探讨了将强化学习目标整合到离线ICRL框架中的可行性与优势。

通过对来自GridWorld和MuJoCo环境的150多个数据集进行广泛实验，作者证明：与广受采用的算法蒸馏（AD）基线相比，直接优化强化学习目标平均可将性能提升约30%。此外，在极具挑战性的XLand-MiniGrid环境中，引入强化学习目标使AD的性能翻了一番。这些发现强调了将ICRL学习目标与奖励最大化目标保持一致的重要性，并证明了离线强化学习是推进ICRL发展的一个强有力的方向。

Yes, Q-learning Helps Offline In-Context RL

Yes, Q-learning Helps Offline In-Context RL

Summary

Summary

现有离线上下文强化学习（ICRL）方法主要依赖于监督学习训练目标，这在离线RL设置中本质上具有局限性。本文探讨了在离线ICRL框架中集成RL目标的方案。通过对来自150多个GridWorld和MuJoCo环境的数据集进行广泛实验，作者证明，与广泛采用的算法蒸馏（AD）基线相比，直接优化RL目标可将性能平均提升约30%。此外，在具有挑战性的XLand-MiniGrid环境中，结合RL目标使AD的性能翻了一番。这些发现强调了将ICRL学习目标与奖励最大化目标对齐的重要性，证明了离线RL是推进ICRL发展的强有力方向。

Existing offline in-context reinforcement learning (ICRL) methods have predominantly relied on supervised training objectives, which inherently possess limitations in offline RL settings. This paper explores the integration of RL objectives within an offline ICRL framework. Through extensive experiments across more than 150 GridWorld and MuJoCo environment-derived datasets, the authors demonstrate that optimizing RL objectives directly improves performance by roughly 30% on average compared to the widely adopted Algorithm Distillation (AD) baseline. Furthermore, in the challenging XLand-MiniGrid environment, incorporating RL objectives doubled the performance of AD. The findings underscore the importance of aligning ICRL learning objectives with reward-maximization goals, proving that offline RL is a powerful direction for advancing ICRL.

Paper Metadata

Paper Metadata

Field	Details
arXiv Identifier	arXiv:2502.17666 [cs.LG]
Primary Subject	Machine Learning (cs.LG)
Secondary Subjects	Artificial Intelligence (cs.AI)
Authors	Denis Tarasov, Alexander Nikulin, Ilya Zisman, Albina Klepach, Andrei Polubarov, Nikita Lyubaykin, Alexander Derevyagin, Igor Kiselev, Vladislav Kurenkov
Submitted Date	24 February 2025 (v1); Last revised 13 August 2026 (v5)
License	Creative Commons Attribution 4.0 International
Project Code / Repository	GitHub Repository (dunnolab/yesq)

Abstract

Abstract

现有的离线上下文强化学习（ICRL）方法主要依赖于监督训练目标，众所周知，这些目标在离线RL设置中存在局限性。在本研究中，我们探讨了在离线ICRL框架中集成RL目标的方法。

通过对来自GridWorld和MuJoCo环境的150多个数据集进行的实验，我们证明：在各种数据集覆盖范围、结构、专家水平和环境复杂度下，与广泛采用的算法蒸馏（AD）相比，优化RL目标可将性能直接平均提升约30%。此外，在充满挑战的XLand-MiniGrid环境中，RL目标的性能达到了AD的两倍。

我们的结果还表明，在价值学习过程中引入保守性（conservatism）在几乎所有测试设置中都带来了额外的性能提升。我们的发现强调了将ICRL学习目标与RL奖励最大化目标对齐的重要性，并证明了离线RL是推进ICRL的一个有前景的方向。

Existing offline in-context reinforcement learning (ICRL) methods have predominantly relied on supervised training objectives, which are known to have limitations in offline RL settings. In this study, we explore the integration of RL objectives within an offline ICRL framework.

Through experiments on more than 150 GridWorld and MuJoCo environment-derived datasets, we demonstrate that optimizing RL objectives directly improves performance by approximately 30% on average compared to widely adopted Algorithm Distillation (AD), across various dataset coverages, structures, expertise levels, and environmental complexities. Furthermore, in the challenging XLand-MiniGrid environment, RL objectives doubled the performance of AD.

Our results also reveal that the addition of conservatism during value learning brings additional improvements in almost all settings tested. Our findings emphasize the importance of aligning ICRL learning objectives with the RL reward-maximization goal, and demonstrate that offline RL is a promising direction for advancing ICRL.

Key Takeaways & Contributions

Key Takeaways & Contributions

1. 以RL目标取代监督目标： 抛弃标准的监督训练目标（如算法蒸馏），转向直接最大化奖励的RL目标，能够带来巨大的性能提升。
2. 广泛的实证验证： 在衍生自GridWorld和MuJoCo环境的150多个数据集上进行了测试，展示了在不同数据集覆盖范围、结构和专家水平下的强健性能提升。
3. 挑战性环境： 在XLand-MiniGrid等复杂设置中，集成RL目标成功将基线性能提升了一倍。
4. 价值保守性： 在价值学习中引入保守性，在几乎所有测试配置下都带来了额外的性能收益。

1. RL Objectives over Supervised Objectives: Moving away from standard supervised training objectives (such as Algorithm Distillation) to direct reward-maximization RL objectives yields a massive performance boost.
2. Extensive Empirical Validation: Tested across more than 150 datasets derived from GridWorld and MuJoCo environments, showing robust improvements across different dataset coverages, structures, and expertise levels.
3. Challenging Environments: In complex settings like XLand-MiniGrid, integrating RL objectives successfully doubled baseline performance.
4. Value Conservatism: The introduction of conservatism during value learning provides extra performance gains across nearly all tested configurations.