跳转至

文章背景与核心概要

在许多序列决策任务中，智能体通常需要在每个时间步不断观察环境并采取行动，这类似于一个人在咖啡冲泡时一直盯着咖啡壶看。然而，这种持续的监控往往是不必要的且消耗大量资源。本文引入了一种“等待策略”框架，使智能体能够有意放弃感知和采取行动，从而暂停指定的若干时间步。

该方法被表述为一个具有字典序优化目标的强化学习问题，在保持任务完成时间的前提下，最大限度地减少了感知和决策的频率。在各种家庭环境和连续状态环境中的实验表明，该方法成功学习到了最优的等待行为，在不牺牲性能的情况下，等待时间超过了总任务时长的 50%。

Let it Cook: Learning to Wait in Sequential Decision Making

Summary

In many sequential decision-making tasks, agents constantly observe their environments and act at every timestep—much like a person continuously staring at a pot of coffee while it brews. However, constant monitoring is often unnecessary and resource-intensive.

This paper introduces a "waiting policy" framework that enables agents to intentionally forgo sensing and action-taking to pause for a designated number of timesteps. Formulated as a reinforcement learning problem with lexicographically ordered objectives, the approach minimizes sensing and decision-making frequency while preserving task completion time. Across various household and continuous environments, the proposed method successfully learns optimal waiting behaviors, achieving wait times of over 50% of the total task duration without sacrificing performance.

在许多序列决策任务中，智能体通常需要在每个时间步不断观察环境并采取行动，这类似于一个人在咖啡冲泡时一直盯着咖啡壶看。然而，这种持续的监控往往是不必要的且消耗大量资源。

本文引入了一种“等待策略”框架，使智能体能够有意放弃感知和采取行动，从而暂停指定的若干时间步。该方法被表述为一个具有字典序优化目标的强化学习问题，在保持任务完成时间的前提下，最大限度地减少了感知和决策的频率。在各种家庭环境和连续状态环境中的实验表明，该方法成功学习到了最优的等待行为，在不牺牲性能的情况下，等待时间超过了总任务时长的 50%。

Metadata & Publication Details

  • arXiv Identifier: arXiv:2608.11511 [cs.LG]
  • Subject Area: Machine Learning (cs.LG), Artificial Intelligence (cs.AI)
  • ACM Classification: I.2.6
  • Submitted Date: August 11, 2026
  • Authors:
  • Christopher Watson
  • Arjun Krishna
  • Dinesh Jayaraman
  • Rajeev Alur
  • Length: 25 pages
  • arXiv Identifier: arXiv:2608.11511 [cs.LG]
  • Subject Area: Machine Learning (cs.LG), Artificial Intelligence (cs.AI)
  • ACM Classification: I.2.6
  • Submitted Date: August 11, 2026
  • Authors:
  • Christopher Watson
  • Arjun Krishna
  • Dinesh Jayaraman
  • Rajeev Alur
  • Length: 25 pages

Abstract

In sequential decision making, an agent typically observes its environment and acts at every timestep. However, such active participation may not always be necessary; tasks such as brewing coffee include periods that are served equally well by letting the environment evolve without constant monitoring and control. During such periods, the agent could simply wait to conserve its resources, or redirect its attention to another task. We capitalize on these opportunities by training a "waiting policy" that decides where and how long to wait. This involves forgoing sensing to commit to a wait action, representing a deliberate pause for a set number of timesteps. We formalize "learning to wait" as minimizing the frequency of sensing and decision making without sacrificing task performance (e.g., the total amount of time to complete a task). To train a waiting policy, we propose an approach that employs reinforcement learning with lexicographically ordered objectives. In experiments across 4 discrete-state household tasks and 3 continuous-state environments, we show that our approach successfully learns waiting behaviors, and can adapt pre-trained policies to wait where appropriate. While different tasks permit different amounts of waiting without sacrificing task performance, our approach consistently finds solutions with significant waiting, sometimes waiting for over 50 percent of the task duration.

在序列决策中，智能体通常在每个时间步观察其环境并采取行动。然而，这种积极参与并不总是必要的；诸如冲泡咖啡之类的任务包含一些时间段，任由环境自行演化而无需持续监控和控制，同样可以达到很好的效果。在此期间，智能体可以通过简单地等待来节省资源，或者将注意力转移到其他任务上。我们通过训练一个决定在哪里以及等待多久的“等待策略”来利用这些机会。这涉及到放弃感知以执行等待动作，即在设定的时间步数内进行刻意的暂停。我们将“学习等待”形式化为：在不牺牲任务性能（例如完成任务的总时间）的前提下，最小化感知和决策的频率。为了训练等待策略，我们提出了一种采用具有字典序优化目标的强化学习方法。在 4 个离散状态家庭任务和 3 个连续状态环境的实验中，我们表明我们的方法成功学习到了等待行为，并且能够调整预训练策略以在适当的时候进行等待。虽然不同的任务允许在不牺牲任务性能的情况下进行不同程度的等待，但我们的方法始终能够找到包含显著等待时间的解决方案，有时等待时间甚至超过了任务总时长的 50%。

Access & Resources