跳转至

文章背景与核心概要

在基于模型的强化学习（Model-based Reinforcement Learning）中，世界模型是核心支柱之一，以往通常根据它们所预测的具体变量（如奖励、观测值、潜变量或信息状态）来进行分类。本文则引入了一种更为基础的分类维度：它们所建模的通道（channel）究竟是什么。

通过运用计算力学（computational mechanics）的工具，作者深入分析了三种不同的建模情况：环境通道、智能体通道以及实现的联合过程。核心贡献包括定义了规范预测模型、构建了由闭环耦合诱导的支持受限（support-restricted）模型、证明了结构分解定理，并通过POMDP/控制器示例展示了如何实现显著的复杂度约简。

World Models of Environment, Agent, and Joint Agent-Environment Systems

Authors: Manuel Baltieri, Filippo Torresan, Yivan Zhang, Alexander Boyd, Fernando E. Rosas
Primary Subject: Artificial Intelligence (cs.AI)
Secondary Subject: Machine Learning (cs.LG)
arXiv ID: 2608.20401
Submitted: 23 July 2026

摘要

世界模型是基于模型的强化学习的基石，通常根据它们预测的具体变量（例如奖励、观测值、潜在状态或信息状态）进行分类。本文引入了一个更基础的区分：它们对哪个通道进行建模。

利用计算力学的工具，作者分析了三种不同的建模情况： 1. 环境通道 (\(O_{:} \mid A_{:}\)) 2. 智能体通道 (\(A_{:} \mid O_{:}\)) 3. 实现的联合过程 (\((A, O)_{:}\))，等价地视为无输入的通道

World models are a fundamental pillar of model-based reinforcement learning, typically categorized by the specific variables they predict (e.g., rewards, observations, latent or information states). This paper introduces a more foundational distinction: which channel they model.

Using tools from computational mechanics, the authors analyze three distinct modeling cases: 1. The environment channel (\(O_{:} \mid A_{:}\)) 2. The agent channel (\(A_{:} \mid O_{:}\)) 3. The realized joint process (\((A, O)_{:}\)), treated equivalently as a channel with no inputs

核心贡献与发现

  • 规范预测模型： 规范环境模型恢复了标准的预测状态表示，同时通过 \(\epsilon\)-转导器（\(\epsilon\)-transducers）或 \(\epsilon\)-机器（\(\epsilon\)-machines）类似地定义了智能体和联合系统的规范模型。
  • 支持受限建模： 作者构建了由闭环耦合诱导的规范支持受限环境模型和智能体模型，其中预测等价性跨越了由实现交互支持的后续状态。
  • 结构分解： 一项核心结构结果表明，规范支持受限环境状态直接通过规范联合因果状态进行分解，其转移结构直接继承自联合模型（智能体侧的构造呈对偶性）。
  • 复杂度约简： 一个 POMDP/控制器示例表明，虽然不受限制的环境模型可以拥有无限多个状态，但由闭环耦合诱导的规范支持受限模型仍然是有限的。
  • Canonical Predictive Models: Canonical environment models recover standard predictive state representations, while analogously defining canonical models for both agents and joint systems via \(\epsilon\)-transducers or \(\epsilon\)-machines.
  • Support-Restricted Modeling: The authors construct canonical support-restricted environment and agent models induced by closed-loop coupling, where predictive equivalences span continuations supported by the realized interaction.
  • Structural Factorization: A core structural result demonstrates that canonical support-restricted environment states factor directly through canonical joint causal states, with their transition structures inherited straight from the joint model (the agent-side construction operates dually).
  • Complexity Reduction: A POMDP/controller example reveals that while an unrestricted environment model can possess infinitely many states, the canonical support-restricted model induced by the closed-loop coupling remains finite.

访问与资源

引用工具与参考文献