跳转至

文章背景与核心概要

在人机交互和可扩展数据集构建中，根据连续环境中的第一视角RGB视频生成导航指令是一项关键且极具挑战性的任务。现有的指令生成器通常依赖于离散的视点图（viewpoint graphs）和全景观测，在这些场景中轨迹结构是显式定义的。然而，在连续环境中，智能体只能接收到稠密的RGB视频流，这使得轨迹线索极难提取。

为了填补这一空白，本文作者推出了 VTInstructor，这是首个专为连续环境设计的视觉-语言导航（VLN）指令生成框架，它无需导航图、预建地图或场景重建。VTInstructor 引入了一种创新的视觉轨迹提示（Visual Trajectory Prompting）范式： 1. EDTC：将长RGB轨迹压缩为对导航至关重要的关键帧。 2. VTP：在这些锚点关键帧上叠加路径、转向和目标线索。 3. VTMod：将生成的轨迹信号直接注入视觉编码器。 4. VT-GRPO：在训练期间对这种空间注入进行校准。

本文的主要实验结果表明，VTInstructor 在挑战性的 R2R-CE 和 RxR-CE Val Unseen 数据集的所有自然语言生成（NLG）指标上均创下了全新基准，并且由其生成的指令能够显著提升下游导航任务的性能与数据增强效果。

VTInstructor: Visual Trajectory Prompting for Navigation Instruction Generation in Continuous Environments

VTInstructor: Visual Trajectory Prompting for Navigation Instruction Generation in Continuous Environments

arXiv: [arXiv:2608.15284 [cs.RO]]
Authors: Haolin Yang, Yuxing Long, Zihan Yang, Hao Dong
Conference: Accepted by ACM MM 2026
Submitted: 15 August 2026

arXiv: [arXiv:2608.15284 [cs.RO]]
Authors: Haolin Yang, Yuxing Long, Zihan Yang, Hao Dong
Conference: Accepted by ACM MM 2026
Submitted: 15 August 2026

📋 Executive Summary

📋 Executive Summary

Generating navigation instructions from ego-centric RGB video in continuous environments is a crucial yet challenging problem for human-robot interaction and scalable dataset construction. Existing instruction generators typically rely on discrete viewpoint graphs and panoramic observations where trajectory structures are explicitly defined. However, in continuous environments, agents receive only dense RGB video streams, making trajectory cues difficult to extract.

Generating navigation instructions from ego-centric RGB video in continuous environments is a crucial yet challenging problem for human-robot interaction and scalable dataset construction. Existing instruction generators typically rely on discrete viewpoint graphs and panoramic observations where trajectory structures are explicitly defined. However, in continuous environments, agents receive only dense RGB video streams, making trajectory cues difficult to extract.

To address this gap, the authors introduce VTInstructor, the first Vision-Language Navigation (VLN) instruction generation framework specifically designed for continuous environments without requiring a navigation graph, pre-built map, or scene reconstruction. VTInstructor leverages a novel Visual Trajectory Prompting paradigm: 1. EDTC: Condenses long RGB trajectories into navigation-critical keyframes. 2. VTP: Overlays path, turn, and goal cues onto these anchor keyframes. 3. VTMod: Injects the resulting trajectory signals directly into the visual encoder. 4. VT-GRPO: Calibrates this spatial injection during training.

To address this gap, the authors introduce VTInstructor, the first Vision-Language Navigation (VLN) instruction generation framework specifically designed for continuous environments without requiring a navigation graph, pre-built map, or scene reconstruction. VTInstructor leverages a novel Visual Trajectory Prompting paradigm: 1. EDTC: Condenses long RGB trajectories into navigation-critical keyframes. 2. VTP: Overlays path, turn, and goal cues onto these anchor keyframes. 3. VTMod: Injects the resulting trajectory signals directly into the visual encoder. 4. VT-GRPO: Calibrates this spatial injection during training.

Key Results

  • State-of-the-Art Performance: Sets new benchmarks across all Natural Language Generation (NLG) metrics on the challenging R2R-CE and RxR-CE Val Unseen datasets, outperforming the strongest baseline by +0.357 CIDEr and +0.109 CIDEr, respectively.
  • Superior Follower Success: Instructions generated by VTInstructor raise a frozen follower's success rate to 63.3% (a +14.7 percentage-point gain over the best competing instruction source).
  • Data Augmentation: Provides consistent data augmentation gains of +3 Success Rate (SR) points on downstream navigation tasks.

Key Results

  • State-of-the-Art Performance: Sets new benchmarks across all Natural Language Generation (NLG) metrics on the challenging R2R-CE and RxR-CE Val Unseen datasets, outperforming the strongest baseline by +0.357 CIDEr and +0.109 CIDEr, respectively.
  • Superior Follower Success: Instructions generated by VTInstructor raise a frozen follower's success rate to 63.3% (a +14.7 percentage-point gain over the best competing instruction source).
  • Data Augmentation: Provides consistent data augmentation gains of +3 Success Rate (SR) points on downstream navigation tasks.

🛠️ Methodology & Framework

🛠️ Methodology & Framework

Unlike traditional approaches relying on structured viewpoint graphs, VTInstructor processes raw, dense RGB video streams through four core components:

Unlike traditional approaches relying on structured viewpoint graphs, VTInstructor processes raw, dense RGB video streams through four core components:

  • Explicit Dense Trajectory Condensation (EDTC): Filters and condenses long, dense video streams into a concise set of navigation-critical keyframes.
  • Visual Trajectory Prompting (VTP): Visually augments keyframes by overlaying essential geometric path guides, turn indicators, and goal markers.
  • Visual Trajectory Modulation (VTMod): Modulates and injects these visual trajectory signals straight into the model's visual encoder.
  • Visual Trajectory Group Relative Policy Optimization (VT-GRPO): Calibrates the spatial injection process during training to maximize instruction alignment and utility.
  • Explicit Dense Trajectory Condensation (EDTC): Filters and condenses long, dense video streams into a concise set of navigation-critical keyframes.
  • Visual Trajectory Prompting (VTP): Visually augments keyframes by overlaying essential geometric path guides, turn indicators, and goal markers.
  • Visual Trajectory Modulation (VTMod): Modulates and injects these visual trajectory signals straight into the model's visual encoder.
  • Visual Trajectory Group Relative Policy Optimization (VT-GRPO): Calibrates the spatial injection process during training to maximize instruction alignment and utility.