跳转至

CM-MAE：面向视觉-无线应用物理机理引导的跨模态自监督学习框架

文章背景与核心概要

同步的摄像头与无线测量通过截然不同的感觉通道观测着完全相同的物理环境。然而，当环境条件发生变化（如视角、车流量、光照或传播几何结构发生改变）时，在某一特定部署中学习到的表征往往会失效。

为了解决这一跨场景泛化瓶颈，本文引入了 CM-MAE——一个物理机理引导的自监督视觉-无线预训练框架。CM-MAE 运行于来自 DeepSense 6G 的真实世界数据之上，仅使用 RGB 帧和测量的 64 射束接收功率向量（无需射线追踪路径、标定深度或射束索引标签），利用软对比对齐损失（soft contrastive alignment loss）和掩码联合解码器（masked joint decoder），实现了学习表征在不同场景间的鲁棒迁移。

执行摘要 / Executive Summary

Synchronized camera and wireless measurements observe the exact same physical environment through distinct sensory channels. However, representations learned in one deployment frequently fail when conditions shift (such as viewpoint, traffic flow, illumination, or propagation geometry).

同步的摄像头与无线测量通过截然不同的感觉通道观测着完全相同的物理环境。然而，当环境条件发生变化（如视角、车流量、光照或传播几何结构发生改变）时，在某一特定部署中学习到的表征往往会失效。

To resolve this cross-scenario generalization bottleneck, this paper introduces CM-MAE, a physics-guided, self-supervised vision-wireless pretraining framework. Operating on real-world data from DeepSense 6G—using solely RGB frames and measured 64-beam received-power vectors (without requiring ray-traced paths, calibrated depth, or beam-index labels)—CM-MAE leverages a soft contrastive alignment loss and a masked joint decoder to robustly transfer learned representations across scenarios.

为了解决这一跨场景泛化瓶颈，本文引入了 CM-MAE——一个物理机理引导的自监督视觉-无线预训练框架。CM-MAE 运行于来自 DeepSense 6G 的真实世界数据之上，仅使用 RGB 帧和测量的 64 射束接收功率向量（无需射线追踪路径、标定深度或射束索引标签），利用软对比对齐损失和掩码联合解码器，实现了学习表征在不同场景间的鲁棒迁移。

核心框架组件 / Key Framework Components

  1. Soft Contrastive Alignment Loss:
    Instead of rigidly treating only the exact synchronized image-wireless pair as a positive match (which forces nonidentical samples with similar directional responses apart as false negatives), this loss constructs a target distribution based on the similarities between measured beam-power profiles.
  1. 软对比对齐损失：
    该损失没有死板地将完全同步的图像-无线对视为唯一的正样本匹配（这会迫使具有相似定向响应的不同样本被作为假阴性推开），而是根据测量的射束功率轮廓之间的相似性构建了目标分布。
  1. Masked Joint Decoder:
    Provides a complementary local objective by reconstructing hidden visual patches and wireless angular clusters even under modality dropout conditions.
  1. 掩码联合解码器：
    即使在模态丢失（dropout）的条件下，也能通过重建隐藏的视觉补丁和无线角度聚类，提供互补的局部优化目标。
  1. Differential-Rate Fine-Tuning Rule:
    Post-pretraining, this mechanism enables a newly introduced fusion head to adapt rapidly while the foundational encoders adjust at a much slower pace.
  1. 差分速率微调规则：
    在预训练之后，该机制使新引入的融合头能够快速适应，而基础编码器的调整速度则慢得多。

实证结果 / Empirical Results

Evaluated under a rigorous sequence-disjoint DeepSense 6G protocol, the framework demonstrates significant performance gains:

在严格的序列不相交 DeepSense 6G 协议下进行评估，该框架展现出了显著的性能提升：

  • Linear-Probe Transfer: Integrating the soft alignment loss improves the matched linear-probe transfer average from 24.88% to 29.49%.
  • Unseen Scenarios (Scenarios 6–8):
  • Mild fusion fine-tuning achieves 77.38% Top-1 accuracy.
  • Optional transductive normalization adaptation further elevates accuracy to 78.69%.
  • 线性探针迁移： 引入软对齐损失将匹配的线性探针迁移平均准确率从 24.88% 提升至 29.49%。
  • 未见场景（场景 6–8）：
  • 轻度融合微调实现了 77.38% 的 Top-1 准确率。
  • 可选的转导归一化自适应（transductive normalization adaptation）将准确率进一步提升至 78.69%。

Note: Because the fusion setting relies on contemporaneous 64-beam power vectors during inference, these results serve as representation-transfer diagnostics rather than claims for proactive beam prediction or reduced sweeping.

注：由于融合设置在推理过程中依赖于同时期的 64 射束功率向量，因此这些结果主要作为表征迁移的诊断指标，而非声称实现了主动射束预测或减少了扫描开销。

访问完整论文 / Access Full Paper