跳转至

文章背景与核心概要

多模态环境下的序列推荐（SR）传统上依赖于小型、冻结的预训练编码器。虽然这种方法效率较高，但它限制了语义容量，并阻碍了协同过滤（CF）信号适当地融入物品表示中。受大语言模型（LLM）作为高容量嵌入器成功经验的启发，本文探讨了将视觉-语言模型（VLM）用作具备协同过滤感知能力的多模态嵌入器。然而，作者发现标准的对比式监督微调（SFT）——通常用于适配 VLM 并注入 CF 信号——往往会引发模态崩溃：优化过程被单一模态主导，而另一个模态则性能退化，从而损害整体推荐性能。

为了克服这一问题，作者推出了 VLM2Rec，这是一个旨在通过以下方式促进平衡模态利用的新颖框架：1. 弱模态惩罚对比学习：缓解优化过程中的梯度不平衡。2. 跨模态关系拓扑正则化：保持不同模态之间的几何一致性。实验表明，VLM2Rec 在各种推荐场景下的准确性和鲁棒性方面均持续优于强基线模型。

VLM2Rec: Resolving Modality Collapse in Vision-Language Model Embedders for Multimodal Sequential Recommendation

arXiv: [2603.17450 [cs.IR]]
DOI: 10.48550/arXiv.2603.17450
Authors: Junyoung Kim, Woojoo Kim, Wonbin Kweon, Jaehyung Lim, Dongha Kim, Hwanjo Yu
Submission History: Submitted on 18 Mar 2026; last revised 12 Aug 2026.

arXiv: [2603.17450 [cs.IR]]
DOI: 10.48550/arXiv.2603.17450
Authors: Junyoung Kim, Woojoo Kim, Wonbin Kweon, Jaehyung Lim, Dongha Kim, Hwanjo Yu
Submission History: Submitted on 18 Mar 2026; last revised 12 Aug 2026.

📋 Summary

Sequential Recommendation (SR) in multimodal environments has traditionally depended on small, frozen pretrained encoders. While efficient, this approach restricts semantic capacity and prevents Collaborative Filtering (CF) signals from properly integrating into item representations.

Inspired by the success of Large Language Models (LLMs) as high-capacity embedders, this paper explores Vision-Language Models (VLMs) as CF-aware multimodal embedders. However, the authors discover that standard contrastive Supervised Fine-Tuning (SFT)—commonly used to adapt VLMs and inject CF signals—often triggers modality collapse: optimization becomes dominated by one modality while the other degrades, harming overall recommendation performance.

To overcome this, the authors introduce VLM2Rec, a novel framework designed to promote balanced modality utilization via: 1. Weak-modality Penalized Contrastive Learning: Mitigates gradient imbalance during optimization. 2. Cross-modal Relational Topology Regularization: Preserves geometric consistency between modalities.

Experiments demonstrate that VLM2Rec consistently outperforms strong baselines in both accuracy and robustness across diverse recommendation scenarios.

📋 Summary

Sequential Recommendation (SR) in multimodal environments has traditionally depended on small, frozen pretrained encoders. While efficient, this approach restricts semantic capacity and prevents Collaborative Filtering (CF) signals from properly integrating into item representations.

Inspired by the success of Large Language Models (LLMs) as high-capacity embedders, this paper explores Vision-Language Models (VLMs) as CF-aware multimodal embedders. However, the authors discover that standard contrastive Supervised Fine-Tuning (SFT)—commonly used to adapt VLMs and inject CF signals—often triggers modality collapse: optimization becomes dominated by one modality while the other degrades, harming overall recommendation performance.

To overcome this, the authors introduce VLM2Rec, a novel framework designed to promote balanced modality utilization via: 1. Weak-modality Penalized Contrastive Learning: Mitigates gradient imbalance during optimization. 2. Cross-modal Relational Topology Regularization: Preserves geometric consistency between modalities.

Experiments demonstrate that VLM2Rec consistently outperforms strong baselines in both accuracy and robustness across diverse recommendation scenarios.

📊 License & Metadata

📊 License & Metadata