跳转至

LoCA：视觉基础模型的空间感知低秩卷积自适应

文章背景与核心概要

预训练视觉基础模型（VFMs）为各种下游任务提供了强大的视觉表征，但全量微调成本极高且容易发生灾难性遗忘。低秩自适应（LoRA）等参数高效微调（PEFT）技术已变得十分流行，但它们通常是为 2D Transformer 自注意力矩阵设计的。当应用于本质上耦合了空间和通道信息的 4D 卷积核时，将其强行展平成单一的 2D 矩阵会破坏其空间拓扑结构。

为了解决这种空间与通道的纠缠问题，作者提出了低秩卷积自适应（LoCA）这一具备卷积感知能力的 PEFT 框架，它解耦了通道与空间自适应。LoCA 利用低秩通道自适应来进行稠密的跨通道混合，同时通过奇异值分解（SVD）精炼预训练的空间基，成功保留了预训练的空间先验，并在细粒度分类、域泛化语义分割以及生成式基准测试中取得了竞争力的或最先进的成果。

LoCA: Spatially-Aware Low-Rank Convolutional Adaptation of Vision Foundation Models

Summary

Pre-trained Vision Foundation Models (VFMs) provide powerful visual representations for various downstream tasks, but full fine-tuning is prohibitively expensive and prone to catastrophic forgetting. Parameter-Efficient Fine-Tuning (PEFT) techniques like Low-Rank Adaptation (LoRA) have become popular, but they are typically designed for 2D transformer self-attention matrices. When applied to 4D convolutional kernels—which inherently couple spatial and channel information—forcing them into a monolithic 2D matrix disrupts their spatial topology.

To resolve this spatial-channel entanglement, the authors propose Low-Rank Convolutional Adaptation (LoCA), a convolution-aware PEFT framework that decouples channel and spatial adaptation. LoCA uses low-rank channel adaptation for dense cross-channel mixing while refining pre-trained spatial bases extracted via Singular Value Decomposition (SVD), successfully preserving pre-trained spatial priors and achieving competitive or state-of-the-art results across fine-grained classification, domain-generalized semantic segmentation, and generative benchmarks.

论文元数据 (Paper Metadata)

  • arXiv ID: arXiv:2607.06918 [cs.CV]
  • 研究主题 (Subjects): 计算机视觉与模式识别 (cs.CV)；人工智能 (cs.AI)；机器学习 (cs.LG)
  • 作者 (Authors): Sojung An, Junha Lee, Sujeong You, Nam Ik Cho, Donghyun Kim
  • 会议状态 (Conference Status): 已被 ECCV 2026 录用
  • 提交时间线 (Submission Timeline):
  • 2026年7月8日提交 (v1)
  • 2026年8月7日最后修订 (v2)

摘要 (Abstract)

Pre-trained Vision Foundation Models (VFMs) provide strong visual representations for diverse downstream tasks. The key challenge of VFM adaptation stems from the prohibitive costs of full fine-tuning and catastrophic forgetting. To address this, Low-Rank Adaptation (LoRA) has emerged as the prevailing paradigm for Parameter-Efficient Fine-Tuning (PEFT). However, LoRA is typically designed for transformer self-attention layers parameterized by 2D matrices. Since convolutional kernels inherently couple spatial and channel information within a 4D tensor, forcing them into a monolithic 2D matrix disrupts the inherent spatial topology. In this paper, we propose Low-Rank Convolutional Adaptation (LoCA), a convolution-aware PEFT framework that addresses spatial-channel entanglement by decoupling channel and spatial adaptation. LoCA introduces a low-rank channel adaptation for dense cross-channel mixing and refines spatial bases extracted from pre-trained kernels via Singular Value Decomposition (SVD). Experimental results show that LoCA preserves pre-trained spatial priors and achieves competitive or state-of-the-art performance across fine-grained classification, domain-generalized semantic segmentation, and generative benchmarks.