跳转至

TaRA：感知训练过程的低阶适应初始化方法

文章背景与核心概要

低阶适应（LoRA）作为参数高效微调（PEFT）的行业标准技术，其性能往往受到低阶因子初始化方式的显著制约。现有的初始化方法虽然尝试利用预训练权重、激活值或梯度信息，但普遍未能直接将全秩模型的实际训练动态纳入考量。

本文介绍了一种名为“感知训练过程的低阶适应初始化”（Training-aware Low-Rank Adaptation Initialization，简称 TaRA）的数学推导初始化方法。TaRA 能够确保由低阶因子生成的梯度高度逼近全秩权重矩阵的梯度，在训练初期即以极低的计算开销显著提升梯度保真度。在各项具有挑战性的微调基准测试中，TaRA 的表现持续超越当前最先进的 PEFT 方法。

TaRA：感知训练过程的低阶适应初始化方法

摘要 (Summary)

Low-Rank Adaptation (LoRA) is widely used for parameter-efficient fine-tuning (PEFT), but its performance is often limited by how the low-rank factors are initialized. While previous methods leverage pretrained weights, activations, or gradients, they fail to directly account for the actual training dynamics of the full-rank model.

This paper introduces Training-aware Low-Rank Adaptation Initialization (TaRA), a mathematically derived initialization method that ensures the gradients generated by low-rank factors closely approximate those of the full-rank weight matrix. TaRA improves gradient fidelity right from the start of training with negligible computational overhead, consistently outperforming state-of-the-art PEFT methods across challenging fine-tuning benchmarks.

低阶适应（LoRA）广泛应用于参数高效微调（PEFT），但其性能往往受限于低阶因子的初始化方式。虽然先前的方法利用了预训练权重、激活值或梯度，但它们未能直接将全秩模型的实际训练动态考虑在内。本文引入了感知训练过程的低阶适应初始化（TaRA），这是一种通过数学推导得出的初始化方法，可确保低阶因子生成的梯度紧密逼近全秩权重矩阵的梯度。TaRA 在训练开始时即可提高梯度保真度，且计算开销微乎其微，在各项具有挑战性的微调基准测试中始终表现优于最先进的 PEFT 方法。

论文元数据 (Paper Metadata)

  • 标题 (Title): TaRA: Training-Aware Low-Rank Adaptation Initialization
  • 作者 (Authors): Taehyeon Kim, Eunhyeok Park
  • 主要学科 (Primary Subject): 计算与语言 (cs.CL)
  • 次要学科 (Secondary Subjects): 人工智能 (cs.AI)，机器学习 (cs.LG)
  • 提交日期 (Submission Date): 2026年9月2日
  • 会议录用 (Conference Acceptance): 已被 EMNLP 2026 主会接受
  • 标识符 (Identifiers):
  • arXiv: 2609.02639
  • DOI: 10.48550/arXiv.2609.02639

摘要正文 (Abstract)

Low-Rank Adaptation (LoRA) has become a de facto standard for parameter-efficient fine-tuning (PEFT), yet its performance is highly sensitive to initialization due to the information bottleneck imposed by low-rank decomposition. Existing approaches attempt to construct high-quality LoRA initializations by exploiting principal components of pretrained weights, activations, or gradients. However, these methods do not directly account for the training dynamics of the full-rank model. In this paper, we propose Training-aware Low-Rank Adaptation Initialization (TaRA), a method that initializes LoRA such that the gradients induced by the low-rank factors closely approximate the gradient of the corresponding full-rank weight matrix. Derived from a mathematical formulation, TaRA improves gradient fidelity at the start of training while introducing negligible computational overhead. Across diverse and challenging fine-tuning tasks, TaRA consistently outperforms prior state-of-the-art methods, establishing a simple, robust, and scalable solution for effective LoRA initialization.

低阶适应（LoRA）已成为参数高效微调（PEFT）的事实标准，然而由于低阶分解带来的信息瓶颈，其性能对初始化高度敏感。现有方法试图通过利用预训练权重、激活值或梯度的主要成分来构建高质量的 LoRA 初始化。然而，这些方法并未直接考虑全秩模型的训练动态。在本文中，我们提出了感知训练过程的低阶适应初始化（TaRA），该方法对 LoRA 进行初始化，使得由低阶因子引出的梯度能够密切近似对应全秩权重矩阵的梯度。源自数学公式推导的 TaRA 在训练开始时提升了梯度保真度，同时引入的计算开销微不足道。在多样化且具有挑战性的微调任务中，TaRA 始终优于先前最先进的方法，为实现有效的 LoRA 初始化建立了一个简单、强大且可扩展的解决方案。