跳转至

LoRA微调过程中公理化注意力模式带来的相关性涌现

文章背景与核心概要

本文研究了大语言模型（LLMs）在进行信息检索重排（特别是使用 RankLLaMA）的 LoRA 微调过程中，特定任务的相关性行为是如何获得的。通过一系列消融和注意力实验，作者证明了将 LoRA 注意力更新限制在一个紧凑的网络中间区域，就可以恢复全注意力适配一半以上的性能增益，而在该区域省略注意力更新则会带来独特的性能损害。此外，性能增益与可解释的公理化信息检索（IR）注意力模式（如词汇匹配、稀疏性敏感度以及文档-查询交互）的出现高度相关。

这些发现为神经排序器中的相关性适配提供了经验性的相关性解释，并为更高效的微调策略指明了方向。

license icon

Summary

This paper investigates how task-specific relevance behaviors are acquired in Large Language Models (LLMs) during LoRA fine-tuning for information retrieval reranking (specifically using RankLLaMA). Through a series of ablation and attention experiments, the authors demonstrate that restricting LoRA attention updates to a compact mid-network region recovers over half of the performance gains of full attention adaptation, while omitting it in this region is uniquely detrimental. Furthermore, performance gains strongly correlate with the emergence of interpretable, axiomatic information retrieval (IR) attention patterns—such as lexical matching, rarity sensitivity, and document-query interaction. These findings offer an empirical, correlational account of relevance adaptation in neural rankers and suggest pathways for more efficient fine-tuning strategies.

Metadata & Document Information

元数据与文档信息

  • arXiv Identifier: arXiv:2608.23338 [cs.CL]
  • DOI: 10.48550/arXiv.2608.23338
  • Primary Subject: Computation and Language (cs.CL)
  • Secondary Subjects: Artificial Intelligence (cs.AI), Information Retrieval (cs.IR)
  • Authors: Matthew Perlman, Atharva Nijasure, James Allan
  • Submission Date: August 24, 2026
  • Comments: Accepted to EMNLP 2026 Findings. 17 Pages, 25 Figures, 5 Tables.

Abstract

摘要

LoRA 微调是将大语言模型（LLM）适配到重排任务的标准方法，但目前尚不清楚特定任务的相关性行为是在网络的哪个部位学习到的，以及伴随这种学习发生了哪些注意力层面的变化。通过消融和注意力实验，我们确定了 RankLLaMA 中 LoRA 注意力更新提高性能的位置，以及这些增益是否与可解释的、面向相关性的注意力模式（如词汇匹配、稀疏性敏感度和查询-文档交互）相吻合。

LoRA fine-tuning is standard for adapting LLMs to reranking, but it remains unclear where in the network task-specific relevance behavior is learned and what attention-level changes accompany that learning. Through ablation and attention experiments, we identify where LoRA attention updates to RankLLaMA improve performance and whether those gains coincide with interpretable relevance-oriented attention patterns such as lexical matching, rarity sensitivity, and query-document interaction.

我们发现，在整个网络中给定经过 LoRA 微调的 MLP 的前提下，将 LoRA 注意力更新限制在一个紧凑的网络中间区域，足以恢复将 LoRA 应用于所有注意力层所获得性能的一半以上，并且在此区域省略注意力微调比网络中其他地方的损害更大应用。此外，我们表明，应用 LoRA 对性能影响最大的区域与微调增加对公理化 IR 特征注意力的区域高度重合。稀疏性敏感度、文档-查询交互以及几个组合特征与排序性能的提升高度相关。我们的结果支持了关于在 LoRA 微调过程中面向相关性的行为如何涌现的可解释、相关性解释，并指向了改进重排器适配的策略。

We find that given LoRA fine-tuned MLPs throughout the network, restricting LoRA attention updates to a compact mid-network region is sufficient for recovering over half of the performance gained by applying LoRA to all attention layers, and that omitting attention fine-tuning in this region hurts performance more than elsewhere in the network. Additionally, we show that regions where applying LoRA affects performance the most overlap with regions where fine-tuning increased attention to axiomatic IR features. Rarity sensitivity, document-query interaction, and several compositional features are highly correlated with gains in ranking performance. Our results support an interpretable, correlational account of how relevance-oriented behavior emerges during LoRA fine-tuning and point toward improved strategies for adapting rerankers.

Access & Resources

访问与资源