跳转至

文章背景与核心概要

大语言模型（LLMs）在生成式排序领域展现出了先进的性能，但传统的自回归解码方式需要为每一个生成的Token进行顺序前向传递，这成为了实时应用中的重大性能瓶颈。本文引入了一种名为 SPD（单次前向传递，Single Forward Pass） 的专有格式解码策略，专为生成式重排而设计。

SPD 摒弃了标准的从左到右生成方式，转而通过一个轻量级的自注意力头，直接从大模型的预填充（prefill）隐藏状态中提取一个 \(N \times K\) 的物品-位置得分矩阵。然后，它通过匈牙利算法在 \(O(1)\) 次前向传递中求解解码问题，构建出有效且最优分配的排列组合。结合基于 LoRA 的微调和自回归蒸馏技术，SPD 将端到端推理延迟降低至 28 毫秒（实现了 64倍的加速），同时完美保留了教师模型的排序质量。

SPD: Single Pass Decoding for Generative Reranking

Summary

Large Language Models (LLMs) provide state-of-the-art generative ranking quality, but traditional autoregressive decoding requires a sequential forward pass for every emitted token, creating a major performance bottleneck for real-time applications.

This paper introduces SPD (Single Forward Pass), a format-specialized decoding strategy designed for generative reranking. Instead of standard left-to-right generation, SPD extracts an \(N \times K\) item-position score matrix directly from the LLM's prefill hidden states using a lightweight self-attention head. It then solves the decoding problem in \(O(1)\) forward passes via the Hungarian algorithm, producing a valid, optimally assigned permutation by construction. Combined with LoRA-based fine-tuning and autoregressive distillation, SPD achieves an end-to-end inference latency of 28 ms (a 64× speed-up) while preserving the ranking quality of the teacher model.

Paper Metadata

  • arXiv ID: arXiv:2609.01807 [cs.LG]
  • Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Information Retrieval (cs.IR)
  • Authors:
  • Emil Laftchiev
  • Prachi Agrawal
  • Moe Kayali
  • Bixing Yan
  • Qi Xu
  • Zijie Lei
  • Chen Qiu
  • Zhi Hua
  • Ke Li
  • Luke Simon
  • Dates:
  • Submitted: 1 Sep 2026
  • Last Revised: 4 Sep 2026 (v2)
  • Length: 10 pages

Abstract

Large language models (LLMs) achieve state-of-the-art generative ranking quality, but the ranking they produce must be decoded, and autoregressive decoding spends one sequential forward pass per emitted token. We observe that the only tokens a ranker must emit are the \(N\) ordinal values naming the items in ranked order, and that this narrow, permutation-structured output format admits decoding strategies which are much more efficient than left-to-right generation. We introduce SPD (Single Forward Pass), a format-specialized decoding strategy that decodes all \(N\) ordinals in \(O(1)\) forward passes. SPD reads an \(N \times K\) item-position score matrix off the LLM's prefill hidden states with a lightweight self-attention head, then decodes the ordinals as the optimal bipartite assignment of that matrix via the Hungarian algorithm, yielding a valid permutation by construction rather than by repair. Through a systematic study of training signals and backbone adaptation, we show that LoRA-based fine-tuning combined with auto-regressive LLM ranking distillation reaches 28 ms end-to-end inference, a speed-up of 64x while maintaining ranking quality on par with the teacher. We provide a complete ablation decomposing the contributions of architecture, training signal, and backbone adaptation. Our framework connects generative ranking to combinatorial optimization, opening a path toward other \(O(1)\)-decode mechanisms for real-time ranking.

Access & Resources