跳转至

检索、打分与解码方式对基于大语言模型的对话式推荐系统性能与稳定性的影响

文章背景与核心概要

本文探讨了大语言模型（LLM）如何在基于 ReDial 电影推荐基准的对话式推荐系统中充当重排器（rerankers）。作者通过实验证明，测得的系统性能对管线配置极其敏感，具体包括候选生成（语义检索池与协同过滤池）、候选池大小、打分策略以及解码温度。值得注意的是，对于顶级重排器而言，将候选集从语义检索切换为协同过滤可使 NDCG@10 提升超过 50%，且相比于严格的候选感知打分，无约束的零样本（zero-shot）生成会导致性能指标出现巨大虚高。这些发现表明，研究人员必须将这些实验配置视为强制报告字段，而不是无关紧要的实现细节。

This paper investigates how large language models (LLMs) function as rerankers in conversational recommender systems using the ReDial movie recommendation benchmark. The authors demonstrate that measured performance is exceptionally sensitive to pipeline configurations—specifically candidate generation (semantic vs. collaborative-filtering pools), candidate-pool size, scoring policies, and decoding temperatures. Notably, switching from semantic to collaborative-filtering candidates raises NDCG@10 by over 50% for top-tier rerankers, and unconstrained zero-shot generation yields vastly inflated performance compared to strict candidate-aware scoring. The findings suggest that researchers must treat these experimental configurations as mandatory reporting fields rather than trivial implementation details.

元数据与参考信息

  • arXiv ID: arXiv:2609.00086 [cs.CL]
  • 学科分类: 计算与语言 (cs.CL)；人工智能 (cs.AI)
  • 作者: Ante Kapetanovic, Tomislav Duricic, Andro Mercep, Emanuel Lacic
  • 提交时间: 2026年8月31日
  • 会议: 将于第35届ACM信息与知识管理国际会议（CIKM '26）论文集发表
  • 相关 DOI: 10.1145/3799682.3840066

Metadata & Reference Information

  • arXiv ID: arXiv:2609.00086 [cs.CL]
  • Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
  • Authors: Ante Kapetanovic, Tomislav Duricic, Andro Mercep, Emanuel Lacic
  • Submitted On: 31 August 2026
  • Conference: To appear in the proceedings of the 35th ACM International Conference on Information and Knowledge Management (CIKM '26)
  • Related DOI: 10.1145/3799682.3840066

摘要

大语言模型（LLM）在对话式推荐系统中作为重排器的应用日益广泛，然而其测得的性能增益在很大程度上取决于检索和推理协议。在 ReDial 对话式电影推荐基准上，我们在共享的“检索-后重排”（retrieve-then-rerank）管线中，将闭源、开源权重以及微调的 LLM 重排器与协同过滤和序列基线进行了对比。我们改变了候选池大小、第一阶段检索器以及解码温度。

在使用共享的语义 Top-250 候选池和严格的候选感知打分协议下，表现最好的闭源重排器实现了 0.1497 的 NDCG@10，而最强的非 LLM 基线得分为 0.0939。同一重排器在零样本生成下的 NDCG@10 达到了 0.2925，这表明无约束打分所呈现的优势明显大于匹配候选池的评估结果。在这一协议下，没有任何受评估的开源权重 LLM 性能超过经过微调的浅层自编码器基线。

对于最强的闭源和开源权重重排器，将候选集从语义检索切换为协同过滤可使 NDCG@10 提升 50% 以上，这表明测得的重排器性能对候选生成方式高度敏感。对于表现最好的闭源重排器，将解码温度从 0 提高到 1.0 会使 Top-10 杰卡德距离（Jaccard distance）从 0.0900 增加到 0.1240，而平均 NDCG@10 的变化微乎其微，相比之下，较弱的 LLM 则表现出更大的性能退化。这些 ReDial 的实验结果支持将候选生成、候选池大小、打分策略以及解码配置作为必需的报告字段，而非单纯的实现细节。

Abstract

Large language models (LLMs) are increasingly used as rerankers in conversational recommender systems, yet measured gains depend strongly on the retrieval and inference protocol. On the ReDial conversational movie recommendation benchmark, we compare proprietary, open-weight, and fine-tuned LLM rerankers with collaborative-filtering and sequential baselines in a shared retrieve-then-rerank pipeline. We vary candidate-pool size, first-stage retriever, and decoding temperature.

With a shared semantic top-250 candidate pool and strict candidate-aware scoring, the best proprietary reranker reaches NDCG@10 of 0.1497, compared with 0.0939 for the strongest non-LLM baseline. The same reranker reaches 0.2925 in zero-shot generation, showing that unconstrained scoring can yield a much larger apparent advantage than matched-pool evaluation. No evaluated open-weight LLM outperforms the tuned shallow autoencoder baseline under this protocol.

For the strongest proprietary and open-weight rerankers, switching from semantic to collaborative-filtering candidates raises NDCG@10 by more than 50%, showing that measured reranker performance is highly sensitive to candidate generation. For the best proprietary reranker, raising temperature from 0 to 1.0 increases top-10 Jaccard distance from 0.0900 to 0.1240 while mean NDCG@10 changes negligibly, whereas weaker LLMs show larger degradation. These ReDial results support treating candidate generation, candidate-pool size, scoring policy, and decoding configuration as required reporting fields rather than implementation details.