跳转至

文章背景与核心概要

检索增强生成（RAG）系统通过从外部知识库中提取相关上下文，有效缓解了大语言模型（LLM）的幻觉问题。然而，随着知识库逐渐演变为包含语音和文本等多种模态，处理这种异构多模态数据源面临着由于前期研究匮乏和缺乏专业训练数据而带来的巨大挑战。

为了解决这一问题，本文作者提出了 STeReO（语音与文本重排编排器，Speech and Text Reranking Orchestrator），这是一种专为聚合不同语音和文本检索器数据库而设计的新型重排器。该论文引入了一个由查询、混合模态证据及相关性排序组成的定制数据集。在单模态和混合模态场景下的评估均表明，STeReO 能够成功识别出最相关的证据，从而显著提升下游问答任务的性能。

A Reranker for Orchestrating Heterogeneous Speech and Text Retrievers

Summary

Retrieval-Augmented Generation (RAG) systems help mitigate hallucinations in Large Language Models (LLMs) by pulling relevant context from external knowledge databases. As these databases expand to incorporate multiple modalities—such as both speech and text—handling heterogeneous multi-modal sources presents a significant challenge due to limited prior research and a lack of specialized training data.

To solve this, the authors propose STeReO (Speech and Text Reranking Orchestrator), a novel reranker designed to aggregate disparate speech and text retriever databases. The paper introduces a custom-curated dataset consisting of queries, mixed-modality evidence, and relevance rankings. Evaluations across both single-modality and mixed-modality scenarios demonstrate that STeReO successfully identifies the most relevant evidence, substantially improving downstream question-answering performance.

Paper Metadata / 论文元数据

  • arXiv ID: arXiv:2608.26194 [cs.CL]
  • Authors / 作者: Inho Kim, Sumyeong Ahn
  • Primary Subject / 主学科: Computation and Language (cs.CL)
  • Secondary Subjects / 次学科: Artificial Intelligence (cs.AI), Information Retrieval (cs.IR)
  • Submission Date / 提交日期: August 24, 2026
  • Status / 状态: Accepted to Interspeech 2026

Abstract / 摘要

Retrieval-Augmented Generation (RAG) systems have attracted significant interest for their ability to mitigate hallucinations in Large Language Models (LLMs). Although knowledge databases for RAG are increasingly diversifying to include various modalities such as speech and text, research on handling such multi-modal database scenarios remains limited. In this paper, we propose STeReO (Speech and Text Reranking Orchestrator), a reranker based on speech and text retrievers that aggregates disparate modality databases. To address the lack of specialized training data, we first curate a dataset comprising queries, mixed-modality evidence, and their corresponding relevance ranks. We then train the reranker and evaluate its effectiveness in both single-modality and mixed-modality scenarios. Our results demonstrate that the proposed algorithm excels at selecting the most relevant evidence, thereby significantly improving downstream question-answering performance.