跳转至

迈向基于查询覆盖率与声明可验证性的查询无关型 RAG 评估

文章背景与核心概要

检索增强生成（RAG）通过将大语言模型的回答根植于外部检索证据中，有效提升了事实准确性。然而，现有的评估框架在面对从封闭式事实检索到开放式解释性请求等各种多样化的用户查询时，往往难以提供一致且细粒度的诊断。

本文介绍了 Q-CARE 这一查询无关（Query-Agnostic）、无需人工参考答案（Reference-Free）的创新框架。它通过将复杂查询分解为子查询、将生成的答案解构为原子声明，并基于“查询覆盖率”与“声明可验证性”建立统一的评估原则。Q-CARE 能够输出强大的检索器指标（C-Prec@k, C-nDCG@k）和生成器指标（Completeness, Conciseness, Verifiableness），在涵盖八个数据集的人工标注基准测试中，展现出比现有方法更高的对齐人类判断的相关性。

摘要 (Summary)

Retrieval-Augmented Generation (RAG) improves Large Language Model (LLM) factuality by grounding responses in external evidence. However, existing evaluation frameworks struggle to provide consistent diagnostics across diverse user queries (ranging from closed-ended fact-seeking to open-ended explanations). This paper introduces Q-CARE, a query-agnostic, reference-free framework that decomposes queries into sub-queries and answers into atomic claims. By establishing a unified evaluation principle based on query coverage and claim verifiability, Q-CARE delivers robust retriever metrics (C-Prec@k, C-nDCG@k) and generator metrics (Completeness, Conciseness, Verifiableness), achieving higher correlation with human judgments than existing methods across eight datasets.

检索增强生成（RAG）通过将大语言模型（LLM）的回答建立在检索到的证据基础之上，提升了其事实性。然而，现有的评估框架难以在各种多样化的用户查询（从封闭式事实寻求到开放式解释请求）中提供一致且细粒度的诊断。本文引入了 Q-CARE，这是一个查询无关且完全免参考（Reference-Free）的框架，通过将查询分解为子查询、将回答分解为原子声明来实现细粒度评估。Q-CARE 建立了一个基于查询覆盖率和声明可验证性的统一评估原则，产生了感知覆盖率的检索器指标（C-Prec@k, C-nDCG@k）以及声明级别的生成器指标（完整性、简洁性和可验证性）。在涵盖八个数据集的人工标注基准测试中，Q-CARE 实现了比包括 RAGEval 和 RAGChecker 在内的四种现有 RAG 评估指标更高的与人类判断的相关性，证明了其作为可靠、自动化评估框架的有效性。

论文元数据 (Paper Metadata)

  • arXiv ID: arXiv:2608.11238 [cs.AI]
  • Primary Subject: Artificial Intelligence (cs.AI)
  • Submission Date: 31 July 2026
  • Conference: Accepted to COLM 2026
  • Authors: Jeonghwan Choi, Taewon Yun, Minjeong Ban, Gyeonghun Sun, Jae-Gil Lee, Hwanjun Song
  • Resources: GitHub Repository
  • arXiv ID: arXiv:2608.11238 [cs.AI]
  • 主要学科: 人工智能 (cs.AI)
  • 提交日期: 2026年7月31日
  • 收录会议: 已被 COLM 2026 接收
  • 作者: Jeonghwan Choi, Taewon Yun, Minjeong Ban, Gyeonghun Sun, Jae-Gil Lee, Hwanjun Song
  • 开源资源: GitHub 仓库

摘要正文 (Abstract)

Retrieval-augmented generation improves the factuality of large language models by grounding responses in retrieved evidence, yet existing evaluation frameworks struggle to provide consistent, fine-grained diagnostics across the diverse spectrum of user queries, ranging from close-ended fact-seeking to open-ended explanatory requests. We propose Q-CARE, a query-agnostic and fully reference-free framework that enables fine-grained assessment by decomposing queries into sub-queries and answers into atomic claims. Q-CARE establishes a unified evaluation principle based on query coverage and claim verifiability, yielding coverage-aware retriever metrics (C-Prec@k, C-nDCG@k) and claim-level generator metrics (Completeness, Conciseness, and Verifiableness). On a human-annotated benchmark spanning eight datasets, Q-CARE achieves higher correlation with human judgments than four existing RAG evaluation metrics, including RAGEval and RAGChecker, proving its effectiveness as a reliable, automated evaluation framework.

检索增强生成通过将大语言模型的回答根植于检索到的证据中，改善了其事实性，然而现有的评估框架在面对从封闭式事实寻求到开放式解释性请求等各种多样化的用户查询时，难以提供一致、细粒度的诊断。我们提出了 Q-CARE，这是一个查询无关且完全无需参考答案的框架，它通过将查询分解为子查询、将回答分解为原子声明来进行细粒度评估。Q-CARE 建立了一个基于查询覆盖率和声明可验证性的统一评估原则，生成了感知覆盖率的检索器指标（C-Prec@k, C-nDCG@k）以及声明级别的生成器指标（Completeness、Conciseness 和 Verifiableness）。在一个跨越八个数据集的人工标注基准上，Q-CARE 实现了比 RAGEval 和 RAGChecker 等四种现有 RAG 评估指标更高的人类判断相关性，证明了其作为可靠、自动化评估框架的有效性。