跳转至

使用经过验证的任务覆盖率评估大语言模型的多次生成能力

文章背景与核心概要

传统的语言模型（LLM）评估通常侧重于单个输出，或者将多个候选响应简化为单一的成功与否或选定的答案。然而，许多实际应用需要生成多个候选输出以便进行对比、组合或验证。本文介绍了 VTC-Bench（一个包含五个领域的基准测试）以及 经过验证的任务覆盖率（Validated Task Coverage, VTC），用以直接评估有限的候选输出集。

通过采用可重复的自动化检查而非基于模型的裁判（model-based judges），作者发现单次抽取性能强劲的模型并不一定能提供最佳的覆盖率，且简单的输出变异性指标也无法可靠地反映出与任务相关的覆盖情况。这项研究揭示了传统逐个输出评估所无法察觉的模型行为差异，为直接评估有限候选集提供了新的视角。

评估多个大语言模型生成的任务：使用经过验证的任务覆盖率

Summary

Traditional Large Language Model (LLM) evaluations often focus on individual outputs or reduce multiple candidate responses down to a single success or selected answer. However, many practical applications require generating several candidate outputs for comparison, combination, or validation. This paper introduces VTC-Bench (a five-domain benchmark) and Validated Task Coverage (VTC) to evaluate finite candidate sets directly. By using reproducible, automated checks rather than model-based judges, the authors show that models with strong single-draw performance do not necessarily offer the best coverage, and simple output variation metrics do not reliably reflect task-relevant coverage.

元数据

Metadata

摘要

Abstract

Many LLM applications are most useful when they provide several candidate outputs for comparison, validation, or combination. Predominant evaluation settings, however, still focus on individual outputs or reduce multiple samples to a single success or selected answer. This can miss whether the outputs include several genuinely different useful results.

许多大语言模型（LLM）应用在提供多个候选输出以供比较、验证或组合时，其效用才能最大化。然而，当前主流的评估环境仍然侧重于单个输出，或者将多个样本简化为单一的成功结果或选定的答案。这可能会遗漏输出中是否包含多个真正不同的有用结果这一关键信息。

We introduce VTC-Bench, a five-domain benchmark for this setting, together with Validated Task Coverage (VTC) as its core evaluation quantity. The benchmark is built from carefully selected real-data tasks where both output quality and task-relevant distinctness can be checked automatically and reproducibly, without model-based judges. VTC measures how many distinct useful results are obtained within \(k\) attempts.

为此，我们引入了 VTC-Bench（一个针对该场景的五领域基准测试），以及作为其核心评估指标的 经过验证的任务覆盖率（Validated Task Coverage, VTC）。该基准测试构建于精心挑选的真实数据任务之上，在这些任务中，输出质量和与任务相关的独特性都可以通过自动化且可重复的方式进行检查，而无需依赖基于模型的裁判。VTC 用于衡量在 \(k\) 次尝试内能获得多少个不同且有用的结果。

Across multiple models and inference settings, the benchmark leads to different conclusions from conventional evaluation: configurations that look strongest from single-draw quality are not necessarily those with the best coverage, and simple measures of output variation do not reliably recover task-relevant coverage. These results show that finite candidate sets can be evaluated directly as objects of interest, revealing differences in model behavior that are not apparent from conventional per-output evaluation.

跨多个模型和推理设置的实验表明，该基准测试得出了与传统评估截然不同的结论：从单次抽取质量来看表现最强的配置，不一定是覆盖率最好的配置；而简单的输出变异性度量也无法可靠地反映出与任务相关的覆盖情况。这些结果表明，有限的候选集可以作为研究对象进行直接评估，从而揭示出传统逐个输出评估所无法发现的模型行为差异。

全文与资源

Full-Text & Resources