跳转至

知道何时停止：大语言模型评估的贝叶斯最优停止策略

文章背景与核心概要

在大语言模型（LLM）的常规评估中，通常采用固定的采样预算——即对每一个基准测试项目进行统一次数的重复测试，即便此时模型的性能估计已经具备极高的精度。这种做法往往造成了大量计算资源的浪费。为了解决这一问题，本文引入了 optstop，这是一个基于精度的自适应停止框架，它将模型评估视为一个顺序测量问题。

该框架利用分层贝叶斯推断（hierarchical Bayesian inference），能够动态分配计算资源：继续对具有高度不确定性的项目进行采样，同时终止那些估计已经足够稳定的测试。optstop 支持二进制、有序和连续的结果，既可在实时评估中运行，也可用于回顾性分析，且无需预先校准的题库。此外，它还包含一项关键的安全保护机制，在实测性能接近零（此时稀疏的成功至关重要）时采取更加谨慎的采样策略。在包含 200 个项目、10 个迭代周期的示例性评估中，该框架在九种验证设置下成功消除了 57% 至 97% 的计划试验，同时保持了与完整穷举运行等效的最终结论。

Summary

Large Language Model (LLM) evaluations traditionally rely on fixed sampling budgets—testing every benchmark item a uniform number of times, even after model performance estimates are already highly precise. This paper introduces optstop, a precision-based adaptive stopping framework that treats evaluation as a sequential measurement problem. By leveraging hierarchical Bayesian inference, the framework dynamically allocates compute: it continues sampling items with high uncertainty while terminating tests where estimates are sufficiently stable.

Operating either live or retrospectively across binary, ordinal, and continuous outcomes without requiring a pre-calibrated item bank, optstop also includes a crucial safeguard that applies heightened caution near zero performance, where rare successes are vital. In an illustrative 200-item, 10-epoch evaluation, the framework successfully eliminated 57% to 97% of planned trials across nine validation settings while preserving final conclusions equivalent to a full, exhaustive run.

Document Metadata

Abstract

LLM evaluations often use fixed sampling budgets, testing every item the same number of times even after estimates are precise. We introduce optstop, a precision-based adaptive stopping framework that treats evaluation as a sequential measurement problem: keep sampling where uncertainty remains high, and stop where estimates are precise or stable enough. The framework builds on hierarchical Bayesian inference, supports binary, ordinal, and continuous outcomes, and keeps every benchmark item eligible for sampling, without requiring a calibrated item bank. It runs live or retrospectively, and includes a safeguard that samples more cautiously as measured performance approaches zero, where rare successes matter most. In an illustrative 200-item, 10-epoch evaluation, it removes 57%-97% of planned trials across nine validation settings, with overall conclusions equivalent to the full run. These results show that LLM evaluation compute can be allocated by uncertainty rather than by fixed repetition counts, with the magnitude of savings depending on evaluation design.

Full-Text & Resources