如何使用决策模型进行自动化评估：平衡成本、性能与置信度

文章背景与核心概要

随着大语言模型 (Large Language Model, LLM) 在各类业务系统中的广泛落地，如何低成本、高效率地对其输出质量进行自动化评估 (Evals) 成为了工业界的核心痛点。传统方案往往依赖昂贵且耗时的大模型裁判 (LLM Judge)，而专门的“决策模型 (Decision Models)”以其轻量、低延迟和低成本的分类能力正在快速崛起。本文通过真实智能体业务链路的测试数据，深入对比了 Perplexity、Liquid、Cloudflare 及 OpenAI 等主流决策模型的性能与成本权衡，并系统介绍了阈值微调与概率校准的关键实战技巧，为开发者搭建高性价比的自动化评估流水线提供了清晰指南。

核心内容摘要

Summary

决策模型（例如 TypeSafe 的 Jev、Perplexity 的 Decider、Liquid 的 d1、Cloudflare 的 Clef 以及 OpenAI 的 Decisions API）作为传统大语言模型裁判 (LLM Judge) 的替代方案迅速崛起。它们在自动化评估场景中表现出分类速度快、调用成本极低的优势。然而，它们能否真正发挥效能，很大程度上取决于你对成本、性能以及延迟 (Latency) 的具体业务需求。

Decision models—such as TypeSafe’s Jev, Perplexity’s Decider, Liquid’s d1, Cloudflare’s Clef, and OpenAI’s Decisions API—have emerged as fast, inexpensive classification alternatives to traditional LLM judges for automated evaluations. However, their effectiveness depends heavily on your specific requirements for cost, performance, and latency.

本指南深入探讨了如何合理配置与评估决策模型，而不是盲目依赖默认设置。核心要点包括：

This guide explores how to properly configure and evaluate decision models instead of relying on default settings. Key takeaways include:

阈值调优 (Threshold Tuning)：系统默认的 0.50 决策阈值往往无法充分发挥模型潜力。根据具体的准确率 (Precision) 和召回率 (Recall) 目标调优阈值，能够显著改善评估效果。
成本与性能权衡：在本次实测中，虽然 OpenAI Decisions 取得了最高的 F1 分数，但调用成本相对较高；而 Perplexity 与 Liquid d1-ultra 则提供了性价比极佳的轻量替代方案。
概率校准 (Probability Calibration)：决策模型输出的分数不仅能做判决，还可作为置信度预估。然而，这些原生概率往往需要经过校准（如采用 Platt 缩放算法），才能真实映射现实中的失败率。
持续迭代优化：将特定业务领域的提示词 (Prompt) 与严谨的阈值调优、概率校准手段相结合，能确保自动化评估结果与人类的真实偏好高度对齐。

Threshold Tuning: The default decision threshold of 0.50 often leaves performance on the table. Tuning thresholds based on your precision and recall goals significantly improves outcomes.

Cost vs. Performance: While OpenAI Decisions delivered the highest F1 score in our tests, it came at a higher price point, whereas models like Perplexity and Liquid d1-ultra offer budget-friendly alternatives.

Probability Calibration: Decision model scores can double as confidence estimates, but they often require calibration (such as Platt scaling) to accurately reflect observed failure rates.

Iterative Improvement: Combining domain-specific prompts with rigorous threshold and calibration checks ensures your automated evals closely align with human preferences.

实测方案设计

What We Tested

我们从一个真实的公寓租赁 AI 智能体 (AI Agent) 业务流中抽取了 100 条追踪记录 (Traces，即完整的对话与工具调用行为日志)，旨在精准识别各类交互错误，例如预约看房失败或未能及时转接人工客服等。每个决策模型都会针对每条追踪记录输出一个失败概率。

We used 100 traces (logs of conversations and actions) from an apartment leasing agent to identify errors, such as a failed booking or a missed handoff to a human. Each decision model returned a failure probability for each trace.

我们在完全一致的评估准则下，对 Jev 1.13、Perplexity Decider 1.1 27B、Liquid d1 和 d1-ultra、Cloudflare Clef Flash 以及 OpenAI Decisions (GPT-6 Luna) 进行了横向评测。所有模型均围绕两大核心维度展开对比：

We tested Jev 1.13, Perplexity Decider 1.1 27B, Liquid d1 and d1-ultra, Cloudflare Clef Flash, and OpenAI Decisions (GPT-6 Luna) using identical evaluation criteria. Models were compared across two core dimensions:

性能表现：将模型的预测判断与经过人工复核的通过/失败 (Pass/Fail) 真实标签进行对照。
调用成本：依据 Token 使用量及各家服务商的定价方案计算实际 API 支出。

Performance: Comparing model predictions against human-reviewed pass/fail labels.

Cost: Calculating API costs based on token usage and provider pricing.

深入理解决策模型与判断阈值

Understanding Decision Models and Thresholds

我们为每条追踪日志赋予二元的通过/失败标签，以此建立清晰明确、与人类偏好保持一致的评判标准。决策模型会输出一个介于 0 到 1 之间的概率分值，用于表示该用例发生失败的可能性。

We assign each trace a binary pass/fail label, encouraging clear criteria that align with human preferences. Decision models output a score from 0 to 1 indicating the likelihood of a failure.

在默认情况下，各家 API 通常采用 0.50 作为决策阈值，即只要分值达到或超过 0.50 就判定为失败。然而，如果不结合自身数据分布就直接套用默认阈值，往往会导致较差的评估效果。挑选恰当的阈值，本质上是在以下两个关键指标之间寻找平衡：

By default, APIs use a decision threshold of 0.50, flagging anything at or above this mark as a failure. However, accepting this default without checking your data can lead to poor results. Choosing the right threshold requires balancing:

准确率 (Precision)：在模型标记为失败的所有用例中，真正存在已知缺陷的比例。
召回率 (Recall)：在所有已知存在的真实缺陷中，被模型成功捕捉拦截的比例。

Precision: The share of flagged traces that contain a known failure.

Recall: The share of known failures successfully caught by the model.

我们采用 F1 分数（准确率与召回率的调和平均数）来综合衡量模型的整体表现，该分值越高，代表模型在查准与查全之间的平衡做得越出色。

We measure performance using the F1 score (the harmonic mean of precision and recall), where higher scores indicate better balance.

决策阈值的精细调优

Tuning the Decision Threshold

为了将模型性能推向极致，你可以借助编程智能体和交叉验证技术，根据设定的最低准确率与召回率门槛来精细调优阈值（例如对于高危告警系统，通常要求这两项指标均不得低于 80%）。通过科学的阈值调整，可以充分挖掘模型潜力，挽救那些在默认阈值下看似表现平平的模型。

To maximize performance, you can use coding agents and cross-validation to tune thresholds based on minimum precision and recall requirements (e.g., demanding at least 80% on both metrics for critical alerting systems). Tuning helps extract maximum value from models that might otherwise underperform at default settings.

性能与成本的权衡分析

Performance Versus Cost

将各模型的 F1 性能表现与其调用成本绘制在同一坐标系中，我们可以发现：

When mapping F1 performance against cost:

Perplexity 成为了所有参评候选模型中最经济实惠的选择，它仅用极低的成本就实现了高达 0.79 的 F1 分数。
OpenAI Decisions 取得了 0.87 的全场最高 F1 分数，但其调用开销大约是最便宜方案的 4.5 倍。

Perplexity proved to be the most economical choice among candidates, delivering an F1 score of 0.79 at a fraction of the cost.

OpenAI Decisions achieved the highest F1 score of 0.87, but at roughly 4.5 times the cost of the cheapest options.

借助 帕累托前沿 (Pareto frontier) 分析，我们能够轻松锁定在任意特定 F1 分数要求下最具性价比的模型选择。

Using a Pareto frontier helps identify the most cost-effective candidate for any desired F1 score threshold.

模型输出的概率值值得信赖吗？

Can We Trust the Probabilities?

除开简单的二元评估，决策模型给出的概率分值还可以直接充当置信度预估指标（例如 80% 的分值理论上代表该用例有 80% 的概率存在缺陷）。然而，许多模型开箱即用的原生概率输出往往缺乏良好的校准。

Beyond binary evals, decision model scores can serve as confidence estimates (e.g., an 80% score meaning an 80% chance of failure). However, models are not always well-calibrated out of the box.

概率校准 (Probability Calibration)：我们可以通过绘制模型预测概率与实际观测结果的对比曲线，来直观检验校准质量。
Platt 缩放 (Platt Scaling)：若模型存在系统性高估或低估风险的倾向，应用 Platt 缩放算法（或类似的概率映射技术）能够修正预测概率，使其更好地吻合实际分布，从而显著降低整体布里尔分数 (Brier Score)。

Probability Calibration: We can check calibration by plotting predicted probabilities against observed outcomes.

Platt Scaling: If a model underestimates or overestimates risk, applying Platt scaling (or similar mapping techniques) can adjust predicted probabilities to better match reality, reducing overall Brier scores.

提示词调优与评估范围界定

Prompt Tuning and Evaluator Scope

为了让决策模型发挥出最大威力，建议遵循以下实践步骤：

To get the best results from decision models:

先从宽泛的通用评估规则入手，摸清各模型的基线差异；待明确主要问题后，再将评估范围收窄到特定的失败模式。
在开展横向模型评测前，预先为准确率和召回率设定明确的硬性底线要求。
针对具体的私有数据集，严谨地进行决策阈值调优，并仔细校验概率校准情况。
全面权衡性能表现与成本、延迟 (Latency) 等工程维度的约束条件。

Start with a broader evaluator to spot baseline differences, then narrow the scope to specific failure modes if necessary.

Set hard minimums for precision and recall before comparing models.

Rigorously tune decision thresholds and check probability calibration for your unique dataset.

Weigh performance against cost and latency requirements.

附录：评测提示词

Appendix: Evaluation Prompt

以下是在 Perplexity、Liquid d1/d1-ultra、Clef Flash 以及 OpenAI Decisions 上统一使用的标准评估提示词：

The standard evaluation prompt used across Perplexity, Liquid d1/d1-ultra, Clef Flash, and OpenAI Decisions is outlined below:

Decide whether `messages` contains at least one substantive failure, risk, or quality problem in the AI assistant's behavior or tool use. This is one complete production trace from an AI that handles apartment-leasing customer support over phone or messaging. The trace includes the conversation plus any tool calls and tool outputs.

Evaluate the AI assistant and its tool use, not the customer. Treat system messages inside `messages` as the instructions governing the assistant's behavior and allowed capabilities. Inspect the full trace, including final turns and all tool calls and outputs. Base the judgment only on evidence visible in this trace. Do not assume external policies, channel behavior, or business requirements that are not present. Do not assume every trace failed. A short, ambiguous, or inconclusive trace is not an issue unless visible evidence establishes a substantive failure.

Return the probability that at least one substantive issue is present; the issue does not need to be named or clustered.

Count as an issue when supported by the trace: the assistant contradicts an applicable system instruction or tool result; makes an unsupported factual or operational claim; misses, misuses, repeats, or falsely claims completion of a required tool action, transfer, booking, cancellation, follow-up, or confirmation; materially misunderstands or leaves an explicit user request unresolved; or exposes internal actions, placeholders, or broken content to the customer in a way that materially harms the interaction.

Do not count as an issue: a minor stylistic preference with no meaningful effect; a failure caused only by the customer's behavior; internal tool metadata that was not exposed to the customer and did not cause an incorrect response; a hypothetical concern requiring information outside the trace; or insufficient evidence and inconclusive conversations.

示例：结构化提示词 (Jev)

Example: Structured Prompt (Jev)

{
  "version": "nurtureboss-any-issue-v1",
  "question_id": "has_substantive_issue",
  "type": "noul",
  "instructions": {
    "task": "Does `messages` contain at least one substantive failure, risk, or quality problem in the AI assistant's behavior or tool use?",
    "context": "This is one production trace from an AI that handles apartment-leasing customer support over phone or messaging. The trace contains the complete conversation plus any tool calls and tool outputs.",
    "evaluation_rules": [
      "Evaluate the AI assistant and its tool use, not the customer.",
      "Treat system messages inside `messages` as the instructions governing the assistant's behavior and allowed capabilities.",
      "Inspect the full trace, including the final turns and all tool calls and outputs.",
      "Base the judgment only on evidence visible in this trace. Do not assume external policies, channel behavior, or business requirements that are not present.",
      "Do not assume every trace failed. A short, ambiguous, or inconclusive trace is not an issue unless the visible evidence establishes a substantive failure.",
      "Return the probability that at least one substantive issue is present. The issue does not need to be named or clustered."
    ]
  },
  "criteria": {
    "true": {
      "definition": "At least one substantive problem is supported by evidence in the trace.",
      "examples_of_qualifying_evidence": [
        "The assistant contradicts an applicable system instruction or a tool result.",
        "The assistant makes an unsupported factual or operational claim.",
        "A required tool action, transfer, booking, cancellation, follow-up, or confirmation is missing, incorrect, repeated, or falsely claimed as complete.",
        "The assistant materially misunderstands or leaves an explicit user request unresolved.",
        "Customer-facing text exposes internal actions, placeholders, or broken content that materially harms the interaction."
      ]
    },
    "false": {
      "definition": "No substantive assistant problem can be established from the visible trace.",
      "do_not_count_as_an_issue": [
        "A minor stylistic preference with no meaningful effect.",
        "A failure caused only by the customer's behavior.",
        "Internal tool metadata that was not exposed to the customer and did not cause an incorrect response.",
        "A hypothetical concern that requires information outside the trace.",
        "Insufficient evidence or an inconclusive conversation."
      ]
    }
  }
}

注释与补充说明

Footnotes

参测版本：Jev 1.13、Perplexity Decider 1.1 27B、Liquid d1 与 d1-ultra、Cloudflare Clef Flash，以及 OpenAI Decisions (GPT-6 Luna)。OpenAI Decisions 通过独立于标准结构化输出 (Structured Outputs) 的专用 API 返回类型化判决，其中针对真/假判断问题会给出 0 至 1 的概率值。
LLM 裁判 (LLM Judges)：借助提示词直接裁决 AI 输出质量的评估器，通常采用二元通过/失败标准以简化与人类偏好的对齐过程。
数据集范围：共评测了 98 条生产环境追踪日志（其中 61 例已知失败，37 例通过），排除了内部 QA 样例。
API 成本：按照各云服务商的定价模型，折算每 100 次评测任务的预估开销。
延迟考量：批处理策略、并发规模、网络基础设施距离以及服务商配置均会对实际延迟产生显著影响。
准确率与召回率的权衡：取决于业务应用的技术风险等级（例如刑事判决要求极高准确率，而癌症筛查则追求极高召回率）。
密度分布图与调优：通过交叉验证评估了无约束 F1 调优与带有硬性最低指标约束的调优效果。
PR 曲线 (Precision-Recall Curves)：直观展现各项指标随决策阈值动态变动的权衡曲线。
候选模型约束：在训练集切分中强制要求准确率和召回率均达到 80% 的最低门槛。
概率校准 (Probability Calibration)：采用 Platt 缩放或等序回归 (Isotonic Regression) 等手段，确保模型预测概率（如 0.80）与真实环境中的错误出现频次严格吻合。
Platt 缩放：一种针对对数几率 (Log-Odds) 概率应用 Sigmoid 曲线映射的经典校准算法。
布里尔分数 (Brier Score)：用于衡量预测概率均方误差的标准指标（数值越低代表校准越好）。
GEPA 与提示词优化：通过基于反馈的循环迭代，不断精炼评估器的指导指令。

Versions tested: Jev 1.13, Perplexity Decider 1.1 27B, Liquid d1 and d1-ultra, Cloudflare Clef Flash, and OpenAI Decisions (GPT-6 Luna). OpenAI Decisions returns typed answers including a 0–1 probability for true/false questions via a separate API from standard Structured Outputs.

LLM judges: Evaluators that leverage prompts to judge AI outputs, ideally using binary pass/fail decisions to streamline human alignment.

Dataset scope: 98 production traces evaluated (61 known failures, 37 passing), excluding internal QA examples.

API costs: Calculated per 100 eval runs using provider pricing models.

Latency considerations: Varies significantly based on batching, concurrency, infrastructure proximity, and vendor setups.

Precision vs. Recall Tradeoffs: Tailored to the stakes of the application (e.g., high precision for criminal convictions vs. high recall for cancer screenings).

Density plots & Tuning: Unconstrained F1 tuning vs. constrained minimum requirements evaluated via cross-validation.

Precision-Recall curves: Visualizations showing how trade-offs scale across threshold adjustments.

Candidate Constraints: Training splits enforcing minimum 80% precision and recall thresholds.

Probability Calibration: Ensuring that model scores (e.g., 0.80) closely mirror real-world error frequencies using Platt scaling or isotonic regression.

Platt Scaling: Sigmoid curve adjustments applied to log-odds probabilities.

Brier Scores: Measurement of average squared probability error (lower is better).

GEPA & Prompt Optimization: Using feedback-driven iterations to continuously refine evaluator instructions.