跳转至

文章背景与核心概要

大语言模型（LLM）通常被视为“黑盒”系统，这使得外界很难判断其输出是建立在稳定的内部推理之上，还是仅仅停留在表面的模式匹配阶段。本文引入了跨语境一致性（Cross-Contextual Consistency, C3）这一行为属性，用于衡量大语言模型的可信度。通过在主题对齐但内容中立的语境扰动下，对 26 个模型和 6 个基准测试的模型响应进行评估，作者证明了更小的跨语境偏移与客观、正确的答案之间存在着强烈的相关性。此外，C3 还可以作为评估基准测试有效性的宝贵诊断工具，尤其适用于总体性能评分已经趋于饱和的场景。

Is This Your Final Answer? Cross-Contextual Consistency as a Measure of LLM Credibility

arXiv: 2608.10315 [cs.CL]
Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
Submitted On: 10 August 2026
Authors: Siyang Wu, Yibo Jiang, Bryon Aragam

📌 Executive Summary

Large language models (LLMs) often function as "black-box" systems, making it difficult to determine whether their outputs are grounded in stable internal reasoning or merely surface-level pattern matching. This paper introduces Cross-Contextual Consistency (C3), a behavioral property used to measure LLM credibility. By evaluating model responses under topic-aligned yet content-neutral context perturbations across 26 models and 6 benchmarks, the authors demonstrate that smaller cross-contextual shifts strongly correlate with factual and correct answers. Furthermore, C3 serves as a valuable diagnostic tool for assessing benchmark utility, especially in scenarios where aggregate performance scores are otherwise saturated.

📖 Abstract

Large language models (LLMs) are powerful black-box systems, making it difficult to discern whether their answers reflect stable internal beliefs or superficial pattern matching. We identify cross-contextual consistency as an underutilized behavioral property of LLMs: a credible answer should remain stable when the same task is placed under topic-aligned, content-neutral contextual variation. Building on this intuition, we operationalize Cross-Contextual Consistency (C3) by comparing model generations under original and perturbed prompts. Across 26 models and six benchmarks spanning reasoning, factuality, and code generation, we find that answers with smaller cross-contextual shifts are more likely to be correct or factual. We demonstrate that C3 provides a complementary axis of evaluation and can serve as a benchmark usefulness diagnostic, identifying which portions of a benchmark remain informative even when aggregated scores are widely considered "saturated".

📄 License

license icon
This article is made available under the Creative Commons Attribution 4.0 International License.