跳转至

文章背景与核心概要

当前的金融大语言模型（LLM）评估方法——例如静态问答或最终盈亏（P&L）追踪——存在明显局限，因为它们要么缺乏自主性（agency），要么无法验证盈利行为究竟是基于扎实且符合个人画像的推理，还是仅仅靠运气。为了填补这一空白，本文作者推出了 InvestLogicBench，这是一个过程原生（process-native）的基准测试，涵盖了来自151位真实投资者的201,247个记录在案的决策。该基准采用了结构化的 P\(\rightarrow\)E\(\rightarrow\)R\(\rightarrow\)D\(\rightarrow\)O 框架（Profile 画像 \(\rightarrow\) Events 事件 \(\rightarrow\) Reasoning 推理 \(\rightarrow\) Decision 决策 \(\rightarrow\) Outcome 结果）。

对四款主流LLM的评估表明，尽管模型表现出的逻辑合理性较高（约4/5分），但其事件依据扎实度（event grounding）却很弱（0.8–2.8/5分）。这暴露出模型存在“表面上文笔流畅、实则依据薄弱”的推理问题，而传统的仅看结果的指标根本无法捕捉到这一点。该研究不仅推动了金融智能体的发展，也为更广泛的个性化、高影响力智能体评估提供了重要的压力测试方法。

Evaluating Investment Logic in Large Language Models: A Real-World Benchmark Towards Personalized Financial Agents

arXiv: 2608.06108 [cs.AI]
Submitted: August 6, 2026
Authors: Yuanhong Jiang, Jingjie Zou, Zhenghong Lin, Xusheng Yu, Qiqi Huang, Shuai Jia, Shijie Dai

📌 Summary

Current evaluation methods for financial Large Language Models (LLMs)—such as static question answering or terminal profit-and-loss (P&L) tracking—fall short because they omit agency or fail to verify whether profitable actions were genuinely grounded and profile-consistent rather than just lucky.

To address this gap, the authors introduce InvestLogicBench, a process-native benchmark featuring 201,247 documented decisions from 151 real-world investors. The benchmark utilizes a structured P\(\rightarrow\)E\(\rightarrow\)R\(\rightarrow\)D\(\rightarrow\)O framework (Profile \(\rightarrow\) Events \(\rightarrow\) Reasoning \(\rightarrow\) Decision \(\rightarrow\) Outcome). Evaluating four leading LLMs reveals that while logical plausibility remains high (~4/5), event grounding is weak (0.8–2.8/5), exposing polished yet weakly grounded reasoning that traditional outcome-only metrics fail to capture.

📋 Paper Metadata

Field Details
Subjects Artificial Intelligence (cs.AI)
Cite As arXiv:2608.06108 [cs.AI]
DOI 10.48550/arXiv.2608.06108
License Creative Commons Attribution 4.0 International license icon

🔍 Abstract

投资能力本质上是个体化的：面对相同的市场证据，具有不同目标、投资期限、投资组合和风险边界的投资者可能会采取截然不同的行动。然而，目前的金融大语言模型评估要么依赖静态问答，要么依赖最终盈亏。前者忽略了自主性，后者则无法揭示盈利行为究竟是有据可依、符合个人画像的，还是仅仅靠运气。我们不禁要问，学术界是否在为具有实质影响力的智能体使用了错误的“标尺”。

我们引入了 InvestLogicBench，这是一个过程原生（process-native）的基准测试，包含来自 151 位真实投资者的 201,247 个记录在案的决策。每个事件片段都实例化了一个 P\(\rightarrow\)E\(\rightarrow\)R\(\rightarrow\)D\(\rightarrow\)O 追踪过程：投资者 Profile（画像）、可观测的市场 Events（事件）、投资 Reasoning（推理）、可执行的 Decision（决策） 以及滞后的 Outcome（结果）。该版本发布内容包括画像构建、时间点事件绑定、结构化逻辑、投资期限、结果及事后剖析（post-mortems），并支持理解、画像条件生成以及端到端重放。在对四款领先的 LLM 的评估中，逻辑合理性保持在近 4/5 分，而事件依据扎实度仅为 0.8–2.8/5 分；回报率与过程质量也往往存在分歧。这些结果暴露出模型推理光鲜亮丽但依据薄弱的问题，而仅关注结果的评估方法则掩盖了这一点。我们进一步论证，P\(\rightarrow\)E\(\rightarrow\)R\(\rightarrow\)D\(\rightarrow\)O 应当成为一种数据系统接口，需要版本化的画像、时间溯源、可检查的检索、决策总账以及可重放的结果。金融领域是我们对更广泛的个性化、具重大影响力的智能体类别进行的压力测试。

Investment competence is inherently personalized: the same market evidence can justify different actions for investors with different goals, horizons, portfolios, and risk boundaries. Yet financial LLMs are evaluated either by static question answering or by terminal profit and loss. The former omits agency; the latter cannot reveal whether a profitable action was grounded, profile-consistent, or merely lucky. We ask whether the community is using the wrong ruler for consequential agents.

We introduce InvestLogicBench, a process-native benchmark containing 201,247 documented decisions from 151 real-world investors. Each episode instantiates a P\(\rightarrow\)E\(\rightarrow\)R\(\rightarrow\)D\(\rightarrow\)O trace: investor Profile, observable market Events, investment Reasoning, executable Decision, and delayed Outcome. The release includes profile construction, point-in-time event binding, structured logic, horizons, outcomes, and post-mortems, and supports comprehension, profile-conditioned generation, and end-to-end replay. Across four leading LLMs, logical plausibility remains near 4/5 while event grounding is only 0.8–2.8/5; return and process quality also disagree. These results expose polished but weakly grounded reasoning that outcome-only evaluation hides. We further argue that P\(\rightarrow\)E\(\rightarrow\)R\(\rightarrow\)D\(\rightarrow\)O should be a data-system interface, requiring versioned profiles, temporal provenance, inspectable retrieval, decision ledgers, and replayable outcomes. Finance is our stress test for a broader class of personalized, consequential agents.