跳转至

文章背景与核心概要

专家调查是安全研究的基石，然而由于安全运营中心（SOC）专业人员面临的工作负载高、职业倦怠以及严格的保密限制，这类调查正变得越来越难以开展。本文探讨了大语言模型（LLM）作为“替代专家”大规模生成合成调查数据的潜力。

作者引入了一个方法论框架，用于评估 LLM 在此角色中的可靠性和有效性。通过将基于角色（persona-based）和聚合的 LLM 回答与 SOC 专业人员的实际数据进行对比，研究发现：尽管 LLM 能够提供内部一致的数据，但它们存在明显的局限性——具体表现为方差减小、集中趋势偏见以及观点同质化。研究结论认为，虽然 LLM 对于前期试验和假设生成很有价值，但目前它们尚无法替代安全研究中的人类专家征询。

Summary

Expert surveys are a cornerstone of security research, yet they are increasingly difficult to conduct due to the high workload, burnout, and strict confidentiality constraints faced by professionals in Security Operations Centres (SOCs). This paper explores the potential of Large Language Models (LLMs) as "surrogate experts" to generate synthetic survey data at scale.

The authors introduce a methodological framework to evaluate the reliability and validity of LLMs in this capacity. By comparing persona-based and aggregate LLM responses against actual data from SOC professionals, the study reveals that while LLMs provide internally consistent data, they suffer from significant limitations—specifically reduced variance, central tendency bias, and a homogenization of opinions. The research concludes that while LLMs are valuable for piloting and hypothesis generation, they are not currently a viable replacement for human expert elicitation in security research.

A Framework for Using and Evaluating LLMs as Surrogate Experts in Security Surveys: Reliability, Bias, and Implications

arXiv: 2608.16893
Date: July 6, 2026
Authors: Despoina Giarimpampa, Roland Meier, Tegawendé F. Bissyandé, Vincent Lenders, Jacques Klein

Summary (核心摘要)

Summary

Expert surveys are a cornerstone of security research, yet they are increasingly difficult to conduct due to the high workload, burnout, and strict confidentiality constraints faced by professionals in Security Operations Centres (SOCs). This paper explores the potential of Large Language Models (LLMs) as "surrogate experts" to generate synthetic survey data at scale.

The authors introduce a methodological framework to evaluate the reliability and validity of LLMs in this capacity. By comparing persona-based and aggregate LLM responses against actual data from SOC professionals, the study reveals that while LLMs provide internally consistent data, they suffer from significant limitations—specifically reduced variance, central tendency bias, and a homogenization of opinions. The research concludes that while LLMs are valuable for piloting and hypothesis generation, they are not currently a viable replacement for human expert elicitation in security research.

专家调查是安全研究的基石，然而由于安全运营中心（SOC）专业人员面临的工作负载高、职业倦怠以及严格的保密限制，这类调查正变得越来越难以开展。本文探讨了大语言模型（LLM）作为“替代专家”大规模生成合成调查数据的潜力。

作者引入了一个方法论框架，用于评估 LLM 在此角色中的可靠性和有效性。通过将基于角色和聚合的 LLM 回答与 SOC 专业人员的实际数据进行对比，研究发现：尽管 LLM 能够提供内部一致的数据，但它们存在明显的局限性——具体表现为方差减小、集中趋势偏见以及观点同质化。研究结论认为，虽然 LLM 对于前期试验和假设生成很有价值，但目前它们尚无法替代安全研究中的人类专家征询。

Key Findings (关键发现)

Key Findings

  • Methodological Framework: The authors establish a systematic approach for evaluating LLMs as substitutes or supplements for human survey respondents.
  • Systematic Divergence: Despite high internal consistency, LLM responses systematically diverge from human expert responses.
  • Identified Biases: The study highlights three primary issues with LLM-generated survey data:
    • Reduced Variance: A lack of the diversity found in human responses.
    • Central Tendency Bias: A tendency for models to gravitate toward "average" or neutral answers.
    • Homogenized Opinions: A lack of the nuanced, outlier, or dissenting perspectives typical of human experts.
  • Practical Guidance: LLMs should be utilized for early-stage research tasks, such as survey piloting and hypothesis generation, rather than as a substitute for primary data collection.
  • 方法论框架： 作者建立了一个系统化方法，用于评估 LLM 作为人类调查受访者的替代品或补充品的效果。
  • 系统性分歧： 尽管内部一致性很高，但 LLM 的回答与人类专家的回答存在系统性分歧。
  • 识别出的偏见： 研究强调了 LLM 生成的调查数据的三个主要问题：
    • 方差减小： 缺乏人类回答中所体现的多样性。
    • 集中趋势偏见： 模型倾向于给出“平均”或中立的答案。
    • 观点同质化： 缺乏人类专家所特有的细微差别、极端值或异见视角。
  • 实践指导： LLM 应被用于早期研究任务（如调查试测和假设生成），而不是替代主要数据收集工作。

Access & Resources (访问与资源)

Access & Resources

Citation & Metadata (引用与元数据)

Citation & Metadata