跳转至

文章背景与核心概要

网络安全团队和研究人员在评估用于网络威胁情报（CTI）报告的知识图谱提取工具时，通常严重依赖已发布的 三元组F1分数（triple-F1 scores）。然而，这项由 Safayat Bin Hakim 和 Houbing Herbert Song 开展的可复现性审计发现，这些报告中的分数往往高度依赖于将预测三元组与黄金标准（gold-standard）注释进行匹配的具体方法。

审计表明，在被检查的 12 个系统中，只有 5 个系统的声明匹配规则能够被成功重新实现。更严重的是，使用 8 种不同的协议对共享文档上的 10 个系统输出进行重新打分，竟逆转了 45 种成对系统排序中的 11 种。根据所选协议的不同，同一组固定的预测结果其 F1 分数甚至会在 0.16 到 0.70 之间大幅波动。为了解决评估标准不统一的问题，作者开发了 CTIForge 框架，允许在保持提取结果字节级相同的同时，变换确定性的验证层。研究发现，验证层的引入对托管骨干网和离线骨干网产生了截然相反的影响，这暴露出手工编写的验证规则往往会编码特定提取器的偏见。

Ranked by the Matcher: A Reproducibility Audit of Knowledge Graph Extraction from Threat Reports

Ranked by the Matcher: A Reproducibility Audit of Knowledge Graph Extraction from Threat Reports

Safayat Bin Hakim, Houbing Herbert Song
Submitted: September 1, 2026
Primary Subject: Cryptography and Security (cs.CR)
Additional Subjects: Artificial Intelligence (cs.AI), Computation and Language (cs.CL)
arXiv ID: 2609.01671
DOI: 10.48550/arXiv.2609.01671

Summary

Summary

安全团队和研究人员严重依赖已发布的 三元组F1分数 来评估针对网络威胁情报（CTI）报告的知识图谱提取工具。然而，这项可复现性审计表明，这些报告的分数很大程度上取决于将预测三元组与黄金标准注释进行匹配的具体方法：

Security teams and researchers heavily rely on published triple-F1 scores to evaluate knowledge-graph extraction tools for cyber threat intelligence (CTI) reports. However, this reproducibility audit reveals that these reported scores are often heavily dependent on the specific methods used to match predicted triples to gold-standard annotations:

  • 匹配差异： 在受检的 12 个系统中，声明的匹配规则仅成功重新实现了 5 个。
  • 分数波动： 使用 8 种不同的协议对共享文档上的 10 个系统输出进行重新打分，逆转了 45 个成对系统排序中的 11 个。根据协议的不同，单一固定的预测集其 F1 分数范围跨越了 0.16 到 0.70。
  • 人工裁决一致性： 在外部的 378 项校准集（GRID）上，机械匹配器（词法、嵌入或蕴涵）与多评审员人工裁决的一致性不超过 71%，而基于 LLM 的裁判则达到了 86%。
  • CTIForge 框架： 为了将组件从匹配器奖励中隔离出来，作者开发了 CTIForge 框架，该框架允许在保持提取结果字节级相同的同时，改变确定性验证层。
  • 部署发现： 在测试的 7 种配置中，引入验证提高了托管骨干网（hosted backbones）的准确率（precision），同时降低了离线骨干网（offline backbones）的准确率。这种行为与对实体类型提出异议的操作增加了约 2.8 倍相吻合，表明手工编写的规则编码了它们所针对的具体提取器的偏见。
  • Matching Discrepancies: The stated matching rules could only be successfully reimplemented for 5 out of 12 inspected systems.
  • Score Volatility: Re-scoring 10 system outputs across shared documents using 8 different protocols reversed 11 out of 45 pairwise system orderings. A single fixed set of predictions spanned an F1 score range from 0.16 to 0.70 depending on the protocol.
  • Adjudication Agreement: On an external 378-item calibration set (GRID), mechanical matchers (lexical, embedding, or entailment) agreed with multi-reviewer human adjudication no more than 71% of the time, whereas an LLM-based judge achieved 86%.
  • CTIForge Framework: To isolate components from matcher rewards, the authors developed CTIForge, a framework allowing deterministic validation layers to vary while keeping extraction byte-identical.
  • Deployment Findings: Across seven tested configurations, introducing validation raised precision for hosted backbones while lowering it for offline backbones. This behavior correlates with a ~2.8-fold increase in actions disputing entity types, pointing to hand-written rules encoding the biases of the specific extractors they were built against.

Abstract

Abstract

安全团队和研究人员根据已发布的三元组 F1 分数来选择威胁报告的知识图谱提取工具，然而这些分数取决于预测三元组与黄金注释的匹配方式。在检查的十二个系统中，我们只能重新实现其中五个系统的声明匹配规则。在八种协议下对共享文档上的十个系统输出进行重新打分，逆转了四十五个成对排序中的十一个；一个固定的预测集的 F1 分数跨度为 0.16-0.70。在 GRID 的外部 378 项校准集上，没有任何机械匹配器（词法、嵌入或蕴涵）与多评审员裁决的一致性超过 71%，而 LLM 裁判达到了 86%。为了将组件效应与匹配器奖励分离，我们构建了 CTIForge，其确定性验证层可以变化，同时提取保持字节级相同。在七个测试的部署配置中，验证提高了所有四个托管骨干网的精度，并降低了所有三个离线骨干网的精度。由于骨干网、解码和后端特定的提示是协变的，这是一个描述性的拆分，而不是孤立的服务效应。它伴随着对实体类型明确提出异议的操作大约 2.8 倍的增加，这与编码其开发所针对的提取器惯例的手工规则相一致。我们发布了管道、协议套件和每个三元组的审计记录。

Security teams and researchers choose knowledge-graph extraction tooling for threat reports on the strength of published triple-F1 scores, yet those scores depend on how predicted triples are matched to gold annotations. We could reimplement the stated matching rule for only five of twelve inspected systems. Re-scoring ten system outputs on shared documents under eight protocols reverses eleven of forty-five pairwise orderings; one fixed prediction set spans 0.16-0.70 F1. On GRID's external 378-item calibration set, no mechanical matcher (lexical, embedding, or entailment) agrees with multi-reviewer adjudication above 71%, whereas an LLM judge reaches 86%. To separate component effects from matcher rewards, we build CTIForge, whose deterministic validation layer can vary while extraction is held byte-identical. Across seven tested deployment configurations, validation raises precision for all four hosted backbones and lowers it for all three offline backbones. Because backbone, decoding, and backend-specific prompting covary, this is a descriptive split rather than an isolated serving effect. It coincides with a roughly 2.8-fold increase in actions explicitly disputing entity type, consistent with hand-written rules encoding the conventions of the extractor against which they were developed. We release the pipeline, protocol suite, and per-triple audit records.

(注：如下方格式说明所示保留了许可证图标) (Note: License icon preserved below as required by formatting instructions) license icon