使用大语言模型发现数据湖中的数据关系：一项工业级案例研究

文章背景与核心概要

数据湖通常面临元数据有限或信息含量弱的问题，特别是企业资源规划（ERP）数据集中充斥着晦涩的代码和缩写，给数据治理和分析带来了巨大挑战。为了解决这一痛点，本文引入了一种名为 ColRel 的全新两阶段框架，旨在提升列关系的发现能力。

该方法通过在数据摄入时充分利用元数据、数据实例以及业务字典，生成自然语言描述来准确解释复杂的架构模式（schemata）。在公开基准数据集以及真实工业级 ERP 数据集上的评估结果表明，ColRel 在具有语义关联但信号较弱的环境中展现出了卓越的有效性。

Discovering Relationships in Data Lakes Using Large Language Models: An Industrial Case

摘要 (Summary)

Data lakes often suffer from limited or weakly informative metadata, particularly in Enterprise Resource Planning (ERP) datasets characterized by cryptic codes and abbreviations. This paper introduces ColRel, a novel two-stage framework designed to improve column relationship discovery. By leveraging metadata, data instances, and business dictionaries at ingestion time, ColRel generates natural-language descriptions to accurately interpret schemata. Evaluations on both public benchmarks and real-world industrial ERP datasets demonstrate its superior effectiveness in semantically related, weak-signal environments.

文档元数据 (Document Metadata)

摘要 (Abstract)

Data lakes rely on metadata to remain usable, yet this metadata is often limited or weakly informative for column relationship discovery, especially in ERP-derived datasets with coded or abbreviated schema labels. We propose ColRel, a two-stage method that builds column embeddings from metadata and data available at ingestion time. In difficult cases, such as coded schemata, business dictionaries help better interpret column names and support the generation of short natural-language descriptions used in the second stage. Experiments on public benchmarks and an industrial ERP dataset show that ColRel is particularly effective in semantically related, weak-signal settings.

访问与资源 (Access & Resources)

• 全文 PDF： 查看 PDF
• TeX 源码： 下载源码
• 外部引用：
• Google Scholar
• Semantic Scholar
• NASA ADS