跳转至

LLM预训练中领域数据重复策略的扩展规律

文章背景与核心概要

随着大语言模型（LLMs）规模的不断扩大，其训练Token预算必须按比例增长，以维持合适的每个参数Token数比例（\(\mathrm{TPP}\)）。然而，高质量的领域特定数据比通用网络数据更难扩展，这往往导致其在训练混合数据中的占比被稀释。尽管重复使用高质量数据有助于抵御这种稀释，但过度重复也会带来过拟合的风险。

本文深入探讨了在实际LLM扩展机制下的权衡取向。研究人员发现：1. 在固定的 \(\mathrm{TPP}\) 下，特定领域的最佳重复次数实际上会随着模型规模的增大而轻微增加；2. 在不同领域之间，最佳重复次数与该领域的最终验证损失呈现出强烈的负相关关系（验证损失更低的领域能够从更多的重复中获益）；3. 独占领域数据的绝对体量与最佳重复次数仅存在微弱的相关性。这些洞见最终表明，在保持相同 \(\mathrm{TPP}\) 的较小代理模型上调优得出的重复次数，可以作为扩展至更大模型时实用且可靠的估计标准。

总结 (Summary)

As large language models (LLMs) scale up, their training-token budgets must grow proportionally to maintain an appropriate tokens-per-parameter ratio (\(\mathrm{TPP}\)). However, high-quality domain-specific data is much harder to scale compared to general web data, often leading to a diluted fraction in the training mixture. While repeating high-quality data helps counteract this dilution, excessive repetition risks overfitting.

This paper investigates the trade-off under practical LLM scaling regimes. The authors discover that: 1. At a fixed \(\mathrm{TPP}\), the optimal repetition count for a specific domain actually mildly increases with model size. 2. Across different domains, the optimal repetition count correlates strongly and negatively with the domain's final validation loss (domains with lower loss benefit more from higher repetitions). 3. The absolute amount of unique domain data has only a weak relationship with the optimal repetition count.

Ultimately, these insights demonstrate that repetition counts tuned on smaller proxy models (maintaining the same \(\mathrm{TPP}\)) can serve as practical, reliable estimates for scaling much larger models.

文档元数据 (Document Metadata)

作者 (Authors)

  • Jingwei Li
  • Xinran Gu
  • Rui Dai
  • Xintong Hao
  • Chengyin Xu
  • Yan Wu
  • Shuran Zheng
  • Jingzhao Zhang

摘要 (Abstract)

As large language models scale, their training-token budgets must also increase to maintain an appropriate tokens-per-parameter ratio (\(\mathrm{TPP}\)). However, high-quality domain data is much harder to scale than general web data. As model size and the training-token budget increase, its fraction in the training mixture tends to decrease. Repeating the available high-quality data provides an effective way to counteract this dilution, but excessive repetition may lead to overfitting. We study this trade-off under practical LLM scaling, where the training-token budget grows proportionally with model size. For a fixed domain, we first find that, surprisingly at a fixed \(\mathrm{TPP}\), the optimal repetition count mildly increases with model size. Across different domains, we find that the optimal repetition count is strongly negatively correlated with the final validation loss of a domain: domains with lower loss can generally benefit from more repetitions. In contrast, the amount of unique domain data is only weakly related to the optimal repetition count. These findings suggest that repetition counts tuned on smaller proxy models with the same \(\mathrm{TPP}\) can provide a practical estimate for larger models.

访问与资源 (Access & Resources)