跳转至

文章背景与核心概要

在大模型与深度学习训练中，数据集的规模通常被认为是决定模型性能的关键因素。然而，这项研究深入探讨了“小数据集与大数据集差距（small-vs-large gap）”这一反直觉现象：通过在较少的样本上进行重复训练，往往能够比使用庞大数据集节省大量的计算成本，并实现更快的收敛。这种效率提升在多种算法任务、网络架构和优化器中均可观察到，且无法用现有的理论框架来解释。

作者在研究中证明，这种加速效果源于采样偏差所带来的适度逐层增长（layer-wise growth），而这种采样偏差在较小的数据集中表现得更为明显。通过提供详尽的理论分析与丰富的实证干预证据，该研究得出结论：利用较小数据集并进行高频重复训练，不仅仅是应对数据稀缺时的无奈之举，更是一种强有力的优化归纳偏差（inductive bias），特别是在处理复杂推理任务时展现出巨大优势。

Less Data, Faster Training: Repeating Smaller Datasets Speeds Up Learning via Sampling Biases

Less Data, Faster Training: Repeating Smaller Datasets Speeds Up Learning via Sampling Biases

📋 Summary

This research investigates the "small-vs-large gap," a phenomenon where repeatedly training on fewer samples yields significant compute savings compared to using a larger dataset. Observed across diverse algorithmic tasks, architectures, and optimizers, this efficiency cannot be explained by existing theories. The authors demonstrate that this speedup stems from appropriate layer-wise growth enabled by sampling biases, which become more pronounced with smaller datasets. Providing both theoretical analysis and empirical evidence, the study concludes that leveraging smaller datasets with frequent repetitions is not merely a fallback for data scarcity, but a powerful inductive bias for optimization—especially in reasoning tasks.

📋 Summary

This research investigates the "small-vs-large gap," a phenomenon where repeatedly training on fewer samples yields significant compute savings compared to using a larger dataset. Observed across diverse algorithmic tasks, architectures, and optimizers, this efficiency cannot be explained by existing theories. The authors demonstrate that this speedup stems from appropriate layer-wise growth enabled by sampling biases, which become more pronounced with smaller datasets. Providing both theoretical analysis and empirical evidence, the study concludes that leveraging smaller datasets with frequent repetitions is not merely a fallback for data scarcity, but a powerful inductive bias for optimization—especially in reasoning tasks.

📄 Metadata

📄 Metadata

  • Primary Subject: Machine Learning (cs.LG)
  • Primary Subject: Machine Learning (cs.LG)
  • Secondary Subjects: Artificial Intelligence (cs.AI)
  • Secondary Subjects: Artificial Intelligence (cs.AI)
  • Conference: ICML 2026
  • Conference: ICML 2026
  • Submitted: May 19, 2026
  • Submitted: May 19, 2026
  • Last Revised: September 4, 2026
  • Last Revised: September 4, 2026

Authors

  • Jingwen Liu
  • Ezra Edelman
  • Surbhi Goel
  • Bingbin Liu

Authors

  • Jingwen Liu
  • Ezra Edelman
  • Surbhi Goel
  • Bingbin Liu

🔎 Abstract

This work investigates the "small-vs-large gap", where repeating on fewer samples can lead to compute saving during training compared to using a larger dataset. This is observed across algorithmic tasks, architectures and optimizers and cannot be explained using prior theory. We argue that the speedup comes from appropriate layer-wise growth enabled by sampling biases, which is more pronounced when the dataset size is smaller. We provide both theoretical analysis and empirical evidence from various interventions. Our results suggest that using a smaller dataset with more repetitions is not just a fallback strategy under data scarcity, but can be proactively leveraged as a favorable inductive biases for optimization, particularly in reasoning tasks.

🔎 Abstract

This work investigates the "small-vs-large gap", where repeating on fewer samples can lead to compute saving during training compared to using a larger dataset. This is observed across algorithmic tasks, architectures and optimizers and cannot be explained using prior theory. We argue that the speedup comes from appropriate layer-wise growth enabled by sampling biases, which is more pronounced when the dataset size is smaller. We provide both theoretical analysis and empirical evidence from various interventions. Our results suggest that using a smaller dataset with more repetitions is not just a fallback strategy under data scarcity, but can be proactively leveraged as a favorable inductive biases for optimization, particularly in reasoning tasks.