跳转至

文章背景与核心概要

在大语言模型（LLM）的训练过程中，选择最优的并行策略（如配置数据并行、张量并行、流水线并行的并行度，以及微批次和全局批次大小）对于训练效率至关重要。传统的先进方法依赖于离线搜索来寻找能够最小化单次迭代时间的单一静态策略，却从根本上忽视了其对验证困惑度（validation perplexity）以及达到目标困惑度所需时间（TTP）的影响。本文指出，在整个训练过程中，最大化困惑度改善的最佳策略实际上会发生多次转变，因此静态方法在 TTP 性能上落后动态策略选择 1.8 倍到 11.4 倍。

为了解决这一问题，作者推出了 CONA——一种利用策略链接的全新在线训练方法。CONA 利用从计算吞吐量和梯度统计量中衍生出的自定义替代指标，在实时运行中对候选策略进行动态排序，并在训练过程中无缝过渡到性能更高的策略。在 GPT-3 (1.3B)、BERT-Large 和 Llama-3.2 (1.0B) 等架构上的评估表明，CONA 实现目标验证困惑度的速度比现有先进方法快 1.4 倍到 9.6 倍，且紧密逼近迭代最优策略选择的理论上限（差距在 2.6% 以内）。

Parallelism Strategy Chaining for Fast Training Convergence

Authors: Minchul Kang, Changyong Shin, Younghun Go, Hyunho Lee, Jinwoo Jeong, Chuck Yoo, Gyeongsik Yang
Primary Field: Machine Learning (cs.LG)
Additional Categories: Artificial Intelligence (cs.AI); Distributed, Parallel, and Cluster Computing (cs.DC)
Submission Date: September 7, 2026
Conference: Accepted to EMNLP 2026 Main Conference
Identifiers: arXiv:2609.07236 [cs.LG] | DOI: 10.48550/arXiv.2609.07236

📌 Summary

Selecting an optimal parallelism strategy—such as configuring degrees of data, tensor, and pipeline parallelism alongside micro- and global-batch sizes—is critical for the training efficiency of large language models (LLMs). Traditional state-of-the-art methodologies rely on offline searches to find a single static strategy that minimizes per-iteration time, fundamentally overlooking the impact on validation perplexity and time-to-perplexity (TTP).

This paper demonstrates that the ideal strategy for maximizing perplexity improvement actually shifts multiple times throughout the training process. Consequently, static approaches lag behind dynamic strategy selection by 1.8× to 11.4× in TTP. To resolve this, the authors introduce CONA, a novel online training method that utilizes strategy chaining. CONA dynamically ranks candidate strategies in real-time using a custom surrogate metric derived from compute throughput and gradient statistics, seamlessly transitioning to higher-performing strategies during training.

Evaluated across architectures like GPT-3 (1.3B), BERT-Large, and Llama-3.2 (1.0B), CONA achieves target validation perplexities 1.4× to 9.6× faster than existing state-of-the-art methods, closely tracking the theoretical upper bound of iteratively optimal strategy selection within a 2.6% margin.