跳转至

文章背景与核心概要

本文探讨了大语言模型（LLM）预训练是否能够被分解为更小、可独立训练的任务，并在后期重新组合成一个连贯的更大模型。作者提出了“训练混合”（Mixture of Training, MoT）这一新型模块化预训练流程，将目标 Transformer 划分为连续的层块，在冻结的预训练“对齐器支架”（aligner scaffold）中进行训练，最后将训练好的块进行重组，并辅以可选的短端到端适应阶段。

通过在 C4 数据集上训练 1.3B 参数的 Gemma 风格模型，MoT 验证了其机制的可行性：独立训练的深度切片可以重新组合成可用的语言模型，且在质量平价调度下，能够达到与单体基线模型相同的报告困惑度。虽然 MoT 目前并非旨在完全替代单体预训练，但它为研究“支架式子运行如何充当可复用的训练单元”提供了一个极具潜力的小规模框架。

训练混合：将小规模支架式预训练运行重新组合为更大的语言模型 (Mixture of Training: Recombining Small-Scale Scaffolded Pretraining Runs into a Larger Language Model)

作者： Mohammed Sabry, Sean Augenstein, Keith Rush, Lucio Dery
日期： 2026年8月13日
会议： 已被 COLM 2026 小型规模方法与机遇研讨会 (MOSS) 接受
arXiv 链接： 2608.13277

摘要 (Summary)

本文介绍了训练混合（Mixture of Training, MoT），这是一种新颖的模块化预训练程序，旨在将大语言模型的训练分解为更小、独立且可并行的任务。通过将 Transformer 划分为连续的层块，并在冻结的预训练“对齐器支架”中对其进行训练，作者证明了这些块可以被有效地重新组合成一个连贯、功能齐全的模型。虽然 MoT 并非旨在直接替代单体预训练，但它作为一种概念验证，展示了利用支架式子运行作为可复用训练单元的可行性，并通过对齐器支架的复用展现出潜在的计算优势。

This paper introduces Mixture of Training (MoT), a novel modular pre-training procedure designed to decompose the training of large language models into smaller, independent, and parallelizable jobs. By partitioning a Transformer into contiguous layer blocks and training them within a frozen, pre-trained "aligner scaffold," the authors demonstrate that these blocks can be effectively recomposed into a coherent, functional model. While not intended as a direct replacement for monolithic pre-training, MoT serves as a proof-of-concept for using scaffolded sub-runs as reusable training units, offering potential compute advantages through the reuse of the aligner scaffold.

抽象 (Abstract)

我们探讨了语言模型预训练是否可以被分解为更小、可独立训练的任务，这些任务随后可以重新组合成一个连贯的更大模型。我们引入了训练混合（MoT），这是一种支架式模块化预训练程序，它将目标 Transformer 划分为连续的层块，在冻结的预训练对齐器支架内训练每个块，然后通过可选的简短端到端适应阶段将训练好的块重新组合。

在一个在 C4 上训练的 1.3B 参数 Gemma 风格模型上，MoT 提供了小规模的机制证明：独立训练的深度切片可以重新组合成可用的语言模型，并且质量平价调度达到了与单体基线相同的报告困惑度。这种平价设置在对齐器准备后处理了更多的聚合令牌（tokens），并且具有更短的理想化层等效关键路径；其有效的计算优势取决于跨运行复用对齐器。因此，我们提出 MoT 不是作为单体预训练的通用替代品，而是作为一个小规模框架，用于研究支架式子运行是否可以充当可复用的训练单元。

We ask whether language-model pre-training can be decomposed into smaller, independently trainable jobs that can later be recomposed into a coherent larger model. We introduce Mixture of Training (MoT), a scaffolded modular pre-training procedure that partitions a target Transformer into contiguous layer blocks, trains each block inside a frozen pretrained aligner scaffold, and then recomposes the trained blocks with an optional short end-to-end adaptation pass.

On a 1.3B-parameter Gemma-style model trained on C4, MoT provides a small-scale proof of mechanism: independently trained depth slices can be recomposed into a usable language model, and a quality-parity schedule reaches the same reported perplexity as the monolithic baseline. This parity setting processes more aggregate tokens and has a shorter idealized layer-equivalent critical path after aligner preparation; its effective compute advantage depends on reusing the aligner across runs. We therefore present MoT not as a general replacement for monolithic pre-training, but as a small-scale framework for studying whether scaffolded sub-runs can act as reusable training units.

访问与资源 (Access & Resources)

元数据 (Metadata)

  • Primary Subject: Computation and Language (cs.CL)
  • Secondary Subject: Artificial Intelligence (cs.AI)
  • DOI: 10.48550/arXiv.2608.13277