跳转至

文章背景与核心概要

现代大规模推荐系统（如点击率预估模型）在训练过程中，传统的端到端前向和反向传播常由于严格的依赖顺序而面临硬件利用率不足的问题。本文介绍了一种名为 ERASE（EaRly bAckpropagation SchEdule）的新型训练技术，旨在打破这种全局依赖链，从而大幅提升训练吞吐量。

ERASE 的核心创新在于将前向-前向（FF）网络中的解耦（detachment）机制重新定义为一个调度原语。通过截断模块输出并解除下游梯度依赖，系统能够在前向传播局部完成后立即启动反向传播，并通过独立的 CUDA 流与后续的前向计算进行重叠执行。在大规模 CTR 模型上的实验表明，该技术在保持模型性能的同时，可将吞吐量提升高达 9.51%。

ERASE: EaRly bAckpropagation SchEdule for Faster Training of Modern Recommendation Systems

Authors: Ergan Shang, Flavio Sales Truzzi
Date: August 19, 2026
Primary Subject: Machine Learning (cs.LG)
arXiv ID: 2608.18469

Summary

The paper introduces ERASE (EaRly bAckpropagation SchEdule), a novel training technique designed to accelerate the training of modern recommendation systems.

本文介绍了 ERASE（EaRly bAckpropagation SchEdule，早期反向传播调度），这是一种旨在加速现代推荐系统训练的新型训练技术。

Traditional training methods treat forward and backward passes as disjoint phases, leading to underutilized hardware capacity. ERASE reinterprets the detachment mechanism from Forward-Forward (FF) networks as a scheduling primitive. By detaching a block's output, the system removes downstream gradient dependencies, allowing the backward pass to be initiated immediately upon the completion of the forward pass. These backward passes are launched on separate CUDA streams, enabling them to overlap with subsequent forward computations. Experimental results on large-scale click-through-rate (CTR) models demonstrate a throughput improvement of up to 9.51% while maintaining model performance.

传统的训练方法将前向传播和反向传播视为互不相交的阶段，导致硬件算力未得到充分利用。ERASE 重新诠释了前向-前向（FF）网络中的解耦机制，将其作为调度原语。通过解耦某个模块的输出，系统去除了下游的梯度依赖，使得反向传播能够在对应前向传播完成后立即启动。这些反向传播任务会在独立的 CUDA 流上启动，从而能够与后续的前向计算实现并行重叠。在大规模点击率（CTR）模型上的实验结果表明，在保持模型性能的同时，吞吐量提升高达 9.51%。

Key Concepts

The Inefficiency of Conventional Training

In standard deep learning pipelines, the backward pass must wait for the entire forward pass to complete, and vice versa. This creates "bubbles" in the execution pipeline where hardware accelerators (like GPUs) are not fully utilized, particularly in lightweight proxy models where kernel sizes are small.

传统训练的低效性

在标准的深度学习流水线中，反向传播必须等待整个前向传播完成，反之亦然。这会在执行流水线中产生“气泡”（bubbles），导致硬件加速器（如 GPU）无法被充分利用，尤其是在内核规模较小的轻量级代理模型中更为明显。

The ERASE Mechanism

  • Detachment as a Primitive: By using a local objective to detach a block's output, the model breaks the global dependency chain.
  • Early Backpropagation: Once a block is detached, its backward pass is no longer dependent on the rest of the network's forward pass.
  • Concurrent Execution: ERASE schedules these ready-to-run backward passes on separate CUDA streams, effectively filling the "spare capacity" of the device with concurrent forward and backward operations.

ERASE 机制

  • 作为原语的解耦： 通过使用局部目标函数来解耦模块的输出，模型打破了全局依赖链。
  • 早期反向传播： 一旦某个模块被解耦，其反向传播将不再依赖于网络其余部分的前向传播。
  • 并发执行： ERASE 将这些准备就绪的反向传播任务调度到不同的 CUDA 流上，有效地利用前向和反向操作的并发执行来填补设备的“闲置算力”。

Performance Highlights

  • Throughput: Achieved up to a 9.51% increase in training speed on large-scale CTR models.
  • Stability: The method maintains normalized entropy levels comparable to baseline models, ensuring that the speed gains do not come at the cost of model quality.
  • Constraint: The authors note that the effectiveness of this overlap is limited by device saturation; if a kernel fully saturates the accelerator, there is no remaining capacity for concurrent execution.

性能亮点

  • 吞吐量： 在大规模 CTR 模型上，训练速度实现了高达 9.51% 的提升。
  • 稳定性： 该方法保持了与基线模型相当的归一化熵水平，确保速度的提升不会以牺牲模型质量为代价。
  • 限制条件： 作者指出，这种重叠执行的有效性受到设备饱和度的限制；如果某个内核完全占满了加速器，则没有多余的算力来进行并发执行。

Accessing the Paper

论文访问