跳转至

文章背景与核心概要

在大语言模型（LLMs）的微调和域自适应过程中，传统的反向传播（Backpropagation）依赖于复杂的自动微分图构建以及高昂的跨层梯度计算开销，这在面对超大规模参数时往往会导致严重的显存瓶颈和吞吐量下降。本文介绍了一种名为“仅前向传递（FPO）MLP训练”的创新方法，旨在彻底摆脱对跨层反向传播的依赖。

该研究的核心技术在于发现：在变压器（Transformer）的后期层中，输出层的预测误差可以作为真实梯度的可靠近似（在六个被调查的模型中，其余弦相似度保持在 0.47 到 0.59 之间）。通过在输出层计算单一误差信号并直接将其应用于目标层，FPO消除了自动微分图构建的需求。实验表明，该方法不仅将标准微调的吞吐量提升了 2.7 至 3.2 倍，还将峰值训练内存减少了约 40%，同时在域外基准测试中保持了与种子噪声相当的模型性能稳定性。

Forward Pass Domain Adaptation (Without Cross-Layer Backpropagation)

Authors: Rivaan Patil, Simon Dennis, Hao Guo, Kevin Shabahang
Date: 26 May 2026
arXiv ID: 2608.14563
Subject: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)

Authors: Rivaan Patil, Simon Dennis, Hao Guo, Kevin Shabahang
Date: 26 May 2026
arXiv ID: 2608.14563
Subject: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)

Summary

This paper introduces Forward-Pass-Only (FPO) MLP training, a novel method for adapting large language models (LLMs) that bypasses the need for cross-layer backpropagation. By computing a single error signal at the output layer and applying it directly to target layers, FPO eliminates the need for autograd graph construction. This approach achieves 2.7–3.2x the throughput of standard fine-tuning while reducing peak training memory by approximately 40%. Crucially, the method maintains model performance on off-domain benchmarks within seed-noise levels, a stability often lacking in full-network fine-tuning.

Summary

This paper introduces Forward-Pass-Only (FPO) MLP training, a novel method for adapting large language models (LLMs) that bypasses the need for cross-layer backpropagation. By computing a single error signal at the output layer and applying it directly to target layers, FPO eliminates the need for autograd graph construction. This approach achieves 2.7–3.2x the throughput of standard fine-tuning while reducing peak training memory by approximately 40%. Crucially, the method maintains model performance on off-domain benchmarks within seed-noise levels, a stability often lacking in full-network fine-tuning.

Key Contributions

Key Contributions

1. Empirical Foundation

The FPO method is built on the observation that in late transformer layers, the output-layer prediction error serves as a reliable approximation of the true gradient, exhibiting a cosine similarity of 0.47–0.59 across six surveyed models.

1. Empirical Foundation

The FPO method is built on the observation that in late transformer layers, the output-layer prediction error serves as a reliable approximation of the true gradient, exhibiting a cosine similarity of 0.47–0.59 across six surveyed models.

2. Diagnostic Tool

The authors provide a two-minute diagnostic procedure that quantifies this gradient approximation per layer, allowing researchers to identify optimal target layers for adaptation in any given model.

2. Diagnostic Tool

The authors provide a two-minute diagnostic procedure that quantifies this gradient approximation per layer, allowing researchers to identify optimal target layers for adaptation in any given model.

3. Performance and Efficiency

  • Efficiency: No signal is propagated between layers, and no autograd graph is constructed.
  • Validation: Tested on OLMo-2-7B, Qwen3-8B, and Falcon3-7B.
  • Results: FPO consistently improves in-domain perplexity while keeping performance on standard benchmarks (MMLU, ARC-Challenge, HellaSwag, and Winogrande) within baseline seed-noise.
  • Comparison: While localizing Standard Fine-Tuning (SFT) to FPO’s target layers is possible, it incurs a 2.2x higher wall-clock cost compared to the FPO approach.

3. Performance and Efficiency

  • Efficiency: No signal is propagated between layers, and no autograd graph is constructed.
  • Validation: Tested on OLMo-2-7B, Qwen3-8B, and Falcon3-7B.
  • Results: FPO consistently improves in-domain perplexity while keeping performance on standard benchmarks (MMLU, ARC-Challenge, HellaSwag, and Winogrande) within baseline seed-noise.
  • Comparison: While localizing Standard Fine-Tuning (SFT) to FPO’s target layers is possible, it incurs a 2.2x higher wall-clock cost compared to the FPO approach.

Access & Resources

Access & Resources

license icon

license icon

Submission History

  • [v1] Tue, 26 May 2026 15:23:57 UTC

Submission History

  • [v1] Tue, 26 May 2026 15:23:57 UTC