跳转至

在 MXFP4 下训练 MoE 大语言模型：如何缩小与 BF16 的精度差距

Training an MoE LLM in MXFP4: How We Closed the Gap to BF16

arXiv 论文编号 / arXiv Identifier: arXiv:2610.32229 [cs.AI]
论文作者 / Authors: Han Wang, Zhitao Wang, Natalia Frumkin, Xuanwu Yin, Alireza Khodamoradi, Dong Li, Emad Barsoum
学科分类 / Primary Subject: 人工智能 / Artificial Intelligence (cs.AI)
提交时间 / Submission Date: 23 October 2026 (Last revised: 26 October 2026)
硬件评测环境 / Hardware Environment: 基于 AMD MI300X 硬件的模拟 MXFP4 算子（实际物理耗时尚未专门做底层工程优化） / Simulated MXFP4 kernels on AMD MI300X hardware (wall-clock speed has not yet been optimized).

文章背景与核心概要

随着大模型训练对显存带宽与算力吞吐的要求日益苛刻，将浮点精度压缩至 8 位以下的低精度格式（如 4 位浮点数 FP4）成为了学术界与工业界竞相探索的前沿方向。然而，极低的数值精度带来的剧烈量化误差极易引发训练不收敛与精度雪崩，如何在极度嘈杂的 FP4 环境中保持与标准 16 位浮点 (BF16) 相当的训练质量是当前的核心挑战。针对这一技术瓶颈，研究人员开源了专为混合专家模型 (Mixture-of-Experts, MoE) 打造的训练配方 ALTO，在拥有 200 亿参数的 GPT-OSS-20B 模型上成功落地了微缩放浮点格式 MXFP4。通过融合混合块量化、随机阿达马变换 (Randomized Hadamard Transforms, RHT)、随机舍入 (Stochastic Rounding, SR) 与创新的权重去震荡方案，该方法成功抹平了约 50% 与 BF16 的精度鸿沟，并将达到预定质量目标的收敛步数额外开销从 +20% 骤降至 +5%。这一突破性工作证实了超低精度 FP4 在超大规模稀疏架构模型训练中的巨大可行性，为下一代低成本绿色 AI 基础设施建设指明了方向。

核心执行概要

Executive Summary

使用低于 8 位的超低精度算术体系训练大语言模型 (Large Language Model, LLM)，在大幅释放显存带宽潜力以及显著提升计算吞吐量方面展现出了巨大的吸引力。然而，4 位浮点数 (FP4) 格式所引入的极端量化误差，往往会导致优化过程严重失稳甚至发散。

Training Large Language Models (LLMs) in sub-8-bit arithmetic presents significant potential for boosting memory bandwidth and compute throughput. However, the extreme quantization errors introduced by FP4 formats typically destabilize optimization.

针对这一技术瓶颈，本文隆重推出了 ALTO——一套开源训练技术方案，专为在 MLPerf Small MoE 训练基准评测中使用 MXFP4 微缩放格式 (Microscaling Format) 训练拥有 200 亿参数的混合专家模型 (Mixture-of-Experts, MoE) GPT-OSS-20B 而量身定制。通过巧妙结合混合块量化、随机阿达马变换 (Randomized Hadamard Transforms, RHT)、随机舍入 (Stochastic Rounding, SR) 以及简化的权重去震荡 (Weight De-oscillation) 机制，研究团队成功抹平了与标准 BF16 基准线之间约 50% 的精度差距。

This paper introduces ALTO, an open-source training recipe designed to train GPT-OSS-20B—a 20-billion-parameter mixture-of-experts (MoE) model—using the MXFP4 microscaling format on the MLPerf Small MoE Training benchmark. By integrating hybrid block quantization, randomized Hadamard transforms, stochastic rounding, and a simplified weight de-oscillation scheme, the researchers successfully closed approximately 50% of the accuracy gap to the BF16 baseline.

方法论：ALTO 训练技术配方

Methodology: The ALTO Recipe

为了稳固 FP4 极低精度下的训练稳定性，并弥合与传统 16 位浮点精度之间的表现差距，ALTO 紧密结合了数项业界前沿的技术手段：

To stabilize FP4 training and bridge the gap to traditional 16-bit precisions, ALTO combines several cutting-edge techniques:

  1. 量化基础构建：采用混合 1D/2D 块量化策略（改良自 Abecassis 等人的方案）；
  2. 误差消除机制：运用随机阿达马变换 (Randomized Hadamard Transforms, RHT) 与随机舍入 (Stochastic Rounding, SR) 来有效抵御低精度噪声带来的扰动；
  3. 权重稳定调控：引入并简化了源自 TetraJet-v2 (arXiv:2510.27527) 的权重去震荡方案 (Weight De-oscillation Scheme)。
  1. Quantization Foundations: Employs hybrid 1D/2D block quantization (adapted from Abecassis et al.).
  2. Error Mitigation: Utilizes Randomized Hadamard Transforms (RHT) and Stochastic Rounding (SR) to handle low-precision noise.
  3. Weight Stabilization: Integrates a simplified weight de-oscillation scheme adapted from TetraJet-v2 (arXiv:2510.27527).

评测结果与模型性能表现

Results & Performance

  • 验证集损失 (C4 数据集)：在全局批大小 (Global Batch Size) 为 16、训练步数为 16,128 步的设定下，集成了 MXFP4 + RHT + SR + 权重去震荡 的完整流程取得了 3.3350 的验证集损失 (Validation Loss)；
  • 极度逼近 Baseline：该数值距离纯 16 位精度基线 (BF16 Baseline) 的 3.3283 仅有 0.007 的极微弱差距；
  • 去震荡机制的显著功效：若未开启权重去震荡，MXFP4 训练的验证损失停留在 3.3418；去震荡方案成功挽回了近 50% 的性能落差；
  • 收敛额外开销大幅收窄：以达到 MLPerf 质量目标（验证损失 3.34）所需的迭代步数计算，去震荡机制将 FP4 相对 BF16 的额外收敛步数开销从 +20.0% 骤降至仅 +5.0%。
  • Validation Loss (C4 Dataset): At a global batch size of 16 and 16,128 steps, the full MXFP4 + RHT + SR + de-oscillation pipeline achieved a validation loss of 3.3350.
  • Proximity to Baseline: This sits just 0.007 above the BF16 baseline of 3.3283.
  • Impact of De-Oscillation: Without de-oscillation, MXFP4 hit a validation loss of 3.3418. The de-oscillation scheme successfully recovered roughly 50% of the performance gap.
  • Convergence Overhead: Measured by the steps required to hit the MLPerf quality target (validation loss of 3.34), de-oscillation reduced the FP4 convergence overhead relative to BF16 from +20.0% down to +5.0%.

负向探索结果（被证实无效的技术手段）

Negative Results (What Didn't Work)

尽管在算子级别的信噪比 (Signal-to-Noise Ratio, SNR) 测试中观察到了一定收益，但作者指出，以下技术在端到端 (End-to-End) 训练中均表现出负向效果：

Despite observing operator-level Signal-to-Noise Ratio (SNR) gains, the authors reported negative end-to-end results for the following techniques:

  • 差分梯度估计 (Differential Gradient Estimation)
  • 离群值截断 (Outlier Clipping)
  • 宏块缩放 (Macro-block Scaling)
  • Differential gradient estimation
  • Outlier clipping
  • Macro-block scaling