跳转至

文章背景与核心概要

归一化 Transformer（nGPT）通过将模型参数向量和激活向量约束在单位超球面上，引入了超球面表示学习。本文提出了一套针对 nGPT 的实用训练配方，并在包含多达 300 亿总参数的现代混合 Mamba-2–Transformer 混合专家（MoE）模型上进行了评估。

该训练配方包含几个核心组件：Logit 梯度预处理（Logit Gradient Preconditioning）、对数学习率衰减（Logarithmic Learning Rate Decay）、GatedAdamW、角度更新控制以及可选的探索机制。核心发现在于：与采用标准 AdamW 训练的相同混合 MoE 架构的未归一化模型相比，300 亿参数的 nGPT 模型在达到相同的验证损失时，所需的训练 Token 数量大约减少了一半。

Training nGTP

Summary

The normalized Transformer (nGPT) introduces hyperspherical representation learning by constraining model parameter and activation vectors to the unit hypersphere. This paper presents a practical training recipe for nGPT, evaluated on modern hybrid Mamba-2–Transformer Mixture-of-Experts (MoE) models containing up to 30B total parameters.

Key components of the training recipe include: * Logit Gradient Preconditioning * Logarithmic Learning Rate Decay * GatedAdamW * Angular update control * Optional exploration mechanisms

Key Finding: Compared to unnormalized models of the same hybrid MoE architecture trained with standard AdamW, a 30B-parameter nGPT model achieves identical validation loss while requiring approximately half as many training tokens.

Paper Metadata

  • Title: Training nGPT
  • Authors: Ilya Loshchilov, Boris Ginsburg
  • Primary Subject: Machine Learning (cs.LG), with cross-listing in Artificial Intelligence (cs.AI)
  • arXiv Identifier: arXiv:2608.01284 [cs.LG]
  • DOI: 10.48550/arXiv.2608.01284
  • Submission History:
  • v1: August 2, 2026
  • v2 (last revised): September 1, 2026

Abstract

The normalized Transformer (nGPT) realizes hyperspherical representation learning by constraining model parameter vectors and activation vectors to the unit hypersphere. In this paper, we describe a practical training recipe for nGPT and evaluate it on modern hybrid Mamba-2–Transformer Mixture-of-Experts (MoE) models. The recipe introduces Logit Gradient Preconditioning, Logarithmic Learning Rate Decay, GatedAdamW, angular update control, and optional exploration mechanisms. Compared with an unnormalized model of the same hybrid MoE architecture trained with AdamW, the 30B-total-parameter nGPT model reaches the same validation loss using approximately half as many training tokens. The recipe scales across the models considered, which contain up to 30B total parameters.

Access & Resources