跳转至

文章背景与核心概要

在非独立同分布（non-IID）数据上的联邦学习通常旨在寻找平坦极小值（flat minima），以提升跨客户端的泛化能力。虽然现有方法大多将集中式训练中的尖锐度感知最小化（SAM）迁移过来，但寻找平坦极小值还有另一条途径：通过直接向参数更新中添加噪声来获得免费的正则化。然而，这种基于噪声的正则化方法此前一直未能成功应用到联邦学习场景中。

本文深入探讨了这一现象背后的原因。作者证明了更新掩码（update masking）会惩罚优化器向尖锐方向移动的行为。具体而言：当每个客户端使用各自独立的掩码时，联邦学习的平均化操作会按群组大小（cohort size）削弱正则化效力；而如果为所有客户端分配完全相同的掩码，则能通过群组逆梯度多样性的因子来恢复该效力。实验表明，在 CIFAR-10 设置下，共享掩码因子平均仅为 1.19（最大为 10）。关闭小批量采样（minibatch sampling）会将其提高到 8.96，而将数据异构性改变百倍则将其约束在 1.17 到 1.50 之间。最终，那些能够成功保留正则化的配置在训练阶段的表现往往极差，不具备实际应用价值。

How Much Regularization Survives Averaging? Update Masking in Federated Learning

arXiv: 2608.23286 [cs.LG]
Submitted: August 24, 2026
Authors: Wenhao Yan, Fu Kuroda, Yucheng Jin, Zhenke Chen
Subjects: Machine Learning (cs.LG), Artificial Intelligence (cs.AI)

📋 Summary

Federated learning on non-IID data typically aims to find flat minima to improve generalization across clients. While existing methods adapt sharpness-aware minimization (SAM) from centralized training, an alternative path to flat minima exists: obtaining free regularization through noise added directly to parameter updates.

This paper investigates why this noise-based regularization has not successfully transitioned to the federated setting. The authors demonstrate that update masking penalizes optimizers for moving in sharp directions. Specifically: * Client-specific masks: When each client uses its own independent mask, federated averaging weakens the regularization charge by the cohort size. * Shared masks: Assigning every client the exact same mask restores the charge by a factor equal to the inverse gradient diversity of the cohort. * Experimental Findings (CIFAR-10): The shared mask factor averages \(1.19\) out of a maximum of \(10\). Disabling minibatch sampling raises this factor to \(8.96\), whereas varying data heterogeneity by a hundredfold keeps it constrained between \(1.17\) and \(1.50\). Ultimately, the configurations that successfully preserve regularization perform too poorly during training to be practically viable.

📑 Abstract

联邦学习在非独立同分布（non-IID）数据上旨在寻找平坦极小值以实现跨客户端的泛化，现有的方法借鉴了集中式训练中的尖锐度感知最小化（sharpness-aware minimization）。通往平坦极小值的第二种途径是：从添加到参数更新中的噪声中“免费”获得正则化，但这一方法从未成功迁移到联邦学习中。我们展示了其中的原因。掩码会惩罚优化器向尖锐方向移动。我们证明了：当每个客户端绘制自己的掩码时，联邦学习的平均化操作会精确地按群组大小削弱该惩罚；而当赋予每个客户端相同的掩码时，它会按群组的逆梯度多样性因子恢复该惩罚。在我们于 CIFAR-10 上的实验设置中，该因子在最大可能值 10 中仅为 1.19。关闭小批量采样将其提高到 8.96，而将数据异构性改变百倍则使其保持在 1.17 到 1.50 之间。能够保留正则化的配置在训练时的表现太差，以至于无法使用。

Federated learning on non-IID data seeks flat minima to generalize across clients, and existing methods borrow sharpness-aware minimization from centralized training. There is a second way to reach flat minima, in which the regularization comes for free from noise added to the parameter updates, and it has never been carried over to the federated setting. We show the reason. Masking charges the optimizer for moving in sharp directions. We prove that when each client draws its own mask, federated averaging weakens that charge by exactly the cohort size, and that giving every client the same mask brings it back by a factor equal to the inverse gradient diversity of the cohort. In our experiment setting on CIFAR-10, that factor is 1.19 out of a possible 10. Turning off minibatch sampling raises it to 8.96, while changing data heterogeneity a hundredfold leaves it between 1.17 and 1.50. The configurations keeping the regularization train far too poorly to use.

链接与资源：

📚 References & External Citations

参考文献与外部引文：