跳转至

文章背景与核心概要

神经架构搜索（NAS）旨在自动化设计神经网络架构，从而减少对人工经验的依赖。在众多NAS方法中，可微分神经架构搜索（differentiable NAS）由于其高效性和高准确度而备受关注。然而，如何在保持模型参数最优性的同时优化架构泛化能力，一直是该领域的核心挑战。

本文提出了一种名为 LP-NAS（基于线性规划的神经架构搜索）的全新数学规划框架，适用于广泛的连续搜索空间。LP-NAS 巧妙地利用验证损失梯度和训练损失 Hessian 矩阵构建了一个线性规划（LP）问题，从而计算出能够同时提升泛化能力并保持模型参数最优性的架构更新方向。为了大幅提升计算效率，作者还推出了两个变体：S-LP-NAS 和 R-LP-NAS。

在 DARTS 搜索空间上的实验（即 S-LP-DARTS 和 R-LP-DARTS）表明，与标准 DARTS 算法相比，这些方法在早期的搜索迭代中表现出显著更快的收敛速度和更高的验证性能。在 CIFAR-10、CIFAR-100 和 ImageNet 数据集上进行的综合实验充分证实了该方法的卓越性能与强大的架构迁移能力。

LP-NAS: Linear Programming-based Neural Architecture Search

LP-NAS: Linear Programming-based Neural Architecture Search

Summary

Summary

LP-NAS (Linear Programming-based Neural Architecture Search) is a novel mathematical programming framework designed for differentiable neural architecture search (NAS). By formulating a linear program (LP) using the validation-loss gradient and the training-loss Hessian, the framework computes an optimal architecture update direction that enhances generalization while preserving model parameter optimality.

LP-NAS (Linear Programming-based Neural Architecture Search) is a novel mathematical programming framework designed for differentiable neural architecture search (NAS). By formulating a linear program (LP) using the validation-loss gradient and the training-loss Hessian, the framework computes an optimal architecture update direction that enhances generalization while preserving model parameter optimality.

To improve efficiency, the authors introduce two variants: S-LP-NAS and R-LP-NAS. When applied to the DARTS search space (S-LP-DARTS and R-LP-DARTS), these methods demonstrate significantly faster convergence and higher validation performance during early search iterations compared to the standard DARTS algorithm. Comprehensive experiments on CIFAR-10, CIFAR-100, and ImageNet confirm the approach's superiority and transferability.

To improve efficiency, the authors introduce two variants: S-LP-NAS and R-LP-NAS. When applied to the DARTS search space (S-LP-DARTS and R-LP-DARTS), these methods demonstrate significantly faster convergence and higher validation performance during early search iterations compared to the standard DARTS algorithm. Comprehensive experiments on CIFAR-10, CIFAR-100, and ImageNet confirm the approach's superiority and transferability.

Paper Metadata

Paper Metadata

  • arXiv Identifier: arXiv:2608.14472 [cs.LG]
  • Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
  • Submission Date: August 14, 2026
  • Authors: Abhishek Shukla, Ankur Sinha, Faiz Hamid
  • Comments: 20 pages, 5 figures
  • DOI: 10.48550/arXiv.2608.14472
  • arXiv Identifier: arXiv:2608.14472 [cs.LG]
  • Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
  • Submission Date: August 14, 2026
  • Authors: Abhishek Shukla, Ankur Sinha, Faiz Hamid
  • Comments: 20 pages, 5 figures
  • DOI: 10.48550/arXiv.2608.14472

Abstract

Abstract

神经架构搜索（NAS）旨在自动化神经网络架构设计，减少对人类专业知识的依赖。在各种 NAS 方法中，可微分 NAS 因其相比传统 NAS 方法更高的效率和准确性而崭露头角。由于可微分 NAS 将架构搜索空间松弛为连续域，因此可以将连续优化的原理应用于 NAS 中。

Neural Architecture Search (NAS) aims to automate neural network architecture design, reducing reliance on human expertise. Among the various NAS methods, differentiable NAS has gained prominence due to its efficiency and accuracy compared to conventional NAS approaches. Since differentiable NAS relaxes the architecture search space into a continuous domain, it is possible to apply principles from continuous optimization to NAS.

在本文中，我们提出了基于线性规划的 NAS（LP-NAS），这是一个用于可微分 NAS 的数学规划框架，适用于广泛的连续搜索空间。LP-NAS 利用验证损失梯度和训练损失 Hessian 矩阵构建线性规划（LP），以计算出既能提高泛化能力又能保持模型参数最优性的架构更新方向。通过遵循此 LP 导出的下降方向，LP-NAS 能够高效地遍历架构搜索空间，从而实现更快、更有效的架构优化。

In this paper, we propose Linear Programming-based NAS (LP-NAS), a mathematical programming-based framework for differentiable NAS that is applicable to a wide range of continuous search spaces. LP-NAS formulates a linear program (LP) using the validation-loss gradient and the training-loss Hessian to compute an architecture update direction that improves generalization while preserving the optimality of the model parameters. By following this LP-derived descent direction, LP-NAS efficiently navigates the architecture search space, leading to faster and more effective architecture optimization.

我们引入了 LP-NAS 的两个计算高效变体，即 S-LP-NAS 和 R-LP-NAS。将 LP-NAS 应用于可微分架构搜索（DARTS）搜索空间，产生了两个算法变体：S-LP-DARTS 和 R-LP-DARTS。与标准的 DARTS 算法相比，这两种变体在早期的搜索迭代中都实现了更快的收敛速度和显著更高的验证性能。在 CIFAR-10 和 CIFAR-100 上进行的广泛实验表明，LP-DARTS 在架构搜索和评估阶段均优于标准 DARTS。此外，我们在 CIFAR-10 数据集上将我们的方法与几种 DARTS 变体（P-DARTS、PC-DARTS 和 STO-DARTS）进行了对比，证明了其有效性。同时，我们通过在 ImageNet 数据集上的实验验证了所发现架构的可迁移性。

We introduce two computationally efficient variants of LP-NAS, namely S-LP-NAS and R-LP-NAS. Applying LP-NAS to the Differentiable Architecture Search (DARTS) search space results in two algorithmic variants, S-LP-DARTS and R-LP-DARTS. Both variants achieve faster convergence and significantly higher validation performance during the early search iterations than the standard DARTS algorithm. Extensive experiments on CIFAR-10 and CIFAR-100 show that LP-DARTS outperforms standard DARTS in both the architecture search and evaluation phases. Additionally, we compare our approach with several DARTS variants (P-DARTS, PC-DARTS, and STO-DARTS) on the CIFAR-10 dataset and demonstrate its effectiveness. Furthermore, we validate the transferability of the discovered architectures through experiments on the ImageNet dataset.