跳转至

文章背景与核心概要

随着混合专家（MoE）模型在保持计算高效性的同时提供高模型容量，将其部署到边缘设备上的核心瓶颈逐渐显现，主要受限于严格的内存约束。由于专家参数规模庞大，受限于容量、成本和功耗，它们通常只能存放在片外内存中，这使得专家加载过程直接成为性能的关键瓶颈。

为了解决这一痛点，本文引入了 APEX（自适应专家预取，Adaptive Expert Prefetching） 这一预测性资源管理框架，旨在将专家加载与有效计算进行重叠。通过使用由学习型置信度模型驱动的轻量级预取路由器，APEX 能够在注意力块执行之前预测候选专家，动态获取额外的专家，并实现超过 99% 的重叠准确率（显著优于传统的固定 top-\(k\) 预取方法）。

APEX: Adaptive Expert Prefeksing for Memory-Efficient Edge MoE Inference

APEX: Adaptive Expert Prefetching for Memory-Efficient Edge MoE Inference

Authors: Alish Kanani, Layan Badawi, Umit Y. Ogras
Subjects: Hardware Architecture (cs.AR); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
Publication: Accepted at IEEE/ACM ESWEEK (CODES) 2026; official version to appear in IEEE TCAD
arXiv: 2608.11688 [cs.AR]

Authors: Alish Kanani, Layan Badawi, Umit Y. Ogras
Subjects: Hardware Architecture (cs.AR); Artificial Intelligence (cs.AI); Machine Learning (cs.LG)
Publication: Accepted at IEEE/ACM ESWEEK (CODES) 2026; official version to appear in IEEE TCAD
arXiv: 2608.11688 [cs.AR]

📋 Summary

📋 Summary

Mixture-of-Experts (MoE) models offer high model capacity with high compute efficiency, but running them on edge devices is fundamentally bottlenecked by memory constraints. Because expert parameters are massive, they typically reside in off-chip memory due to capacity, cost, and power limitations, placing expert loading directly on the critical path.

Mixture-of-Experts (MoE) models offer high model capacity with high compute efficiency, but running them on edge devices is fundamentally bottlenecked by memory constraints. Because expert parameters are massive, they typically reside in off-chip memory due to capacity, cost, and power limitations, placing expert loading directly on the critical path.

This paper introduces APEX (Adaptive Expert Prefetching), a predictive resource management framework designed to overlap expert loading with useful computation. By using a lightweight prefetch router driven by a learned confidence model, APEX predicts candidate experts before the attention block, dynamically fetching additional experts and achieving over 99% overlap accuracy (significantly outperforming traditional fixed top-\(k\) prefetching).

This paper introduces APEX (Adaptive Expert Prefetching), a predictive resource management framework designed to overlap expert loading with useful computation. By using a lightweight prefetch router driven by a learned confidence model, APEX predicts candidate experts before the attention block, dynamically fetching additional experts and achieving over 99% overlap accuracy (significantly outperforming traditional fixed top-\(k\) prefetching).

🔑 Key Features & Execution Modes

🔑 Key Features & Execution Modes

APEX supports two specialized execution modes to optimize edge MoE inference:

APEX supports two specialized execution modes to optimize edge MoE inference:

  1. Correctness-Preserving Mode: Guarantees exact routing semantics while reducing per-token latency by up to 26% and improving the Energy-Delay Product (EDP) by up to 41% over state-of-the-art baselines.
  2. Stall-Free Mode: Eliminates residual stalls by operating strictly on available experts, delivering additional efficiency gains with negligible impact on application accuracy.
  1. Correctness-Preserving Mode: Guarantees exact routing semantics while reducing per-token latency by up to 26% and improving the Energy-Delay Product (EDP) by up to 41% over state-of-the-art baselines.
  2. Stall-Free Mode: Eliminates residual stalls by operating strictly on available experts, delivering additional efficiency gains with negligible impact on application accuracy.