文章背景与核心概要
大语言模型（LLM）的动态层路由（Dynamic layer routing）是一种通过为单个Token选择性跳过某些层来降低推理成本的流行技术。然而，传统的路由方法将每个决策视为完全基于当前隐藏状态的孤立局部操作。这种方法忽视了跨网络深度的路由在序列上的路径依赖性，即早期的路由选择会直接影响下游的表征以及整体的层使用目标。
为了克服这一局限性，作者提出了历史感知路由（HeRo），这是一个动态路由框架，其核心在于引入了一个显式的路由器记忆机制，通过线性注意力机制追踪跨深度的路由状态。通过将路由决策同时建立在累积历史和当前隐藏状态的基础之上，HeRo在性能保持和计算效率方面均超越了现有的基线方法。
Do Dynamic Routers Need Memory? HeRo: History-Aware Routing for Efficient LLM Inference
- Authors: Hongjin Lin, Wentao Wan, Keze Wang
- Subjects: Artificial Intelligence (
cs.AI); Computation and Language (cs.CL) - ArXiv ID: arXiv:2609.08189
- Submitted: September 8, 2026
📌 Executive Summary
Dynamic layer routing is a popular technique for reducing the inference cost of Large Language Models (LLMs) by selectively skipping layers for individual tokens. However, conventional routing methods treat each decision as an isolated local operation based entirely on the current hidden state. This approach ignores the sequential and path-dependent nature of routing across network depth, where earlier routing choices directly influence downstream representations and overall layer-usage objectives.
To overcome this limitation, the authors introduce History-Aware Routing (HeRo), a dynamic routing framework featuring an explicit router memory mechanism that tracks routing states across depth via linear attention. By jointly conditioning routing decisions on both accumulated history and current hidden states, HeRo achieves superior performance retention and compute efficiency compared to existing baseline methods.
📋 Abstract
动态层路由通过学习为单个 Token 跳过层，从而降低大语言模型（LLM）的推理成本。然而，现有方法将每个路由决策视为仅以当前隐藏状态为条件的局部操作——这种设定忽略了跨深度路由的序列性和路径依赖性：先前的决策塑造了下游路由器看到的表征，并且层使用目标将所有决策紧密耦合在一起。
Dynamic layer routing reduces the inference cost of Large Language Models (LLMs) by learning to skip layers for individual tokens. Existing methods, however, treat each routing decision as a local operation conditioned solely on the current hidden state—a formulation that overlooks the sequential, path-dependent nature of routing across depth: earlier decisions shape the representations seen by downstream routers, and the layer-usage objective couples all decisions jointly.
我们提出了历史感知路由（HeRo），这是一个动态路由框架，通过引入路由器记忆机制来维护跨模型深度的显式路由状态，从而解决了这种不匹配问题。该记忆是通过线性注意力构建的，能够将先前的路由分数及其引发的残差更新增量聚合为紧凑的历史表征。在每个路由层，路由器将这个累积状态与当前的隐藏表征结合起来，共同决定所执行的分支。
We propose History-Aware Routing (HeRo), a dynamic routing framework that resolves this mismatch by introducing a router memory mechanism to maintain an explicit routing state across model depth. The memory is constructed via linear attention, incrementally aggregating preceding routing scores and their induced residual updates into a compact history representation. At each routed layer, the router conditions jointly on this accumulated state and the current hidden representation to select the executed branch.
通过针对 Token 级别的 FFN 路由进行实例化，HeRo 仅在冻结的主干网络上训练轻量级路由器和适配器，无需修改预训练参数。在 Llama 3.1-8B、Llama 2-7B 和 Llama 2-13B 上，HeRo 在十个基线中始终保持了最高的总体性能保留率： * 在 Llama 3.1-8B 上，它绕过了 26.87% 的模型参数，同时在七个基准测试中达到了密集模型性能的 100.24%。 * 在更严格的计算预算下，它保留了 97.01% 的性能，同时绕过了 38.82% 的参数。
Instantiated for token-wise FFN routing, HeRo trains only lightweight routers and adapters on a frozen backbone, requiring no modification to pretrained parameters. Across Llama 3.1-8B, Llama 2-7B, and Llama 2-13B, HeRo consistently achieves the highest aggregate performance retention among ten baselines: * On Llama 3.1-8B, it bypasses 26.87% of model parameters while achieving 100.24% of dense model performance across seven benchmarks. * Under a tighter computation budget, it retains 97.01% of performance while bypassing 38.82% of parameters.
消融实验证实，移除路由历史会持续导致性能下降——在多步推理和代码生成任务上尤为明显，这验证了显式路由记忆比仅依赖隐藏状态进行条件化能够实现更准确、更具自适应性的动态路由。
Ablation studies confirm that removing routing history consistently degrades performance—most notably on multistep reasoning and code generation tasks—validating that explicit routing memory enables more accurate and adaptive dynamic routing than solely conditioning on hidden states.
📊 Key Highlights & Results
- 方法论： 引入了一种使用线性注意力的显式路由器记忆，以在模型深度上增量聚合路由分数和残差更新。
- 效率： 仅训练轻量级的路由器和适配器，同时保持主干模型完全冻结。
- 卓越性能： 在多个基准架构（Llama 3.1-8B、Llama 2-7B 和 Llama 2-13B）上表现优于十个基线模型。
- 显著影响： 消融实验表明，路由历史对于多步推理和代码生成等复杂任务至关重要。
- Methodology: Introduces an explicit router memory using linear attention to incrementally aggregate routing scores and residual updates across model depth.
- Efficiency: Trains only lightweight routers and adapters while keeping the backbone model entirely frozen.
- Superior Performance: Outperformed ten baseline models across multiple benchmark architectures (Llama 3.1-8B, Llama 2-7B, and Llama 2-13B).
- Notable Impact: Ablations reveal that routing history is critical for complex tasks such as multistep reasoning and code generation.