跳转至

文章背景与核心概要

在大模型技术飞速发展的背景下，如何将大型、高性能模型的知识高效迁移到较小的模型中，一直是学术界和工业界关注的核心问题。传统的知识蒸馏或模型微调方法通常需要消耗大量的计算资源，且在面对不同规模和架构的异构模型时显得力不从心。本文介绍了一种名为激活-剪枝-合并（Activation-Prune-Merge, APM）的全新无训练框架，旨在无需显式神经元级语义对齐的情况下，实现跨模型规模的知识无缝融合。

该技术的核心创新在于利用“激活引导映射”技术，从“供体（donor）”大模型中精准识别并提取出关键的功能组件（如显著层、隐藏维度、注意力头和MLP神经元），然后通过微插值系数将其注入到“受体（recipient）”小模型中。在包含推理、数学、代码生成和指令遵循等16个多样化基准测试上的实验结果表明，APM能够显著提升小模型的性能，例如将3B参数的受体模型平均准确率从55.5%大幅提升至60.6%，并在多项具体任务中展现出卓越的增益效果。

Training-Free Knowledge Transfer Across Model Scales through Activation-Guided Pruning

arXiv ID: 2608.13596
Authors: Jiahe Fan, Si Chen, Yinghao Hou, Aiyuan Zhang, Hong Xie
Date: July 31, 2026
Subject: Machine Learning (cs.LG)

Summary

本文介绍了激活-剪枝-合并（Activation-Prune-Merge, APM），这是一个新颖的、无需训练的框架，旨在通过从更大、更强大的模型中转移知识来改进较小的语言模型。尽管模型规模之间存在显著的架构不匹配，但APM成功实现了模型的融合，且不需要显式的神经元级语义对齐。通过使用激活引导映射，该框架识别并提取出“供体（donor）”模型中的功能组件，然后使用微插值系数将它们注入到“受体（recipient）”模型中。在16个基准测试上的实验结果证明了显著的性能提升，特别是将一个3B参数的受体模型的平均准确率从55.5%显著提升至60.6%。

This paper introduces Activation-Prune-Merge (APM), a novel, training-free framework designed to improve smaller language models by transferring knowledge from larger, more powerful models. Despite significant architectural mismatches between model scales, APM successfully fuses models without requiring explicit neuron-wise semantic alignment. By using activation-guided mapping, the framework identifies and extracts functional components from a "donor" model and injects them into a "recipient" model using a micro-interpolation coefficient. Experimental results across 16 benchmarks demonstrate significant performance gains, notably increasing average accuracy from 55.5% to 60.6% for a 3B parameter recipient model.

Key Contributions

异构融合的挑战

作者指出了结合在规模和架构上存在差异的模型的困难所在。传统方法通常需要复杂的对齐或微调；本研究探讨了是否可以通过将供体模型视为集中功能组件的来源，来直接转移功能能力。

The Challenge of Heterogeneous Fusion

The authors address the difficulty of combining models that differ in scale and architecture. Traditional methods often require complex alignment or fine-tuning; this research investigates whether functional capabilities can be transferred directly by treating the donor model as a source of concentrated functional components.

APM 框架

激活-剪枝-合并（APM） 过程包含三个主要步骤： 1. 任务条件激活映射： 分析供体模型，以识别显著层、隐藏维度、注意力头和MLP神经元。 2. 结构剪枝： 根据识别出的显著组件，对供体模型进行剪枝，使其与较小受体模型的架构相匹配。 3. 微插值注入： 使用最小的混合权重将提取出的“供体切片”注入到受体模型中，在保留受体原始结构的同时增强其能力。

The APM Framework

The Activation-Prune-Merge (APM) process involves three primary steps: 1. Task-Conditioned Activation Mapping: The donor model is analyzed to identify salient layers, hidden dimensions, attention heads, and MLP neurons. 2. Structural Pruning: The donor model is pruned to match the architecture of the smaller recipient model based on the identified salient components. 3. Micro-Interpolation Injection: The extracted "donor slice" is injected into the recipient model using a minimal mixing weight, preserving the recipient's original structure while enhancing its capabilities.

经验性能

APM在16个不同的基准测试中进行了评估，包括推理、数学、代码生成和指令遵循。主要的性能提升包括： * 整体准确率： 在3B受体模型上，准确率从55.5%提升至60.6%。 * RTE 准确率： 从64.3%提升至82.3%。 * QNLI 准确率： 从52.3%提升至65.7%。 * BoolQ 准确率： 从70.8%提升至79.2%。

Empirical Performance

APM was evaluated across 16 diverse benchmarks, including reasoning, mathematics, code generation, and instruction following. Key improvements include: * Overall Accuracy: Improved from 55.5% to 60.6% on a 3B recipient model. * RTE Accuracy: Increased from 64.3% to 82.3%. * QNLI Accuracy: Increased from 52.3% to 65.7%. * BoolQ Accuracy: Increased from 70.8% to 79.2%.

Conclusion

该研究提供了强有力的证据，证明了跨规模异构模型融合在没有显式语义对齐的情况下是可行的。通过精细的激活引导选择来集中供体的贡献，研究人员能够以无训练、计算高效的方式有效地“升级”较小的模型。

Conclusion

The study provides strong evidence that cross-scale heterogeneous model fusion is viable without explicit semantic alignment. By concentrating the donor's contribution through careful, activation-guided selection, researchers can effectively "upgrade" smaller models in a training-free, computationally efficient manner.

Metadata

Metadata