SplitLite：用于拆分学习的低秩残差压缩技术
文章背景与核心概要
SplitLite 是一种旨在优化边缘设备上大语言模型（LLM）联邦微调的新型通信高效框架。尽管拆分学习（Split Learning, SL）在将训练负载卸载至高性能服务器方面表现出色，但由于需要频繁交换高维激活值和梯度，通常会面临巨大的通信开销。
该研究的核心在于利用连续训练周期之间激活值和梯度残差的“低秩结构”。研究人员发现，在使用秩为 \(r\) 的低秩自适应（LoRA）技术时，残差表现出有效的秩-\(2r\) 和秩-\(4r\) 结构。通过仅传输量化后的截断奇异值分解（SVD）残差因子，SplitLite 在不牺牲模型性能的前提下，实现了显著的带宽节省。
SplitLite：用于拆分学习的低秩残差压缩技术
作者： Tao Li, Yulin Tang, Qi Guo, Xianhao Chen
日期： 2026年8月24日
arXiv ID： 2608.23018
学科： 机器学习 (cs.LG)；人工智能 (cs.AI)
SplitLite: Low-Rank Residual Compression for Split Learning
Authors: Tao Li, Yulin Tang, Qi Guo, Xianhao Chen
Date: August 24, 2026
arXiv ID: 2608.23018
Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)
摘要
SplitLite 是一种旨在优化边缘设备上大语言模型（LLM）联邦微调的新型通信高效框架。尽管拆分学习（SL）在将训练负载卸载至高性能服务器方面表现出色，但由于需要交换高维激活值和梯度，通常会面临巨大的通信开销。
SplitLite 通过利用连续训练周期之间激活值和梯度残差的“低秩结构”来解决这一问题。研究人员发现，在使用秩为 \(r\) 的低秩自适应（LoRA）技术时，残差表现出有效的秩-\(2r\) 和秩-\(4r\) 结构。通过仅传输量化后的截断奇异值分解（SVD）残差因子，SplitLite 在不牺牲模型性能的前提下，实现了显著的带宽节省。
Summary
SplitLite is a novel, communication-efficient framework designed to optimize federated fine-tuning of Large Language Models (LLMs) on edge devices. While Split Learning (SL) is effective for offloading training workloads to powerful servers, it typically suffers from high communication overhead due to the exchange of high-dimensional activations and gradients.
SplitLite addresses this by leveraging the low-rank structure of activation and gradient residuals between consecutive training epochs. The researchers discovered that when using Low-Rank Adaptation (LoRA) with rank \(r\), the residuals exhibit effective rank-\(2r\) and rank-\(4r\) structures. By transmitting only quantized, truncated Singular Value Decomposition (SVD) residual factors, SplitLite achieves significant bandwidth savings without compromising model performance.
主要贡献
- 残差秩分析： 确定了基于 LoRA 的拆分学习中，激活值和梯度残差具有固有的低秩属性，为压缩提供了数学基础。
- 通信效率： 该方法将激活值的上行流量减少了高达 93.5%，并将总通信成本降低了高达 83.7%。
- 性能稳定性： 经 GLUE 基准测试验证，这些巨大的通信增益是在不降低模型准确率的情况下实现的。
Key Contributions
- Residual Rank Analysis: Identified that activation and gradient residuals in LoRA-based split learning possess inherent low-rank properties, providing a mathematical foundation for compression.
- Communication Efficiency: The method reduces activation uplink traffic by up to 93.5% and total communication costs by up to 83.7%.
- Performance Stability: Demonstrates that these massive communication gains are achievable without any degradation in model accuracy, as validated on the GLUE benchmark.
技术概述
SplitLite 的核心创新在于对参数空间更新的利用。通过关注激活值和梯度的残差而非原始数值，系统可以通过以下方式显著压缩数据： 1. 截断 SVD： 降低残差矩阵的维度。 2. 量化： 进一步最小化所传输 SVD 因子的占用空间。 3. LoRA 集成： 与现有的 LoRA 微调工作流无缝协作，确保与端侧 LLM 的兼容性。
Technical Overview
The core innovation of SplitLite lies in its exploitation of the parameter space updates. By focusing on the residuals of activations and gradients rather than the raw values, the system can compress the data significantly through: 1. Truncated SVD: Reducing the dimensionality of the residual matrices. 2. Quantization: Further minimizing the footprint of the transmitted SVD factors. 3. LoRA Integration: Seamlessly working with existing LoRA fine-tuning workflows to ensure compatibility with on-device LLMs.
获取资源
如需更多书目工具、引用导出或相关研究，请参考 arXiv 官方页面。
Access & Resources
For further bibliographic tools, citation exports, or related research, please refer to the official arXiv page.