跳转至

遗忘、可塑性与协同观测：持续学习的第三维度

文章背景与核心概要

在深度学习领域，持续学习（Continual Learning）旨在让模型在不遗忘旧知识的前提下学习新任务。传统研究通常将顺序训练与离线联合训练之间的性能差距归因于两大核心问题：灾难性遗忘（Catastrophic Forgetting）和可塑性丧失（Loss of Plasticity）。

本文提出并论证了第三个关键因素——“数据协同观测”（Data Co-observation）。研究表明，即使在完全解决了遗忘和可塑性问题的情况下，顺序训练仍无法达到联合训练的性能水平。作者通过实验证明，同时观测训练数据所带来的泛化优势远超单纯的知识保留。这一发现重新定义了记忆回放（Memory Replay）的作用：它不仅是缓解遗忘的手段，更是将“协同观测”带来的泛化红利重新引入学习过程的关键机制。

📌 总结

Traditional research in continual learning attributes the performance gap between sequential training and offline joint training primarily to two issues: catastrophic forgetting and the loss of plasticity. This paper introduces a third distinct factor: data co-observation. By decoupling data access constraints from stability and plasticity, the authors demonstrate that the simultaneous observation of training data yields generalization benefits that go far beyond mere knowledge retention or distribution shift mitigation. Furthermore, the study contextualizes existing mechanisms, suggesting that memory replay actively reintroduces the benefits of data co-observation into the learning process.

持续学习中的传统研究将顺序训练与离线联合训练之间的性能差距主要归因于两个问题：灾难性遗忘和可塑性丧失。本文引入了第三个独特的因素：数据协同观测。通过将数据访问限制与稳定性和可塑性解耦，作者证明了同时观测训练数据所带来的泛化收益远不止于知识保留或缓解分布偏移。此外，该研究对现有的机制进行了语境化分析，表明记忆回放（Memory Replay）主动将数据协同观测的益处重新引入了学习过程。

📝 摘要

Efficient continual learning remains a fundamental challenge for deep neural networks. While catastrophic forgetting and loss of plasticity are widely considered the primary obstacles to overcome, we show that these two issues cannot fully explain the performance gap between naive sequential training and offline joint training.

高效的持续学习仍然是深度神经网络面临的一项根本性挑战。尽管灾难性遗忘和可塑性丧失被广泛认为是需要克服的主要障碍，但我们证明，这两个问题并不能完全解释朴素顺序训练与离线联合训练之间的性能差距。

In this paper, we highlight data co-observation as a distinct factor influencing continual learning performance. By decoupling the constraints of separate data access from stability and plasticity, we systematically investigate the representational benefits gained by observing training data together.

在本文中，我们强调了数据协同观测是影响持续学习性能的一个独特因素。通过将独立数据访问的约束与稳定性和可塑性解耦，我们系统地研究了通过共同观测训练数据所获得的表征优势。

Empirically, we demonstrate a consistent performance difference between joint and separate training across both supervised and self-supervised paradigms in generic data-incremental "chunking" scenarios, whilst mitigating forgetting and controlling for plasticity. Our findings indicate that simultaneous observation of training data (co-observation) yields benefits to the learner's generalization that extend well beyond mere knowledge retention, and that this effect does not require a specific continual distribution shift.

在实证方面，我们在通用的数据增量“分块”场景中，针对监督学习和自监督学习范式，证明了联合训练与独立训练之间存在持续的性能差异，同时我们排除了遗忘和可塑性的影响。我们的研究结果表明，同时观测训练数据（协同观测）为学习者的泛化能力带来的益处远超单纯的知识保留，且这种效应并不依赖于特定的持续分布偏移。

Furthermore, we contextualize prominent continual learning mechanisms through this lens: while distillation-based approaches act only as effective knowledge retention mechanisms, our results suggest that the empirical success of memory replay goes beyond the mitigation of forgetting, actively reintroducing the benefits of data co-observation into the learning process.

此外，我们通过这一视角对主流的持续学习机制进行了语境化分析：虽然基于蒸馏的方法仅作为有效的知识保留机制发挥作用，但我们的结果表明，记忆回放的实证成功不仅在于缓解遗忘，更在于主动将数据协同观测的益处重新引入学习过程。

🔗 快速链接与资源