跳转至

文章背景与核心概要

多模态大语言模型（MLLMs）在处理复杂任务时展现出强大的性能，但其可扩展性受到键值（KV）缓存存储所需的巨额内存和计算开销的严重瓶颈制约。现有的淘汰方法通常结合基于余弦相似度的多样性度量与重要性得分来过滤关键 Token，然而余弦相似度通过归一化丢弃了模长信息，并且由于隐藏表示的各向异性，往往在各层间表现出高度一致的相似度值。

为了克服这些局限性，本文提出了 ECOKV，这是一种几何感知的复合度量方法，它结合了欧氏距离和余弦相似度，从互补的视角捕捉 Token 的多样性。通过利用这些度量来估计注意力头的冗余度，ECOKV 能够自适应地平衡多样性和重要性得分，缩减必要的近期 Token 观察窗口，并将缓存容量重新分配给高价值的信息密集型 Token。大量实验证明，ECOKV 在各种压缩比下均实现了 SOTA（最先进）的性能，同时与现有的淘汰框架完全兼容。

ECOKV: Geometry-Aware KV Cache Eviction via Complementary Diversity Metrics

Authors: Chin Ting Hsu, Yu-Syuan Xu, Ling Zou, Hsien-Kai Kuo, Wen-Huang Cheng
Subjects: Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV)
arXiv: 2609.06663 [cs.CL]
Submitted: 6 September 2026

📌 Summary

多模态大语言模型（MLLMs）在各项复杂任务中展现出强劲的性能，但其扩展性受限于键值（KV）缓存（KV Cache）存储所需的大量内存与计算开销。尽管近期的淘汰方法将基于余弦相似度的多样性度量与重要性得分相结合以过滤关键 Token，但余弦相似度由于归一化操作而丢弃了模长信息，且由于隐藏表示的各向异性属性，往往在各个层中表现出均匀的高相似度值。

Multimodal Large Language Models (MLLMs) deliver powerful performance across complex tasks, but their scalability is bottlenecked by the heavy memory and computational overhead required for Key-Value (KV) cache storage. While recent eviction methods combine cosine similarity-based diversity metrics with importance scores to filter critical tokens, cosine similarity discards magnitude information via normalization and often exhibits uniformly high similarity values across layers due to hidden representation anisotropy.

为了克服这些局限性，ECOKV 引入了一种几何感知的复合度量标准，它将欧氏距离（Euclidean distance）与余弦相似度结合起来，从互补的视角捕获 Token 的多样性。通过利用这些度量来估计注意力头的冗余度，ECOKV 能够自适应地平衡多样性和重要性得分，缩减必需的近期 Token 观察窗口（recent-token observation window），并将缓存容量重新分配给高价值的信息 Token。广泛的实验证明，ECOKV 在各种压缩比下均达到了最先进（SOTA）的性能，同时与现有的淘汰框架完全兼容。

To overcome these limitations, ECOKV introduces a geometry-aware composite metric that combines Euclidean distance and cosine similarity to capture token diversity from complementary perspectives. By utilizing these metrics to estimate attention head redundancy, ECOKV adaptively balances diversity and importance scores, reduces the necessary recent-token observation window, and reallocates cache capacity to high-value informative tokens. Extensive experiments prove that ECOKV achieves state-of-the-art performance across diverse compression ratios while remaining fully compatible with existing eviction frameworks.

📖 Abstract

尽管多模态大语言模型（MLLMs）在各种任务中表现优异，但其可扩展性仍然受到 KV 缓存存储的内存和计算开销的限制。近期的 KV 缓存淘汰方法将基于余弦相似度的多样性度量与重要性度量相结合，以选择性地保留关键的键值对。然而，余弦相似度涉及丢弃模长信息的归一化过程，并且由于隐藏表示的各向异性（anisotropy）特性，它往往在不同层之间产生统一的高相似度值。

Although multimodal Large Language Models (MLLMs) excel in diverse tasks, their scalability remains limited by the memory and computational overhead of KV cache storage. Recent KV cache eviction approaches incorporate a cosine similarity-based diversity metric with importance metrics to selectively retain critical key-value pairs. However, cosine similarity involves normalization that discards magnitude information, and it often yields uniformly high similarity values across layers due to the anisotropy property of hidden representations.

在我们的研究 ECOKV 中，我们严谨地剖析了现有多样性度量的能力。超越简单的测量，我们提出了一种几何感知的复合度量，它共同利用欧氏距离和余弦相似度，从互补的视角捕获 Token 的多样性。此外，我们使用这两个度量来估计每个注意力头的冗余水平，从而在 Token 选择过程中实现多样性得分与重要性得分之间的自适应加权。最后，我们证明了通常用于保留近期 Token 的观察窗口可以大幅缩减，从而为信息丰富的 Token 分配更多缓存容量，并带来持续的性能提升。

In our study ECOKV, we rigorously deconstruct the capabilities of existing diversity metrics. Moving beyond simple measurement, we propose a geometry-aware composite metric that jointly leverages Euclidean distance and cosine similarity to capture token diversity from complementary perspectives. Furthermore, we use these two metrics to estimate the redundancy level of each attention head, allowing adaptive weighting between diversity and importance scores during token selection. Finally, we demonstrate that the observation window commonly employed to preserve recent tokens can be substantially reduced, thereby allocating more cache capacity to informative tokens and yielding consistent improvements.

广泛的实验表明，ECOKV 在各种压缩比下均实现了最先进的性能，并且可以无缝集成到现有的 KV 缓存淘汰方法中。我们进一步分析了重要性与多样性之间的关系，并检查了层与注意力头之间的冗余模式。

Extensive experiments demonstrate that ECOKV achieves state-of-the-art performance under various compression ratios and can be seamlessly integrated with existing KV cache eviction methods. We further analyze the relationship between importance and diversity, and examine redundancy patterns across layers and attention heads.