HFresh：内存高效的向量搜索引擎

文章背景与核心概要

随着向量数据从百万级增长到数十亿级，传统的内存型 HNSW 索引面临着巨大的内存瓶颈。为了在牺牲部分峰值查询吞吐量的同时显著降低内存消耗，Weaviate 推出了基于磁盘的向量索引——HFresh。它通过结合内存中的量化质心索引与由后台持续任务（分裂、合并和重新分配）管理的磁盘驻留 Posting，能够高效扩展至数十亿个向量，而无需进行破坏性的全局重建。

HFresh 的技术核心在于借鉴了 SPFresh 研究论文的设计理念，并结合 Weaviate 经过实战检验的组件。它采用两阶段搜索策略：首先通过紧凑的内存质心索引（使用 RQ8 量化）识别相关的向量空间区域，然后仅从磁盘获取对应的 Posting 并进行搜索（使用 RQ1 量化）。配合诸如分裂、合并和 LIRE 重新平衡等异步后台任务，HFresh 能够在不进行全局重建的情况下保持索引的新鲜度。它是 Weaviate Cloud 成本优化配置（免费层）的底层支撑，为大规模和受资源限制的工作负载提供了低堆内存、可预测的低延迟替代方案。

理解相似性搜索中 HNSW 的内存瓶颈

Understanding HNSW's Memory Bottleneck in Similarity Search

当需要快速查找相似向量时，HNSW（Hierarchical Navigable Small World，分层可导航小世界）已成为黄金标准。它速度快且准确，但随着数据集从数百万增长到数十亿个向量，HNSW 暴露出一个根本局限：它的图结构和向量缓存都必须保留在内存中。

When it comes to finding similar vectors quickly, HNSW (Hierarchical Navigable Small World) has become the gold standard. It's fast and accurate, but as datasets grow from millions to billions of vectors, HNSW reveals a fundamental constraint: its graph and vector cache are kept in memory.

HNSW 是一种基于图的索引，将向量组织成层次结构。在最顶层，有一个稀疏图，其中包含长距离连接，可帮助你快速导航到正确的邻近区域。随着层数的下降，图变得更加密集，具有更多的局部连接，最终引导你在最底层找到最相似的向量。

HNSW is a graph-based index that organizes vectors into a hierarchical structure. At the top layer, you have a sparse graph with long-distance connections that help you quickly navigate to the right neighborhood. As you descend through the layers, the graphs become denser with more local connections, eventually guiding you to the most similar vectors at the bottom layer.

问题不在于 HNSW 是否优秀。它绝对非常优秀。如果你需要最低的延迟和最高的吞吐量，HNSW 很难被超越。但许多应用程序更倾向于较低的内存消耗和更大的规模，而不是峰值查询性能。

The question isn't whether HNSW is good. It absolutely is. If you need the lowest possible latency and highest throughput, HNSW is hard to beat. But many applications prioritize lower memory use and larger scale over peak query performance.

这就是基于磁盘的索引变得有趣的地方。如果你能用一些延迟换取极低的内存消耗以及扩展到更大数据集的能力，会怎么样呢？

This is where disk-based indexes become interesting. What if you could trade some latency for much lower memory use and the ability to scale to larger datasets?

HFresh 简介

Introducing HFresh

HFresh 是一种现代的基于磁盘的向量索引，专为大规模高召回率、强更新性能和受控查询 I/O 而设计。它建立在 SPFresh 研究论文引入的思想基础之上，并调整了设计以使用 Weaviate 中已经存在的、经过实战检验的组件。

HFresh is a modern disk-based vector index designed for high recall, strong update performance, and controlled query I/O at large scale. It builds on ideas introduced by the SPFresh research paper, adapting the design to use battle-tested components already present in Weaviate.

从整体架构来看，HFresh 属于基于分区的向量索引系列。HFresh 没有像 HNSW 那样将每个向量连接到全局图中的邻居，而是将向量划分为许多称为 Posting 的小区域。每个 Posting 包含向量空间中彼此靠近的向量，并存储在磁盘上的 LSM 存储中。

At a high level, HFresh belongs to the family of partition-based vector indices. Instead of connecting every vector to neighbors in a global graph like HNSW, HFresh divides vectors into many small regions called postings. Each posting contains vectors that are close to each other in vector space and is stored on disk in an LSM store.

为了使这种布局高效，HFresh 采用了两阶段搜索策略。

To make this layout efficient, HFresh uses a two-stage search strategy.

1. 质心识别（Centroid Identification）： 一个紧凑的内存质心索引用于识别向量空间的哪些区域与查询相关。
2. Posting 检索（Posting Retrieval）： 仅从磁盘获取相应的 Posting 并进行详细搜索。

1. Centroid Identification: A compact in-memory centroid index identifies which regions of the vector space are relevant to a query.
2. Posting Retrieval: Only corresponding postings are fetched from disk and searched in detail.

通过将磁盘读取限制在数据集的一小部分，HFresh 保持了有界的 I/O 和可预测的延迟，即使数据集增长到数十亿级别也是如此。

By limiting disk reads to a small subset of the dataset, HFresh keeps I/O bounded and latency predictable, even as the dataset grows into the billions.

无需重建的实时更新性

Freshness Without Rebuilds

SPFresh 背后的核心思想（也被 HFresh 继承）是，大多数更新只影响向量空间的一小块区域。

The key idea behind SPFresh, and inherited by HFresh, is that most updates only affect a small region of the vector space.

在传统的基于分区的索引中，更新会不断累积，分区可能会漂移，最终需要进行完全重建以恢复召回率和延迟——在大规模下，这个过程可能需要数小时甚至数天。SPFresh 表明这通常是不必要的。在一个结构良好的分区索引中，插入或删除向量通常只会影响一个小邻域。你不需要重建一切，而是可以通过使用一组少量的本地操作进行增量重新平衡来维护索引质量：

In traditional partition-based indexes, updates can accumulate and partitions can drift, eventually requiring a full rebuild to restore recall and latency—a process that can take hours or days at scale. SPFresh shows that this is often unnecessary. In a well-structured partitioned index, inserting or deleting a vector typically only affects a small neighborhood. Instead of rebuilding everything, you can maintain index quality through incremental rebalancing using a small set of local operations:

• 分裂（Splitting）过大的 Posting
• 合并（Merging）过小的 Posting
• 重新分配（Reassigning）当边界发生位移时的向量

• Splitting oversized postings
• Merging undersized ones
• Reassigning vectors when boundaries shift

这些操作大多在后台异步运行，在微小的局部不平衡累积成全局问题之前持续对其进行修复。其结果是一个能够随着时间推移保持新鲜且平衡良好的索引，而无需经历破坏性的重建周期。

These operations run mostly asynchronously in the background, continuously repairing small local imbalances before they accumulate into global problems. The result is an index that stays fresh and well-balanced over time without disruptive rebuild cycles.

HFresh 如何在 SPFresh 基础上进行演进

How HFresh Builds on SPFresh

HFresh 提取了 SPFresh 背后的核心思想，并将其调整以适应 Weaviate 的架构，同时保留了： * 本地维护而非重建 * 受控的查询 I/O * 内存路由层与磁盘 Posting 之间的清晰隔离

HFresh takes the core idea behind SPFresh and adapts it to fit Weaviate's architecture, retaining: * Local maintenance instead of rebuilds * Controlled query I/O * A clear separation between the in-memory routing layer and disk-based postings

Weaviate 没有引入全新的 ANN 机制，而是重用了系统中已经存在的、经过实战检验的部件，重新塑造它们以服务于这种新布局。

Rather than introducing entirely new ANN machinery, Weaviate reused battle-tested pieces already present in the system, reshaping them to serve this new layout.

用作质心索引的 HNSW

HNSW as the Centroid Index

HFresh 中的一个关键设计选择是使用 HNSW 作为质心索引，而不是 SPTAG（微软原始 SPANN 设计中使用的 ANN 索引）。

A key design choice in HFresh is the use of HNSW as the centroid index, rather than SPTAG (the ANN index used in Microsoft's original SPANN design).

对于 Weaviate 来说，HNSW 是一个更自然的选择： * 生产环境验证： 它是 Weaviate 中使用最广泛的向量索引，这意味着其在各种工作负载和数据集规模下的行为都得到了充分理解。 * 处理动态工作负载： 质心不是静态的——分裂、合并和重新分配会不断重塑向量空间的分区。HNSW 可以吸收插入和删除操作，而无需昂贵的重建。 * 量化兼容性： 质心层本身可以进行量化。HFresh 将 HNSW 与 RQ8 结合使用，将质心的内存使用量减少了 4 倍。 * 生态系统协同： 对 HNSW 的改进（例如用于过滤向量搜索的 ACORN）会自动使 HFresh 的查询路径受益。

For Weaviate, HNSW was a more natural fit: * Production-proven: It is the most widely used vector index in Weaviate, meaning its behavior is well understood across varied workloads and dataset sizes. * Handles Dynamic Workloads: Centroids are not static—splits, merges, and reassignments continuously reshape the partitioning of the vector space. HNSW can absorb insertions and deletions without expensive rebuilds. * Quantization Compatibility: The centroid layer can itself be quantized. HFresh uses HNSW with RQ8, reducing the memory usage of centroids by 4x. * Ecosystem Synergy: Improvements to HNSW (such as ACORN for filtered vector search) automatically benefit HFresh's query path.

量化

Quantization

HFresh 在两个地方使用了具有两种不同压缩级别的旋转量化（Rotational Quantization），这反映了两个搜索阶段的不同任务：

HFresh uses Rotational Quantization in two places with two different compression levels, reflecting the different jobs of the two search stages:

• 用于质心的 RQ8： 将质心向量内存减少 4 倍。路由错误代价高昂（错过正确的区域意味着错过真正的最近邻），因此质心需要足够的精度来准确路由。
• 用于 Posting 的 RQ1： 与 32 位浮点数相比，将存储的向量数据最多减少 32 倍。Posting 驻留在磁盘上，因此积极的压缩可保持较低的存储成本和磁盘 I/O。最终的候选列表随后使用原始未压缩向量进行重新打分（rescore）。

• RQ8 for Centroids: Reduces centroid vector memory by 4x. Routing mistakes are expensive (missing the right region means missing the true nearest neighbors), so centroids require enough precision to route accurately.
• RQ1 for Postings: Reduces stored vector data by up-to 32x compared to 32-bit floats. Postings live on disk, so aggressive compression keeps storage costs and disk I/O low. Final candidate lists are later rescored using the original uncompressed vectors.

后台操作

Background Operations

维护工作通过三种持续的后台任务类型直接构建到 HFresh 中：

Maintenance is built directly into HFresh via three continuous background task types:

• 分裂（Split）： 当 Posting 变得过大时，陈旧的条目将被垃圾回收，使用平衡 K-Means（Balanced K-Means）算法将向量划分为两个平衡的组，并且两个新的质心将替代旧的质心。

• Split: When a posting grows too large, stale entries are garbage-collected, vectors are divided into two balanced groups using the Balanced K-Means algorithm, and two new centroids replace the old one.

• 合并（Merge）： 当 Posting 变得太小（由于删除或数据漂移）时，HFresh 会将它们合并到能够吸收它们的邻近 Posting 中，从而移除多余的质心以防止碎片化。

• Merge: When postings become too small (due to deletes or data drift), HFresh merges them into a nearby posting that can absorb them, removing the extra centroid to prevent fragmentation.

• 重新分配（Reassign）： 使用 LIRE（Lightweight Incremental Rebalancing，轻量级增量重新平衡）协议，在发生分裂或合并后，不再属于某个 Posting 的向量会被动态重新路由到更好的相邻 Posting 中。

• Reassign: Using the LIRE (Lightweight Incremental Rebalancing) protocol, vectors that no longer belong in a posting following a split or merge are dynamically re-routed to better neighboring postings.

过滤搜索

Filtered Search

过滤向量搜索将向量相似性与元数据约束结合起来。HFresh 使用白名单（表示为位图），并根据匹配规模切换策略：

Filtered vector search combines vector similarity with metadata constraints. HFresh uses an allow list (represented as a bitmap) and switches strategies based on match size:

1. 针对小子集的直接搜索： 当白名单包含少于 5,000 个 ID 时，HFresh 会绕过质心路由和 Posting 扫描，直接获取原始向量以进行精确的距离计算。
2. 感知 Posting 的过滤： 对于更广泛的过滤，HFresh 通过元数据将对象级别的白名单转换为 Posting 的有效性，使用 ACORN 导航质心 HNSW，仅读取匹配的 Posting，并在扫描和重新打分期间应用向量级别的检查。

1. Direct Search for Small Subsets: When the allow list contains fewer than 5,000 IDs, HFresh bypasses centroid routing and posting scans, fetching original vectors directly for exact distance calculations.
2. Posting-Aware Filtering: For broader filters, HFresh translates the object-level allow list into posting eligibility via metadata, navigates the centroid HNSW using ACORN, reads only matching postings, and applies vector-level checks during scanning and rescoring.

如何使用 HFresh

How to Use HFresh

在通过 Python 客户端创建集合时，将 HFresh 配置为向量索引：

Configure HFresh as the vector index when creating a collection via the Python client:

from weaviate.classes.config import Configure, VectorDistances

collection = client.collections.create(
    name="Article",
    vector_config=Configure.Vectors.self_provided(
        name="Title",
        vector_index_config=Configure.VectorIndex.hfresh(
            distance_metric=VectorDistances.COSINE,
        ),
    ),
)

💡 调优 HFresh 从默认值开始。如果召回率太低，请增加： * search_probe 以便每次查询搜索更多 Posting。 * quantizer.rescore_limit 以使用全精度向量对更多候选向量进行重新打分。

这两个设置都可以在不重建索引的情况下进行更改。

💡 Tuning HFresh Start with the defaults. If recall is too low, increase: * search_probe to search more postings per query. * quantizer.rescore_limit to rescore more candidates using full-precision vectors.

Both settings can be changed without rebuilding the index.

实验

Experiments

基准测试在 DBpedia OpenAI 1M 数据集上进行，对比了未压缩的 HNSW、带 RQ1 的 HNSW、带 RQ8 的 HNSW 以及 HFresh。

Benchmarks were performed on the DBpedia OpenAI 1M dataset comparing uncompressed HNSW, HNSW with RQ1, HNSW with RQ8, and HFresh.

堆内存使用情况

Heap Usage

• HFresh： 239 MB（Go 堆内存）
• 未压缩的 HNSW： 6.67 GB
• 带 RQ1 的 HNSW： 715 MB（约为 HFresh 的 3 倍）
• 带 RQ8 的 HNSW： 2.38 GB（约为 HFresh 的 10 倍）

• HFresh: 239 MB (Go heap)
• Uncompressed HNSW: 6.67 GB
• HNSW with RQ1: 715 MB (~3x HFresh)
• HNSW with RQ8: 2.38 GB (~10x HFresh)

查询吞吐量

Query Throughput

虽然 HNSW 变体由于内存中的图遍历而提供了更高的查询吞吐量，但 HFresh 显著减少了堆内存占用，这使其非常适合资源受限或海量规模的环境，在这些环境中，完全的内存缓存成本过高。

While HNSW variants deliver higher query throughput due to in-memory graph traversals, HFresh provides a dramatic reduction in heap footprint, making it ideal for resource-constrained or massive-scale environments where full in-memory caching is cost-prohibitive.

扩展到十亿向量

Scaling to One Billion Vectors

HFresh 使用 32 个 vCPU、256 GB RAM（n2-highmem-32）以及 4 TB SSD 持久磁盘（pd-ssd），对扩展到 10 亿个随机生成的 256 维向量进行了测试：

HFresh was tested scaling to 1 billion randomly generated 256-dimensional vectors using 32 vCPUs, 256 GB RAM (n2-highmem-32), and a 4 TB SSD Persistent Disk (pd-ssd):

结论

Conclusion

HFresh 为 Weaviate 带来了强大的基于磁盘的向量索引，通过增量的后台平衡，在保持高召回率和新鲜度的同时，最大限度地减少了内存开销。从 Weaviate 1.36（技术预览版） 和 1.38（正式发布版） 开始提供，它为大规模或受内存限制的工作负载提供了一个高效的选择。

HFresh brings a powerful disk-based vector index to Weaviate, minimizing memory overhead while maintaining high recall and freshness through incremental background balancing. Available starting in Weaviate 1.36 (Technical Preview) and 1.38 (General Availability), it provides an efficient choice for large-scale or memory-constrained workloads.

• 查看快速入门教程或注册免费的 Weaviate Cloud 账户。

• Check out the Quickstart tutorial or sign up for a free Weaviate Cloud account.