跳转至

科学数据高保真学习压缩的残差建模

文章背景与核心概要

随着科学模拟生成的海量时空数据集不断增长，有损压缩（Lossy Compression）成为了数据管理与存储的关键技术。尽管基于学习的压缩器（Learned Compressors）在适度精度要求下能够实现高压缩比，但它们的整体重建损失无法保证单个数据块的严格误差界限。为了解决这一问题，现有的保证自编码器（Guaranteed Autoencoder, GAE）方法通过保留 SVD/PCA 风格的系数来进行逐块残差校正，直到满足目标的误差容限。

然而，这种方法在高保真区间（块级 NRMSE 介于 \(10^{-6}\) 到 \(10^{-4}\) 之间）会失效：所需系数的数量呈指数级增长，导致校正数据流占据了总比特率的主导地位。为此，本文作者引入了以残差为中心的新视角，提出两种新颖的残差编码器：LBRC（基于学习的残差 coder）和 NGLR（神经引导的洛伦佐残差 coder）。在 E3SM、JHTDB 和 ERA5 数据集上的评估表明，这些方法显著提升了高保真压缩的性能，超越了传统的 SZ 压缩器。

  • Authors: Liangji Zhu, Sanjay Ranka, Anand Rangarajan
  • Subjects: Artificial Intelligence (cs.AI)
  • ArXiv ID: arXiv:2606.05389
  • DOI: 10.48550/arXiv.2606.05389
  • Submission History: Submitted on 3 June 2026; Last revised 13 August 2026 (v2).
  • Authors: Liangji Zhu, Sanjay Ranka, Anand Rangarajan
  • Subjects: Artificial Intelligence (cs.AI)
  • ArXiv ID: arXiv:2606.05389
  • DOI: 10.48550/arXiv.2606.05389
  • Submission History: Submitted on 3 June 2026; Last revised 13 August 2026 (v2).

Abstract Summary

Lossy compression is crucial for managing massive spatiotemporal datasets generated by scientific simulations. While learned compressors excel at achieving high compression ratios under moderate accuracy requirements, their aggregate reconstruction losses do not guarantee strict error bounds for individual data blocks.

To address this, existing Guaranteed Autoencoder (GAE) methods apply per-block residual corrections by retaining SVD/PCA-style coefficients until target error tolerances are satisfied. Although this approach functions effectively at moderate tolerances, it breaks down in high-fidelity regimes (block-level NRMSE ranging from \(10^{-6}\) to \(10^{-4}\)): the number of required coefficients grows exponentially, causing the correction stream to dominate the total bit rate.

Lossy compression is crucial for managing massive spatiotemporal datasets generated by scientific simulations. While learned compressors excel at achieving high compression ratios under moderate accuracy requirements, their aggregate reconstruction losses do not guarantee strict error bounds for individual data blocks.

To address this, existing Guaranteed Autoencoder (GAE) methods apply per-block residual corrections by retaining SVD/PCA-style coefficients until target error tolerances are satisfied. Although this approach functions effectively at moderate tolerances, it breaks down in high-fidelity regimes (block-level NRMSE ranging from \(10^{-6}\) to \(10^{-4}\)): the number of required coefficients grows exponentially, causing the correction stream to dominate the total bit rate.

Key Contributions

The authors introduce a residual-centric perspective: because learned residuals are structurally distinct from original scientific fields, they require a specialized representation. Two novel residual coders are proposed:

  1. LBRC (Learning-Based Residual Coder): A deterministic, training-free pipeline that adaptively quantizes the learned residual to meet target NRMSE levels. It losslessly encodes the resulting integer residual using 3D Lorenzo differencing, zigzag mapping, bit-plane coding, and entropy coding.
  2. NGLR (Neural-Guided Lorenzo Residual): Enhances LBRC by introducing a causal neural predictor that outputs a normalized bias for integer-rounded Lorenzo predictions. This reduces the entropy of the remaining residual code while preserving deterministic decoding (with predictor weights serialized into the bitstream).

Key Contributions

The authors introduce a residual-centric perspective: because learned residuals are structurally distinct from original scientific fields, they require a specialized representation. Two novel residual coders are proposed:

  1. LBRC (Learning-Based Residual Coder): A deterministic, training-free pipeline that adaptively quantizes the learned residual to meet target NRMSE levels. It losslessly encodes the resulting integer residual using 3D Lorenzo differencing, zigzag mapping, bit-plane coding, and entropy coding.
  2. NGLR (Neural-Guided Lorenzo Residual): Enhances LBRC by introducing a causal neural predictor that outputs a normalized bias for integer-rounded Lorenzo predictions. This reduces the entropy of the remaining residual code while preserving deterministic decoding (with predictor weights serialized into the bitstream).

Performance Highlights

Evaluated across E3SM, JHTDB, and ERA5 datasets at high-fidelity block-level NRMSE targets (\(10^{-6}\) to \(10^{-4}\)): * LBRC improves compression ratios over GAE by 30–60% and remains broadly competitive with traditional SZ compressors. * NGLR achieves an additional 10–40% improvement over LBRC, successfully outperforming SZ in the evaluated high-fidelity regimes.

Performance Highlights

Evaluated across E3SM, JHTDB, and ERA5 datasets at high-fidelity block-level NRMSE targets (\(10^{-6}\) to \(10^{-4}\)): * LBRC improves compression ratios over GAE by 30–60% and remains broadly competitive with traditional SZ compressors. * NGLR achieves an additional 10–40% improvement over LBRC, successfully outperforming SZ in the evaluated high-fidelity regimes.