What question did this study set out to answer?

The aim is to evaluate SZY v8's effectiveness as a fast, lossy compression engine for scientific 2D data with controllable error and integrity checks.

May 3, 2026Open Access

SZY v8 - Lossy compression of scientific 2D data: transmission, archiving, fastdecode

Puntos clave

The aim is to evaluate SZY v8's effectiveness as a fast, lossy compression engine for scientific 2D data with controllable error and integrity checks.
Developed SZY v8 as a tile-based compression engine for 2D data types including float64 and uint16.
Implemented SHA-256 for data integrity verification and established a benchmark comparison with existing compression methods like SZ3 and ZFP.
Released reference C implementation and benchmark scripts for public access to facilitate further development and evaluation.
SZY v8 demonstrates high throughput and controllable error in lossy compression, optimizing data integrity during transmission.
Achieved faster decoding times compared to traditional codecs with multi-threaded designs and predictable scheduling.
Proved effective for various scientific applications, including archiving and real-time data processing in fields like climate modeling and medical imaging.

Resumen

SZY v8 is a fast, tile-based lossy compression engine for 2D scientific arrays (float64/float32/uint16), designed for real-world workflows where you need controllable error, high throughput, and robust integrity checks. Unlike “just a codec”, SZY v8 ships as an engine + portable container format (SZ2D v8) that supports independent tiles, optional tile length indexing for random access and parallel decoding, and built-in SHA-256 integrity verification to detect silent corruption during storage or transmission. The engine source code is available on GitHub: https://github.com/Szyryngo/SZY Highlights Bounded-error lossy compression using a clear, engineering-friendly model (maximum absolute error target). Tile independence (B×B blocks): enables region-of-interest decoding, parallelism, and resilience to local damage. Fast multi-threaded decode with predictable scheduling via an optional tile-length index. Decode-into buffer API: decompress directly into a user-provided output buffer (useful for HPC/Fortran/C/C++ pipelines). Container-level integrity: SHA-256 for the data section (index + tile records) SHA-256 for the full container Practical format spec included: endianness rules, varuint encoding, container layout, tile-record layout, error codes. Benchmark tooling: Python scripts and wrappers used to compare SZY v8 against SZ3, ZFP, and lossless baselines (Zstd/Blosc2) on public datasets. Typical use cases Scientific archiving with explicit, controlled error tolerance High-throughput transmission pipelines (network / SSD) with corruption detection Climate/ocean model fields, satellite imagery bands, sensor grids, medical/astronomy-like 2D frames On-the-fly decoding in compute pipelines (ROI and decode-into workflows) What this release contains Reference C implementation (engine + format parser/validator) SZ2D v8 container + tile record specification (including integrity and index rules) DLL/API interface and build instructions (MSVC), with Zstd/Zlib entropy backends Python ctypes wrapper and public benchmark scripts/results format Notes SZY v8 is intentionally not a lossless codec. Setting eps=0 disables the bounded-error guarantee but still uses lossy quantization. The encoder can explicitly refuse compression when the requested bound cannot be guaranteed (returns a bounds-related error), instead of producing a stream that violates the declared constraint.

Leer artículo completoexternamente

Me gusta

Guardar

Ver artículo completo