Performance of Compression Algorithms for Higher Dimensional Transmission Electron Microscopy Dataset Images

Recent advances in high-speed, pixelated direct electron detectors increase the temporal and spatial sampling achievable in four-dimensional scanning transmission electron microscopy (4D-STEM), resulting in unprecedented data volumes, with acquisition rates approaching ∼1 TB/s. This rapid growth in data collection rate has created an urgent need for efficient and lossless compression strategies to reduce data storage and transfer costs. However, the intrinsic structural and statistical complexity of transmission electron microscopy (TEM) and 4D-STEM datasets presents significant challenges for storage that requires the innovation of novel, conventional compression algorithms to mitigate costs. In this study, Shannon entropy [1] is employed as a quantitative measure of image and diffraction pattern complexity, providing a physically motivated metric for assessing compressibility across diverse TEM datasets. Multiple lossless compression approaches are investigated, including LZ compression [2], ZFS compression employing LZ-based algorithms at the filesystem level [3], and a sparse data compression method implemented at NCEM. The LZ algorithm reduces redundancy by encoding repeated patterns by constructing its dictionary during encoding. ZFS compression works on fixed-size filesystem blocks, compressing each block independently rather than the entire file, using generic LZ-based pattern detection. In contrast, the NCEM-advanced compression method primarily exploits sparsity by reducing datasets to a compact index corresponding only to the locations of non-zero intensities, eliminating zeros prior to compression and encoding the resulting sparse representation within an HDF5 container [4]. Shannon entropy is shown to strongly correlate with achievable compression ratios. Images and diffraction datasets with higher Shannon entropy consistently exhibit lower compression efficiency, while low Shannon entropy and highly sparse datasets—particularly those arising in 4D-STEM experiments—benefit substantially from index-based, application-level compression strategies as seen in Figure 1. Compression performance is evaluated using experimental 4D-STEM datasets [5-7]. This work highlights the importance of Shannon entropy-aware and sparsity-aware data management in modern TEM and 4D-STEM experiments and underscores the need for compression algorithms tailored to the unique statistical and structural properties of TEM data. By directly linking Shannon entropy, sparsity, and compression algorithm performance using publicly available NCEM datasets [5,6], this study provides practical guidance for selecting and designing efficient compression pipelines for next-generation TEM data. The authors acknowledge support from those in reference [7]. (a) Compression ratios of the NCEM and LZ algorithms are plotted against the Shannon entropy of the image. Here the compression ratios determined using the NCEM and LZ algorithms are plotted as functions of Shannon entropy or bit density. The uncertainties on the intersection are determined by the error on the fit. (b) The ratio of the non-zero pixel value array size to the compressed file size is plotted for the NCEM and LZ algorithms as a function of spot size or current. All files have the same size, 16 GB. The NCEM algorithm has the advantage when processing images of low Shannon entropy.