Frequency-aware resemblance detection for group compression

Data deduplication and compression systems critically depend on accurately detecting and grouping similar data blocks to maximize storage efficiency. However, traditional resemblance detection techniques like MinHash, Rabin fingerprints, and Finesse treat all shingles equally, leading to suboptimal grouping because high-frequency shingles are already well-handled by intra-block compression (e.g., ZSTD). To address this misalignment between estimated similarity and true compressibility, we propose LF-Hash, a frequency-aware resemblance detection method tailored for group compression. LF-Hash filters out high-frequency shingles and prioritizes low-frequency shingles, especially single-occurrence ones, which more accurately reflect inter-block structural redundancies. It employs an efficient one-permutation locality-sensitive hashing scheme and constructs a weighted similarity graph, followed by a multilevel graph partitioning algorithm for scalable block grouping. Integrated with group compression, our method bypasses the limitations of base-dependent delta compression and allows fine-grained, inter-block redundancy elimination. Extensive experiments covering twelve real-world datasets of different types demonstrate that LF-Hash consistently outperforms state-of-the-art baselines, achieving up to 48% higher compression ratios on system/application workloads and up to 130% on enterprise datasets while maintaining competitive throughput, making it well-suited for modern storage, backup, and deduplication scenarios.