Memory-Efficient Feature Merging for Residual Connections with Layer-Centric Tile Fusion

Convolutional neural networks (CNNs) have achieved remarkable success in computer vision tasks, driving the rapid development of hardware accelerators. However, memory efficiency remains a key challenge, as conventional accelerators adopt layer-by-layer processing, leading to frequent external memory accesses (EMAs) of intermediate feature data, which increase energy consumption and latency. While layer fusion has been proposed to enhance inter-layer feature reuse, existing approaches typically rely on fixed data management tailored to specific architectures, introducing on-chip memory overhead and requiring trade-offs with EMAs. Moreover, prevalent residual connections further weaken fusion benefits due to diverse data reuse distances. To address these challenges, we propose layer-centric tile fusion, which integrates residual data loading with feature merging by leveraging receptive field relationships among feature tiles. A reuse distance-aware caching strategy is introduced to support flexible storage for various data types. We also develop a modeling framework to analyze the trade-off between on-chip memory usage and EMA-induced energy-delay product (EDP). Experimental results demonstrate that our method achieves 5.04–43.44% EDP reduction and 20.28–58.33% memory usage reduction compared to state-of-the-art designs on ResNet-18 and SRGAN.