Accelerating Sparse Deep Learning via Multi-Layer Tensor Reordering and Partitioning

Sparse deep learning models reduce memory and computational costs by eliminating redundant parameters; however, many inference engines and hardware platforms lack native support for sparse computation, resulting in underutilized performance. We propose a reordering and partitioning framework that transforms sparse parameter tensors into a singly-bordered block diagonal form, concentrating nonzeros into diagonal and border blocks. The resulting tensors are split into smaller and denser sublayers that enable parallel execution across compute units. In sparsity-aware runtimes, sublayers can run concurrently using specialized sparse kernels. On platforms limited to dense operations, the same partitioning increases compute density and improves the efficiency of standard dense kernels. We formulate this transformation as a hypergraph partitioning problem and extend it to reorder multiple layers jointly, eliminating costly activation reshuffling. Experiments on synthetic and real-world models demonstrate significant reductions in compute, memory usage, and inference latency across diverse hardware targets. Our method complements existing compilers and inference pipelines, enabling scalable and efficient deployment of sparse models.