An Autonomous Parallelization of Transformer Model Inference on Heterogeneous Edge Devices

The utilization of advancing transformer-based deep neural network (DNN) models in edge environments holds the promise of improving productivity for intelligent tasks. However, deploying these models on edge devices with limited resources encounters significant performance challenges. Previous solutions have attempted to distribute computation tasks across devices and perform parallel inferences but often fall short of meeting service-level objectives (SLO). This limitation arises from their inability to effectively harness parallelization in transformer-based models and consider the resource diversity of edge devices. In this paper, we propose Hepti, a practical framework designed to facilitate parallel inference of transformer-based DNN models on heterogeneous edge environments. Hepti is armed with: 1) an understanding of transformer model architecture to enable effective parallel inference and 2) dynamic workload optimization to adapt to changing network and device resource capabilities. Our evaluations confirmed that the Hepti autonomously assesses the resource diversity of edge devices and network status. Furthermore, Hepti achieves a maximum performance improvement of 49.1% and 37.1% compared to the local inference approach and state-of-the-art model parallelisms on the BERT-Large model.