PREMA: A Predictive Multi-task Scheduling Algorithm For Preemptible Neural Processing Units

To amortize cost, cloud vendors providing DNN acceleration as a service to-users employ consolidation and virtualization to share the underlying among multiple DNN service requests. This paper makes a case for a"preemptible" neural processing unit (NPU) and a "predictive" multi-task to meet the latency demands of high-priority inference while high throughput. We evaluate both the mechanisms that enable NPUs be preemptible and the policies that utilize them to meet scheduling. We show that preemptive NPU multi-tasking can achieve an average7.8x, 1.4x, and 4.8x improvement in latency, throughput, and SLA satisfaction,.