POSTER:In-network Model Inference for Distributed Systems via Programmable Switches

Model parallelism is crucial for accelerating distributed DNN inference. As a core component of distributed systems, programmable switches have demonstrated their ability in assisting both communication and partial computation. Exploiting the characteristics of convolution operations, this work introduces ConvP4, a novel system, that can perform in-network DNN inference with model parallelism on programmable switches. Preliminary testbed experiments demonstrate that ConvP4 significantly improves inference throughput and reduces CPU usage at end hosts, with minimum overhead incurred.