A system design optimized for heterogeneous GPU workloads reduced maximum latency by 21–30% and improved latency distribution flatness by 17–25% compared to a single-GPU baseline.
Optimizing GPU workload distribution using CUDA MPS and isolating compute from graphics tasks significantly improves latency determinism for concurrent medical AI applications.
The introduction of AI and ML technologies into medical devices has revolutionized healthcare diagnostics and treatments. Medical device manufacturers are keen to maximize the advantages afforded by AI and ML by consolidating multiple applications onto a single platform. However, concurrent execution of several AI applications, each with its own visualization components, leads to unpredictable end-to-end latency, primarily due to GPU resource contentions. To mitigate this, manufacturers typically deploy separate workstations for distinct AI applications, thereby increasing financial, energy, and maintenance costs. This paper addresses these challenges within the context of NVIDIA’s Holoscan platform, a real-time AI system for streaming sensor data and images. We propose a system design optimized for heterogeneous GPU workloads, encompassing both compute and graphics tasks. Our design leverages CUDA MPS for spatial partitioning of compute workloads and isolates compute and graphics processing onto separate GPUs. We demonstrate significant performance improvements across various end-to-end latency determinism metrics through empirical evaluation with real-world Holoscan medical device applications. For instance, the proposed design reduces maximum latency by 21–30% and improves latency distribution flatness by 17–25% for up to five concurrent endoscopy tool tracking AI applications, compared to a single-GPU baseline. Against a default multi-GPU setup, our optimizations decrease maximum latency by 35% for up to six concurrent applications by improving GPU utilization by 42%. This paper provides clear design insights for AI applications in the edge-computing domain including medical systems, where performance predictability of concurrent and heterogeneous GPU workloads is a critical requirement.
Sinha et al. (Mon,) conducted a other in Medical AI systems. System design optimized for heterogeneous GPU workloads leveraging CUDA MPS vs. Single-GPU baseline and default multi-GPU setup was evaluated on End-to-end latency determinism (maximum latency and latency distribution flatness). A system design optimized for heterogeneous GPU workloads reduced maximum latency by 21–30% and improved latency distribution flatness by 17–25% compared to a single-GPU baseline.