Orchestrating a DNN training job using an iScheduler Framework: a use case

Orchestrating DNN training jobs efficiently on HPC centers such as Ohio Supercomputer Center (OSC), Texas Advanced Computing Center (TACC), and San Diego Supercomputer Center (SDSC) is crucial due to the prevalence of AI-driven workloads. However, managing these workloads effectively requires a deep understanding of available resources, allocation policies, and suitable execution configurations. Current approaches often lead to job interruptions, prolonged wait times, and inefficient resource utilization. To address these challenges, we propose the deployment of an iScheduler framework. This framework aims to automate workflow orchestration for DNN training by estimating resource needs (using existing state-of-art estimation models) and generating an infrastructure-aware execution plan. In this study, we demonstrate the practical application of the iScheduler framework in orchestrating a user-specific DNN training workflow, showcasing its capabilities in optimizing resource allocation and scheduling. This poster dives deeper into the user case and shows all user interactions with iScheduler and the responses.