Systems for training massive deep learning models (billions of parameters) today assume and require specialized "hyperclusters": hundreds or thousands of GPUs wired with specialized high-bandwidth interconnects such as NV-Link and Infiniband. Besides being expensive, such dependence on hyperclusters and custom high-speed inter-connects limits the size of such clusters, creating (a) scalability limits on job parallelism; (b) resource fragmentation across hyperclusters.
No takes yet. Share an insight, caveat, or question.
Athlur et al. (2022) studied this question.
Synapse has enriched 5 closely related papers on similar clinical questions. Consider them for comparative context: