Fragment: Efficient DNN Checkpoint with Relaxed Model Consistency

Training large Deep Neural Networks (DNNs) is inherently resource-intensive and time-consuming, driving significant research into high-frequency checkpointing for enhanced fault tolerance. However, the overhead associated with checkpointing can prolong overall training time. State-of-the-art (SOTA) solutions break down the checkpointing operation into smaller phases, such as snapshot and persist, and pipeline these phases with foreground training operations. Additionally, some strategies leverage faster dynamic or persistent memory devices to minimize overheads. Despite these advancements, SOTA methods still struggle to eliminate training stalls, primarily due to two critical factors: (i) a bandwidth-limited PCIe bus and (ii) consistency requirements for copying the entire model state from the GPU. We propose Fragment — a novel checkpointing solution that relaxes model consistency requirements for fault tolerance, strategically divides the model state into multiple independent pieces (referred to as fragments ), decreases checkpointing overheads, and increases checkpointing frequency, all without significantly impacting model accuracy. Partitioning complex models into fragments creates challenges that we address by ensuring the integrity of the model state during both creation and restoration, while optimizing the selection of layers. Fragment reduces checkpointing overheads by 15% to 94% while decreasing the total data checkpointed by 43% to 80%, without compromising fault-tolerance needs compared to SOTA solutions. Fragment can also reduce recovery time by up to 97% upon each failure recovery, in the worst case.