Communication compression in distributed deep learning systems: A survey from the architecture and implementation perspective

Communication overhead has become a primary systems bottleneck in distributed deep learning as model scales, parallelism diversity, and fault-tolerance demands continue to grow. Existing surveys mostly emphasize algorithm taxonomies within limited settings, whereas modern training stacks often combine gradient synchronization, tensor and expert exchange, graph data loading, and checkpoint traffic within one software and hardware pipeline. This survey reviews communication compression from a unified systems perspective. We organize representative methods into three basic primitives: sparsification, quantization, and low-rank approximation, and use this taxonomy to analyze compression design across data parallelism, federated learning, model and expert parallelism, distributed checkpointing, and GNN training. We further examine implementation choices in frameworks and communication libraries, highlighting how collective compatibility, overlap, codec overhead, tensor structure, and deployment regime determine whether compression improves end-to-end performance. Based on this cross-scenario synthesis, the survey summarizes method-selection principles and identifies open problems in adaptive, architecture-aware, and software-co-designed compression. It therefore provides a deployment-oriented reference for deciding which communicated state to compress, where compression should be inserted in the software stack, and under which system conditions a given method is likely to improve end-to-end efficiency.