On the Synchronization Bottleneck of OpenStack Swift-Like Cloud Storage Systems

As one type of the most popular cloud storage services, OpenStack Swift and its follow-up systems replicate each object across multiple storage nodes and leverage object sync protocols to achieve high reliability and eventual consistency . The performance of object sync protocols heavily relies on two key parameters: <tex-math notation="LaTeX">r</tex-math> (number of replicas for each object) and <tex-math notation="LaTeX">n</tex-math> (number of objects hosted by each storage node). In existing tutorials and demos, the configurations are usually <tex-math notation="LaTeX">$r=3$</tex-math> and <tex-math notation="LaTeX">$n<1,000$</tex-math> by default, and the sync process seems to perform well. However, we discover in data-intensive scenarios, e.g., when <tex-math notation="LaTeX">$r>3$</tex-math> and <tex-math notation="LaTeX">n 1,000</tex-math> , the sync process is significantly delayed and produces massive network overhead, referred to as the sync bottleneck problem . By reviewing the source code of OpenStack Swift, we find that its object sync protocol utilizes a fairly simple and network-intensive approach to check the consistency among replicas of objects. Hence in a sync round, the number of exchanged hash values per node is <tex-math notation="LaTeX">Θ (n× r)</tex-math> . To tackle the problem, we propose a lightweight and practical object sync protocol, LightSync , which not only remarkably reduces the sync overhead, but also preserves high reliability and eventual consistency. LightSync derives this capability from three novel building blocks: 1) Hashing of Hashes , which aggregates all the <tex-math notation="LaTeX">h</tex-math> hash values of each data partition into a single but representative hash value with the Merkle tree; 2) Circular Hash Checking , which checks the consistency of different partition replicas by only sending the aggregated hash value to the clockwise neighbor; and 3) Failed Neighbor Handling , which properly detects and handles node failures with moderate overhead to effectively strengthen the robustness of LightSync. The design of LightSync offers provable guarantee on reducing the per-node network overhead from <tex-math notation="LaTeX">Θ (n× r)</tex-math> to <tex-math notation="LaTeX">Θ (n/h)</tex-math> . Furthermore, we have implemented LightSync as an open-source patch and adopted it to OpenStack Swift, thus reducing the sync delay by up to 879 <tex-math notation="LaTeX">×</tex-math> and the network overhead by up to 47.5 <tex-math notation="LaTeX">×</tex-math> .