HARENS: Hardware Accelerated Redundancy Elimination in Network Systems

With the increasing growth in the amount of information stored on remote locations and cloud systems, many service providers are seeking ways to reduce the amount of redundant information. Data deduplication can reduce the network traffic without loss of information, and consequently increase the available network bandwidth. However, due to the heavy computation overhead for detecting and reducing the redundant data, deduplication itself can become a bottleneck in high capacity links. In this paper, we propose a method named Hardware Accelerated Redundancy Elimination in Network Systems (HARENS). HARENS can significantly improve the performance of redundancy elimination in a network system by leveraging General Purpose Graphic Processing Unit (GPGPU) optimizations, as well as other optimizations such as the use of a hierarchical multi-threaded pipeline, Hash-Match, and memory efficiency techniques. Our results indicate that throughput can be increased by a factor of 14 compared to a native implementation of a network deduplication algorithm, providing a net transmission increase of up to 10.7 Gigabits per second (Gbps).