Inter-data-center network traffic prediction with elephant flows

With the ever increasing number of large scale Internet applications, inter data center (inter-DC) data transfers are becoming more and more common. Traditional inter-DC transfers suffers from both low-utilization and congestion, and traffic prediction is an important method to optimize these transfers. Inter-DC traffic is harder to predict than many other types of network traffic, because it is dominated by a few large applications. We propose a model that significantly reduces the prediction errors. In our model, we combine wavelet transform with artificial neural network (ANN) to improve prediction accuracy. Specifically, we explicitly add information of elephant flows, the least predictable yet dominating traffic in inter-DC network, into our prediction model. To reduce the amount of monitoring overhead for the elephant flow information, we added interpolation to fill in the unknown values in the elephant flows. We demonstrate that we can reduce prediction errors over existing methods by 5%~10%. Our prediction is already in production at Baidu, one of the largest Internet companies in China, helping reducing the peak network bandwidth.