Understanding and Improving Convolutional Neural Networks via Concatenated Rectified Linear Units

Recently, convolutional neural networks (CNNs) have been used as a powerful to solve many problems of machine learning and computer vision. In this, we aim to provide insight on the property of convolutional neural, as well as a generic method to improve the performance of many CNN. Specifically, we first examine existing CNN models and observe intriguing property that the filters in the lower layers form pairs (i.e., with opposite phase). Inspired by our observation, we propose a novel, yet effective activation scheme called concatenated ReLU (CRelu) and analyze its reconstruction property in CNNs. We integrate CRelu several state-of-the-art CNN architectures and demonstrate improvement in recognition performance on CIFAR-10/100 and ImageNet datasets with fewer parameters. Our results suggest that better understanding of the of CNNs can lead to significant performance improvement with a modification.