Noise-resilient designs and analysis for optical neural networks

Abstract All analog signal processing is fundamentally subject to noise, and this is also the case in next generation implementations of Optical Neural Networks (ONNs). Therefore, we propose the first hardware-based approach to mitigate noise in ONNs. A tree-like and an accordion-like design are constructed from a given Neural Network (NN) that one wishes to implement. Both designs have the capability that the resulting ONNs give outputs close to the desired solution. To establish the latter, we analyze the designs mathematically. Specifically, we investigate a probabilistic framework for the tree-like design that establishes the correctness of the design, i.e., for any feed-forward NN with Lipschitz continuous activation functions, an ONN can be constructed that produces output arbitrarily close to the original. ONNs constructed with the tree-like design thus also inherit the universal approximation property of NNs. For the accordion-like design, we restrict the analysis to NNs with linear activation functions and characterize the ONNs’ output distribution using exact formulas. Finally, we report on numerical experiments with LeNet ONNs that give insight into the number of components required in these designs for certain accuracy gains. The results indicate that adding just a few components and/or adding them only in the first (few) layers in the manner of either design can already be expected to increase the accuracy of ONNs considerably. To illustrate the effect we point to a specific simulation of a LeNet implementation, in which adding one copy of the layers components in each layer reduces the Mean Squared Error (MSE) by 59.1% for the tree-like design and by 51.5% for the accordion-like design. In this scenario, the gap in accuracy of prediction between the noiseless NN and the ONNs reduces even more: 93.3% for the tree-like design and 80% for the accordion-like design.