Mean-field theory of Boltzmann machine learning

I present a mean-field theory for Boltzmann machine learning, derived by employing Thouless-Anderson-Palmer free energy formalism to a full extent. Using the Plefka expansion an extended theory that takes higher-order correction to mean-field free energy formalism into consideration is presented, from which the mean-field approximation of general orders, along with the linear response correction, are derived by truncating the Plefka expansion up to desired orders. A theoretical foundation for an effective trick of using ``diagonal weights,'' introduced by Kappen and Rodr\'{}guez, is also given. Because of the finite system size and a lack of scaling assumptions on interaction coefficients, the truncated free energy formalism cannot provide an exact description in the case of Boltzmann machines. Accuracies of mean-field approximations of several orders are compared by computer simulations.