Stochastic Approximation Algorithms for System Identification, Estimation, and Decomposition of Mixtures

A stochastic approximation procedure that minimizes a mean-square-error criterion is proposed in this paper. It is applied first to derive an algorithm for recursive estimation of the mean-square-error approximation of the function which relates the input signals and the responses of a memoryless system. The input signals are assumed to be generated at random with an unknown probability density function, and the response is measured with an error which has zero mean and finite variance. A performance index for evaluating the rate of convergence of the algorithm is defined and then the optimal form of the algorithm is derived. It is shown that the least-square-error fit of the measured output signals of the systems offers a recursive formula which is a special case of the proposed algorithm. A recursive formula for estimation of a priori probabilities of the pattern classes using unclassified samples is then presented. The rate of convergence is computed. A minimum square-error estimate of a continuous probability density function is also obtained by the same algorithm.