Local models and Gaussian mixture models for statistical data processing

In this dissertation, we present linear models for dimension reduction and Gaussian mixture models for classification and regression. When the data has different structure in different parts of the input space, fitting once global model can be slow and inaccurate. Simple learning models can quickly learn the structure of the data in small (local) regions. Thus, learning techniques can offer us faster and more accurate model fitting. Gaussian mixture models form a soft model of the data; data points belong to all regions (Gaussians) at once with differing degrees of membership. Thus, mixture models blend together the different (local) models. We show that linear dimension reduction approximates maximum likelihood signal extraction for a mixture of Gaussians signal-plus-noise model. The thesis of this document is that local learning models can perform efficient (fast and accurate) data processing. We propose linear dimension reduction algorithms which partition the input space and build separate low dimensional coordinate systems in disjoint regions of the input space. We compare the linear models with a global linear model (principal components analysis) and a global non-linear model (five layered auto-associative neural networks). For speech and image data, the linear models incur about half the error of the global models while training nearly an order of magnitude faster than the neural networks. Under certain conditions, the linear models are related to a mixture of Gaussians data model. Motivated by the relation between linear dimension reduction and Gaussians mixture models we present Gaussian mixture models for classification and regression and propose algorithms for regularizing them. Our results with speech phoneme classification and some benchmark regression tasks indicate that the mixture models perform comparably with a global model (neural networks). To summarize, models or Gaussian mixture models can be efficient tools for dimension reduction, exploratory data analysis, feature extraction, classification and regression.