GAMLS: a generalized framework for associative modular learning systems

Learning a large number of simple local concepts is both faster and easier than learning a single global concept. Inspired by this principle of divide and conquer, a number of modular learning approaches have been proposed by the computational intelligence community. In modular learning, the classification/regression/clustering problem is first decomposed into a number of simpler subproblems, a module is learned for each of these subproblems, and finally their results are integrated by a suitable combining method. Mixtures of experts and clustering are two of the techniques that are describable in this paradigm. In this paper we present a broad framework for Generalized Associative Modular Learning Systems (GAMLS). Modularity is introduced through soft association of each training pattern with every module. The coupled problems of learning the module parameters and learning associations are solved iteratively using deterministic annealing. Starting at a high temperature with only one module, GAMLS framework automatically evolves the required number of modules through a systematic growing and pruning technique. Each phase begins by splitting every module in the previous phase into two, updating these new modules and then pruning and merging any redundant modules. A phase transition is induced by temperature decay. A number of existing modular learning problems, both unsupervised (clustering, mixture model density, mixture of principal components) and supervised (mixture of experts, radial basis function networks), can be effectively tackled in GAMLS. Case studies for clustering and regression using mixture of experts are provided for a number of datasets showing the efficacy of the GAMLS framework in evolving the right number of modules, inducing interpretable localizations among modules and robustness of the solution obtained. More importantly, this framework provides a unifying view for understanding and characterizing modular learning methods.