Negative correlation learning and evolutionary design of neural network ensembles

Many real-world problems are too large and too complex for a single monolithic system to solve alone. Instead of using a single neural network to solve a task, neural network ensembles combine a set of neural networks which learn to subdivide the task and thereby solve it more efficiently and elegantly. Given the advantages of neural network ensembles and the complexity of the problems that are beginning to be investigated, it is clear that the neural network ensemble method is and will be an important and pervasive problem-solving technique. However, designing neural network ensembles is a very difficult task. It relies heavily on human experts and prior knowledge about the problem. This dissertation aims at developing evolutionary ensembles with negative correlation learning (EENCL) for automatically designing neural network ensembles. This dissertation first presents negative correlation learning for learning of neural network ensembles. Negative correlation learning is different from previous work on learning of neural network ensembles. It emphasises interaction and cooperation among the individual networks in neural network ensembles, and uses an unsupervised penalty term in the error function to produce negatively correlated networks. This dissertation extensively analyses negative correlation learning in terms of the bias-variance-covariance trade-off in both the noise free condition and the noisy condition on a regression task. Negative correlation learning has also been studied on a classification problem in terms of the correct response sets and their intersections. Two time series prediction problems, i.e., the Mackey-Glass differential equation and the chlorophyll-a prediction in Lake Kasumigaura, are used as examples to demonstrate the effectiveness of negative correlation learning. Based on negative correlation learning and evolutionary learning, EENCL have been developed for designing neural network ensembles. Two major issues of automatic determination of the number of individual neural network in an ensemble and the exploitation of the interaction between individual neural network design and combination have been addressed in EENCL. In EENCL, an evolutionary algorithm based on evolutionary programming has been used to search for a population of diverse individual neural networks that together solve a problem. To maintain a diverse population, fitness sharing and negative correlation learning have been used to encourage the formation of different species. Empirical evaluation of EENCL has been carried out on two benchmark problems, including the Australian credit card assessment problem and the diabetes problem. Very competitive results have been produced by EENCL in comparison with other algorithms.