Towards a unification of information theoretic learning and kernel methods

In this paper, we discuss an intriguing relationship between information theoretic learning (ITL), based on Parzen window density estimation, and kernel-based learning algorithms. We show that some of the widely used ITL cost functions, when estimated by the Parzen method, can be expressed in terms of inner products in a kernel feature space defined by a Mercer kernel, where the Mercer kernel, in fact, is the Parzen window. This link gives a theoretical criterion for the selection of the Mercer kernel, based on density estimation. Also, we show that the support vector machine (SVM), as an example of a well-known kernel-based learning algorithm, can be examined in an information theoretic framework, using weighted Parzen windows for density estimation