A statistical analysis of soft-margin support vector machines for non-separable problems

The statistical properties of support vector machines (SVMs) for non-separable problems are studied. SVMs with hard margins are not always solvable for non-separable problems. Introducing soft margin alleviates this difficulty, but SVMs still fail to successfully solve these problems for heavily overlapped data. From the practical viewpoint, increasing the velocity of a soft margin depending of the number of examples is a way to adapt to increasing data generated from an identity distribution. However, systematic control of soft margin from the theoretical viewpoint is in development. A concept called “lifting up” for overlapped distributions gives a slightly different geometrical structure from the one for linearly separable distributions. In this study, the probability that an SVM can not solve a problem properly is mathematically derived in the one-dimensional case for both a hard-margin and a soft-margin. Some computer simulations confirm the theoretical validity.