A Critical Assessment of Feature Selection Methods for Biomarker Discovery in Clinical Proteomics

In this paper, we compare the performance of six different feature selection methods for LC-MS-based proteomics and metabolomics biomarker discovery—t test, the Mann–Whitney–Wilcoxon test (mww test), nearest shrunken centroid (NSC), linear support vector machine–recursive features elimination (SVM-RFE), principal component discriminant analysis (PCDA), and partial least squares discriminant analysis (PLSDA)—using human urine and porcine cerebrospinal fluid samples that were spiked with a range of peptides at different concentration levels. The ideal feature selection method should select the complete list of discriminating features that are related to the spiked peptides without selecting unrelated features. Whereas many studies have to rely on classification error to judge the reliability of the selected biomarker candidates, we assessed the accuracy of selection directly from the list of spiked peptides. The feature selection methods were applied to data sets with different sample sizes and extents of sample class separation determined by the concentration level of spiked compounds. For each feature selection method and data set, the performance for selecting a set of features related to spiked compounds was assessed using the harmonic mean of the recall and the precision (f-score) and the geometric mean of the recall and the true negative rate (g-score). We conclude that the univariate t test and the mww test with multiple testing corrections are not applicable to data sets with small sample sizes (n = 6), but their performance improves markedly with increasing sample size up to a point (n > 12) at which they outperform the other methods. PCDA and PLSDA select small feature sets with high precision but miss many true positive features related to the spiked peptides. a recall and precision for data sets of level and of for selecting features related to the spiked the classification error In this paper, we compare the performance of six different feature selection methods for LC-MS-based proteomics and metabolomics biomarker discovery—t test, the Mann–Whitney–Wilcoxon test (mww test), nearest shrunken centroid (NSC), linear support vector machine–recursive features elimination (SVM-RFE), principal component discriminant analysis (PCDA), and partial least squares discriminant analysis (PLSDA)—using human urine and porcine cerebrospinal fluid samples that were spiked with a range of peptides at different concentration levels. The ideal feature selection method should select the complete list of discriminating features that are related to the spiked peptides without selecting unrelated features. Whereas many studies have to rely on classification error to judge the reliability of the selected biomarker candidates, we assessed the accuracy of selection directly from the list of spiked peptides. The feature selection methods were applied to data sets with different sample sizes and extents of sample class separation determined by the concentration level of spiked compounds. For each feature selection method and data set, the performance for selecting a set of features related to spiked compounds was assessed using the harmonic mean of the recall and the precision (f-score) and the geometric mean of the recall and the true negative rate (g-score). We conclude that the univariate t test and the mww test with multiple testing corrections are not applicable to data sets with small sample sizes (n = 6), but their performance improves markedly with increasing sample size up to a point (n > 12) at which they outperform the other methods. PCDA and PLSDA select small feature sets with high precision but miss many true positive features related to the spiked peptides. a recall and precision for data sets of level and of for selecting features related to the spiked the classification error the of and of for biomarker and on biomarker and with from other of to the of a and which to the of a The selection of biomarker the of studies and the of to and with high and The of biomarker on and of the from samples are the of the which the of the the accuracy of the methods to and to the biomarker from and the performance of the applied methods the selection of a list of compounds with the to of biomarker of a biomarker and a biomarker biomarker and the and to that to many compounds a of The of data and analysis to select a of candidates, which are to of samples for shrunken component discriminant least squares discriminant vector machine–recursive features shrunken component discriminant least squares discriminant vector machine–recursive features applied biomarker of of a of data by to the related to compounds a feature which and to samples and related to data and the of features to features on the performance of the data a for analysis of data for biomarker data sets a of features to the sample features are by their and and the ideal they and to and In LC-MS-based proteomics and metabolomics sample analysis that to the of samples to a level that the of features a data the of biomarker on feature selection methods that with a sample size and a high of features. of the and the of the to the selected biomarker a set of a using methods for biomarker and cerebrospinal fluid range nearest shrunken centroid principal component discriminant analysis partial least squares discriminant analysis support vector machine–recursive features elimination Mann–Whitney–Wilcoxon cerebrospinal fluid range nearest shrunken centroid principal component discriminant analysis partial least squares discriminant analysis support vector machine–recursive features elimination Mann–Whitney–Wilcoxon selection on classification methods that are by feature selection methods which have feature selection and that to select a list of that the classification performance for sample of feature selection methods are to sample a sample feature selection methods test for sample classification methods the nearest shrunken centroid method have feature selection other classification methods principal component discriminant analysis and partial least squares with discriminant analysis should with a feature selection are feature selection that the classification using support vector that data methods without the to select features for biomarker methods to samples but the of that the of the classification data the of proteomics methods with feature selection have to the of the and have of feature selection data for feature to biomarker of methods to and are a biomarker and the many data of biomarker from selection and with of classification methods with to their classification and performance have that on the for data sets with sample sizes from to the performance of for metabolomics the of sample size metabolomics data set on the performance of different and a that for small sample the of methods by samples and and the to to sample size the of that the of the feature a classification which the of features the of samples by a a set of features to the feature with the sample size and the of classification methods that with accuracy the sample size and the feature size are In this we different classification methods on feature selection of spiked data sets that the of a biomarker for biomarker other studies with to sample size and selecting the feature selection method for a data We the univariate t test and Mann–Whitney–Wilcoxon test (mww with multiple testing the a and to multiple nearest shrunken centroid of multiple by shrunken of by nearest shrunken with to support vector machine–recursive features elimination selection for classification using support vector PLSDA least squares for and PCDA analysis by principal component PCDA and PLSDA were with the a feature selection a method to methods were with data sets different sample and and of the compounds. were from human urine and porcine cerebrospinal fluid samples that were spiked with a set of peptides at different concentration levels. samples were peptides spiked at and high concentration levels. The performance of the classification methods with feature selection was on their to select features that were related to the spiked peptides. true were data set, we performance on the harmonic mean of precision and and the geometric mean of urine samples were from and the range of to were from each urine urine sample was The urine was spiked with a of with peptides at different and a and the of the and each concentration the sample was using were with a of using the proteomics proteomics with of to and and a list of with of from the peptides are by the data and compounds. related to but they related to data features are features selected by the feature selection and negative features are the and negative features are true to they to spiked peptides not peptides the of the to features a data set that to spiked peptides. of the sample data and method of of features related to spiked peptides are the and a for analysis of data for biomarker that this their The was with and the of data sets were from the of sample with and high levels. sample were by different and sample which data sets of different and of spiked peptides. a of the of the data and a of the sample size and and of spiked peptides of the data of the sample that were to data sets and was to select for the of each of feature selection methods and data sets for sample was to spiked human urine samples sets and and porcine sample was to spiked and samples set and size samples to class class = class = samples to spiked samples samples class = spiked class = samples were from each of the and spiked class = of class = of samples were from each of the and samples from and samples were from each of the and spiked class = of class = of samples were from each of the and samples from and a set and a class sample size of spiked class was from samples of and spiked class from samples of to to The six data sets were from different sample sizes and by and the data set and a high and and for from were for the high spiked class and sample were for the spiked class for data sets with high set from were for the high spiked class and sample were for the spiked class for data sets with set were selected for data sets and with six samples and data sets and with samples from the and samples were for data sets and with samples selection for each of feature selection methods and data sets was each using a different of samples the and of the The sample and analysis of the of porcine was by a for analysis of data for biomarker and proteomics The spiked samples were by with of a of at of and samples were and was from of spiked was with a data was using the centroid to size and analysis data were using proteomics a to that for the human urine a of features were this data set, from which to the spiked on and of features set was from and high level sample with a sample size of and with to and of peptides to the human urine data set The univariate t test features to their and not a classification the data sets to samples was to test the of the which this the of the We a univariate the mww the data sets a of we the for multiple testing using the the a and to multiple feature was the was multiple testing The to a set of features that the classification error the of class a set of samples using by features that not to the of the shrunken class method was by for the classification of samples on data of multiple by shrunken of by nearest shrunken with to The for this method classification methods with feature selection this were to and The a feature class and centroid the the mean and the mean to the The error using the of the feature a of the to to small and the The from this and features shrunken negative are test sample to the class to which the class The discriminant for class and for test sample which the of the each feature the test sample and the shrunken centroid by the of class to a the test sample and the shrunken centroid of the feature elimination but classification of the test sample to the shrunken centroid at a we this a on the discriminating we the class which the of sample to class the class for each test sample and for each the of the true class for each sample We have on which we select the of the that the classification error and the that the of true class In the was on the true class of the test data set, a and a the was the was applied to the data set the to the feature on this feature set, the discriminant and the true were for the test data set the to the classification each the a different for the we the of the class at each from the at the of the of the true class a of the for the of the and small class the data The at the of the true class was to select the feature set using samples the data support vector by a method that a that a data separation by a that the to the nearest data point of sample the of features. The of the to a with that the on their The a discriminant that data to a have and for classification and which the feature size the of for the of support vector classification of and support vector for classification on vector feature selection for classification of of using concentration and support vector biomarker selection by support vector for support vector to of and vector to and support vector for the of peptides for and of by of using support vector feature of using support vector have to for feature selection selection for classification using support vector selection for support vector of the on from selection for In this we a linear with a feature elimination for a feature selection method by selection for classification using support vector the vector which to the of the selection The using the features the set, the vector the feature with the from the classification the using the features. the feature we a The of features determined the the feature with the the classification error on the set of features a classification error for a set of and the of each feature by The of features the feature set that the mean classification select the feature a applied to the feature the In the the classification error of the feature using test data The PCDA and PLSDA the features feature component analysis features by linear that the the with linear discriminant analysis the applied to the by analysis by principal component for feature selection and classification using classification for the proteomics data of discriminant of and PLSDA a method studies using principal component discriminant partial least and of and of using of urine from to using for using high performance metabolomics classification of by for of using and to for classification and other least squares for least a for the analysis of of of a analysis which the the class are by to a the data set and the and of this method the of of PLSDA for PLSDA data PLSDA In we a of and for PCDA and In the of principal and the of partial least squares are the the of each a a method to to the features on their discriminant the The the classification error from a that the of the and different of features on the feature The feature size that the classification error the selected the of features. on this the of and the of features are select the feature we from the In the the using the of and the of the discriminant of the features at the using each The feature set selected from this feature the size of which by the The complete of PCDA and PLSDA selecting the feature set We the of the on their performance biomarker selection by each to feature In the discriminating features the data sets were the feature sets were to features related to the spiked peptides true and were each feature set by a method a were to compare and the of the performance the of selected features to features that are related to the spiked peptides. to the of features that are related to the spiked peptides to features selected by a The geometric mean accuracy the of a method to negative related to the spiked features and positive features the performance of the feature selection the to true positive and true negative features. The a that on the classification of true positive features on recall and the of the features that were selected of the feature set to features and the of the true positive features. the of the features the selected feature sets and the of a and precision are the the set to and of precision > In we set to We the we were the of which recall and precision to the the to features by a the to the class of the a of the that were to compare the performance of different feature selection = = = a In to the feature sets on the the performance of the methods on the classification error of the that on selected the of the selected and the of the selected feature the of the on the of the feature set, the selected feature size on performance and of the In biomarker the size of the feature set the of the of selected peptides and their the of In this we data sets which the true were recall and precision and different The selecting a set of biomarker by the of the selected feature set of the selection the feature selection method feature sets multiple of using different sample the of the feature sets the classification error with to test a for selecting feature from different the list of features different were data sets from human urine and porcine samples spiked with a range of peptides at different concentration a of biomarker the of the of the with the from for each of a biomarker selection method and a data The was the of the not the of the feature size and the of true that are the feature the recall and precision on the of true positive features the feature the and which are a of and true negative of with error of recall and precision and and were to compare and the performance of each method with to sample and of spiked of and of and of the of recall and precision for each of feature selection methods and data and precision were not for the t test and the mww test on data sets and samples by for the mww test on data sets and samples by t on data sets and and univariate test and mww on data sets and were samples class without error the the of recall and and of the and for each of feature selection methods and data The and were not for the t test and the mww test on data sets samples by for the mww test on data sets and samples by t on data sets and and univariate test and mww on data sets and were samples class without error the the of recall and of the feature selection methods with of with data sets and data sets are of different urine for and porcine for a sample size of and a of spiked peptides. are of samples of data sets with of spiked peptides. In data of compounds by a of the data sets and which are of human urine samples of different sample sizes and to and data set and and of spiked peptides to and the data set compounds data sets the of spiked peptides the small to data sets and The of spiked peptides to from data sets the of high and with sample with a concentration In compounds a of the samples sample the concentration of spiked the of and on the data set that for data sets and the t test high precision and precision with For data the t test the complete list of true with and the other feature selection methods. The and the of have on the performance of the t test by the of data sets and For data the mww test applied of the sample mww and that the t test the mww test of discriminating features the the data sets with of compounds the sample size sets and six samples of the of the of spiked peptides. improves markedly the of samples class to sets and at which point are the methods. from univariate the sample size a on with a performance for samples class and performance for samples were with data sets and a sample size of and data sets and sample sizes of and that t test performance by the of the biomarker the a sample size to performance of true positive selection with a of and that the high of true the from data sets with samples and a of spiked compounds data set are by a high of precision a high univariate feature selection methods the and for data sets data sets and that the recall and precision the performance with to and to that univariate a recall but a precision data set which to a of true data set The was for data sets and with samples The and that the performance of univariate feature selection methods by the sample size and and by the of peptides. that univariate feature selection methods the methods with a class size of samples the of of with a sample size of the of peptides The of the of true positive features with a of positive features data sets with sets and of the the of high to high not a complete list of true positive recall the of human urine data set and for data set this the of true for the sample size of samples class with of peptides sets and but at the of selecting a high of features that were not related to the spiked peptides data sets and to high recall but precision to the other data sets and PLSDA with to precision for data sets with a of samples of the of selected features data sets with of compounds sets and that are not related to the spiked peptides the for data sets with a of samples sets and performance data which and from increasing sample size to precision features that are not related to the spiked the of true precision without a of that the of the with increasing sample size on the error The recall and precision data sets with and samples was not data set and recall and precision were for data sets with class separation and to with small class separation data sets and data sets and which for the and the selected the of features data the of selected true was for of the other many were the of selected features data sets with of peptides and selected features data sets and by the performance of true positive was the of selected features the porcine data set and true were the selected features from the human urine The of data sets to a selection of features without to true a performance the sample size and of spiked peptides set with a recall of and a precision of The of selected features precision the The for other a of that many which that the of a feature not a for feature the size of the feature set selected by with human urine porcine samples that the not for the selection of biomarker For of the data PCDA to select a of true positive features and a of features recall and high for sample data sets and PCDA a to biomarker the selected feature list a high of true positive features. In to and the univariate test and mww test), PCDA to select features from data sets with high set this for data set the of selected features of the selection true positive features of high precision of data sets and a of this to of the selected biomarker of of the true are using this selection of a of true positive features with recall by the of the and The precision of the PCDA method data sets with sample sizes of the and of peptides data sets and which the method for biomarker selection data sets a sample size to samples for data sets and that the performance of PCDA not by the of the of spiked peptides. The of PLSDA the high precision of the sample size the class separation data sets of spiked human urine the analysis of spiked porcine precision to a of true the selected features of 6), which related to the of spiked features with human urine The selection of a of true positive features with high precision biomarker and the with The of the feature selection with increasing sample by the precision at a the of selected true small to the of true positive features on the spiked peptides The of selected true from the that PLSDA and PCDA select with high the features with error and that the for the for selecting features related to the spiked peptides are PCDA and PLSDA select a of that the of spiked peptides PLSDA not by the of the of spiked peptides the data set, the of recall and precision data sets and are sample size recall improves In to PLSDA high precision on data sets with a sample size data sets and this method for selecting true positive features data sets with samples methods from a sample of are by the and of spiked peptides. The univariate t test and mww test are by the and of spiked peptides on the of and they a sample size to methods that feature to selecting a feature set, PCDA and are not by the of spiked peptides sample The performance of of the of spiked peptides and sample the of the data set were sample size and high of spiked univariate methods test and mww they of the true a feature were the and methods to methods the sample size was small six samples the of spiked peptides was high and the of spiked peptides was biomarker which compounds have to and to to sample on the and the other of feature the recall and and The methods PLSDA and PCDA feature sets that are high precision at the of a applicable to data sets and a biomarker are the classification error rate of true class the of and the of the the a of the test we assessed the of feature selection performance data sets with sample sizes of and and on and of the classification error and the of true class the of The of and was using the for a data In was a for the to with the of samples error for which to the performance of the selected feature The of the classification error rate the of true class each the of the that a different of classification error from a the error the which the for the and the classification error the using which the error rate of the for sets of The classification error rate of true class on the sample size of the data the of spiked peptides not have on classification error on the of true class The error the for data sets with high of spiked peptides and for data sets with of spiked peptides of the applied The of the classification error rate the of true class to different for the determined the from samples sets and assessed a classification error of true class a the which the a that were to the this are The of the error with increasing sample that the of the with increasing sample The that not to rely on a classification from a from a data set with six samples the of the the of the selected feature sets by each we the of features that were selected at least to the of features that were selected each The of the feature set not was for feature selection methods for data sets and of the sample the of the feature set was not determined for the mww test the t test for data sets and of the of a for methods. this the feature set of by the high of the of features to the of features. We have assessed different feature selection methods with to their to biomarker from a of data sets that were analysis of human urine porcine feature selection methods were and their performance on they true that are related to the spiked and they other for data sets with different sample sizes and and of spiked peptides. We six from this methods from a sample methods and methods are to the and of spiked methods are by the of spiked peptides. on data sets with to selecting that the vector not a for feature methods PCDA and PLSDA at high precision by recall they are with to selecting true positive PLSDA PCDA for data sets with a sample size the of spiked peptides The the recall and precision of sample size and and of spiked peptides. The performance of feature selection methods on the of discriminating the the of for data of the feature selection methods on the for data sets with different and of spiked peptides and sample The a the of a feature selection method and should select the method for biomarker biomarker to support to a discriminating feature set with a of and with the to sets of samples on this PLSDA a of discriminating features are to support miss features that In that a recall and with a of true at a positive In which data from samples are samples univariate test mww test with multiple testing are to biomarker with high For with sample samples class the to select biomarker that on from data sets with a small sample classification and the for are to biomarker selection the of the feature selection methods and the of test data sets to other feature selection we have the and data with of and spiked compounds and of features the of on with