On the biases in predictions of protein stability changes upon variations: the INPS test case

INPS (Fariselli et al., 2015) is a simple method to predict the protein stability changes upon single point variation. It is based on Support Vector Machine and it was trained using 7 input features that are: (i) Blosum62 substitution, (ii) residue mutability, (iii) molecular weight wild type, (iv) molecular weight variant, (v) hydrophobicity wild type, (vi) hydrophobicity variant, (vii) difference between Viterbi scores of the wild type and variant (Fariselli et al., 2015). The first two input features are not anti-symmetric, the last is anti-symmetric by construction, while the other features can be learned to be anti-symmetric. To enforce anti-symmetry INPS was trained using both direct and inverse variations (Fariselli et al., 2015). When the alignment is poor we can expect a deviation from the anti-symmetry. INPS3D (Savojardo et al., 2016), adds to INPS information about contact potential and solvent accessibility. The solvent accessibility is intrinsically non anti-symmetric, and for this reason INPS3D was trained only with direct variations (Savojardo et al., 2016). The Ssym dataset is a manually curated selection of variations from the ProTherm database (Pucci et al., 2018). It contains variations with experimental ΔΔG values for which the 3D structures of both the wild-type and variant proteins were solved by X-ray crystallography. Ssym consists of 684 variations, half of which are direct (reported in the literature) and half are obtained by anti-symmetry (Pucci et al., 2018). To overcome the bias problem, a new method, PopMuSiCSYM, was explicitly developed to furnish completely anti-symmetric predictions of the variations (Pucci et al., 2015, 2018). The authors of the Ssym dataset estimated the performance of fifteen selected predictors using the average bias parameter (<δ> = ∑(ΔΔGdir+ΔΔGinv⁠)/N) and the linear correlation coefficient (rdir-inv) between the predicted ΔΔG values of the direct and the corresponding inverse variations. A perfect predictor should have bias close to zero (<δ> = 0) and a correlation coefficient close to -1 (rdir-inv = −1). In the same paper, they also tested the performances of these fifteen predictors using the experimental ΔΔG values on both direct and inverse variations using the root mean square deviation (σdir, σinv) and the linear correlation coefficient between the predicted and experimental values (rdir, rinv). In Table 1 we report the predictor performances and biases taken from the original paper (Pucci et al., 2018) and we add those computed using INPS and INPS3D. From Table 1, it is clear that the partial anti-symmetry of the INPS input and its training performed on both direct and inverse variations paid off. INPS shows a very low bias (<δ>, just second best), the highest correlation between direct and inverse variations (rdir-inv) and the highest performance on the inverse variation sets (σinv and rinv). The graph of the correlation between direct and inverse variation predicted by INPS is presented in Figure 1. INPS and INPS3D prediction on Ssym. The INPS (black) and INPS3D (grey) predictions for the 342 pairs (direct and inverse) of single-point variations in Ssym (Pucci et al., 2018) are shown. ΔΔG for the direct (x-axis) and inverse (y-axis) variations are reported as dots in the graph. The ‘ideal’ relationship ΔΔGAB + ΔΔGBA = 0 is shown as a solid line Bias analysis on the Ssym dataset (Pucci et al. 2018) Notes: Standard deviations σ and the average bias <δ> are in kcal/mol. All the values, except those of INPS and INPS3D, are taken from (Pucci et al., 2018). Bold character highlights INPS method. Bias analysis on the Ssym dataset (Pucci et al. 2018) Notes: Standard deviations σ and the average bias <δ> are in kcal/mol. All the values, except those of INPS and INPS3D, are taken from (Pucci et al., 2018). Bold character highlights INPS method. The INPS training set includes some of the proteins in the Ssym dataset (which is also true for all the other tested methods reported in Table 1). For sake of comparison, we also computed the performances of a ‘blind’ INPS. In this case, each ΔΔG prediction is obtained by using a svm model that does not include the protein to predict or a similar one (sequence identity <25%) in the training set. The standard deviations and Pearson correlation coefficients computed for the blind-INPS are: 1.44 kcal/mol and 0.48 for direct variations and 1.45 kcal/mol and 0.47 for inverse variations. The correlation between the predictions of direct and inverse variations is -0.99 while the average bias (measured through the δ) is -0.06 kcal/mol. The structure-based INPS3D has the second best correlation between direct and inverse correlation but a considerably higher bias. This is expected given that INPS3D has not been trained on inverse variations and conains as input feature the solvent accessibility, which is not anti-symmetric. Finally, it must be stressed that the correlations reported in Table 1 are only intended to assess the anti-symmetricity of the methods, and not as estimators of their overall performances, which is a very difficult task that also requires an evaluation of stability of the dataset (Yang et al., 2018). A second dataset was built by Usmanova et al. (2018), by extracting high-resolution pairs of proteins from the Protein Data Bank (PDB) differing by one to ten amino acids. Here we test INPS on the subset of all the single-site variations extracted from the Usmanova et al. (2018) dataset, that is, all the pdb pairs whose sequences differ by exactly one residue. This subset comprises 1000 pairs of single-point protein variations. Although for this set the experimental ΔΔG values are not known, given its important size, it is very informative for assessing predictor biases. In Usmanova et al. (2018), bias for an individual variation is estimated as: <bias>=(ΔΔGAB + ΔΔGBA)/2. In Table 2 we add the INPS and INPS3D anti-symmetry indexes to those previously computed on other predictors (Usmanova et al., 2018). The bias of INPS is the smallest and the correlation between direct and inverse variations is the highest. INPS3D, while showing less anti-symmetricity than INPS, has the second best bias and the second best correlation. Bias analysis on the Usmanova et al. (2018) dataset Notes: First column: mean and standard error of the mean for the biases. Second column: r, Pearson correlation coefficient with the associated P-value, between direct and inverse variations. All the values except INPS and INPS3D, are taken from Usmanova et al. (2018). 0.0*: P-value set to 0, since it is undetectable by the machine precision. Bold character highlights INPS method. Bias analysis on the Usmanova et al. (2018) dataset Notes: First column: mean and standard error of the mean for the biases. Second column: r, Pearson correlation coefficient with the associated P-value, between direct and inverse variations. All the values except INPS and INPS3D, are taken from Usmanova et al. (2018). 0.0*: P-value set to 0, since it is undetectable by the machine precision. Bold character highlights INPS method. The very high correlation of -0.95 between direct and inverse variations obtained by INPS on this set is shown in the graph of Figure 2, to be contrasted with those reported for the other methods in the figures of the paper by Usmanova et al. (2018). In Figure 2, the points very far from the diagonal correspond to variations whose protein sequences are poorly aligned (number of aligned sequences <20). INPS and INPS3D prediction for the 1000 pairs (direct and inverse) of single-point variations (Usmanova et al., 2018). ΔΔG for the direct (x-axis) and inverse (y-axis) variations are reported as dots in the graph. The ‘ideal’ relationship ΔΔGAB + ΔΔGBA = 0 is shown as a solid line Recently, two groups (Pucci et al., 2018; Usmanova et al., 2018) compiled two datasets to test a very important bias that affects most of the computational methods designed to predict free energy changes upon protein variation. This bias is the lack of anti-symmetry in the ΔΔG predictions between direct and inverse variations. These studies verified this bias on several relevant methods. Here we add the test on INPS (Fariselli et al., 2015) that was specifically designed to take into account anti-symmetry problem using an input that was partially anti-symmetric (the HMM Viterbi score in particular). Here we show that INPS performs very well on both datasets proposed in the two papers. The fact that INPS uses only sequence information makes it less sensitive to structural rearrangement upon residue substitution and in turn more robust and suitable for testing any kind of protein variations. This work has been supported by EBA-PRISM an Israel-Italy collaborative project, the Israel Ministry of Science and Technology and Italian Ministry of Foreign Affair and International Cooperation. This research received funding specifically appointed to Department of Medical Sciences from the Italian Ministry for Education, University and Research (MIUR) under the programme “Dipartimenti di Eccellenza 2018 – 2022” Project code D15D18000410001. Conflict of Interest: none declared.