How to build machine learning models able to extrapolate from standard to modified peptides

Bioactive peptides are an important class of natural products with great functional diversity. Chemical modifications can improve their pharmacology, yet their structural diversity presents unique challenges for computational modeling. Furthermore, data for standard peptides (composed of the 20 canonical amino acids) is more abundant than for modified ones. Thus, we set out to identify whether predictive models fitted to standard data are reliable when applied to modified peptides. To do this, we first considered two critical aspects of the modeling problem, namely, choice of similarity function for guiding dataset partitioning and choice of molecular representation. Similarity-based dataset partitioning is an evaluation technique that divides the dataset into train and test subsets, such that the molecules in the test set are different from those used to fit the model. We demonstrate, across four peptide function prediction tasks, that chemical fingerprint-based similarity measures outperform traditional sequence alignment-based metrics for partitioning standard peptide datasets, challenging usual practices. Similarly, we have found that chemical fingerprints are the best option for building predictive models able to interpolate between classes (standard to standard and modified to modified). Despite a 50% drop in performance, chemical fingerprints are also the best choice in the more challenging extrapolation scenario (standard to modified). All code and data necessary for reproducing the experiments are available in Github (https://github.com/IBM/PeptideGeneralizationBenchmark).