The discovery of new ionizable lipids for efficient lipid nanoparticle (LNP)-mediated RNA delivery remains a major bottleneck in RNA therapeutics development. Recent advances demonstrate the potential of machine learning (ML) models to predict transfection efficiency directly from lipid structure, enabling high-throughput virtual screening and accelerating lead identification. However, as new models for LNP transfection prediction continue to emerge, the lack of rigorous and standardized benchmarking poses a significant risk and may undermine confidence in their reliability for discovery. Here, we present a robust ML benchmarking framework for evaluating transfection prediction models based on ionizable lipid structures. This framework systematically benchmarks diverse molecular representations—including Morgan fingerprints, Expert RDKit descriptors, and Grover graph-based embeddings—paired with a broad range of ML architectures spanning traditional models, feedforward neural networks, and state-of-the-art graph-based methods. In addition, the presented framework supports assessment of model generalization through Murcko scaffold splitting and evaluates prediction reliability beyond standard regression metrics by incorporating analyses of relative error distributions and ranking accuracy. Using a curated dataset of 1100 unique ionizable lipid structures derived from the HeLa transfection dataset originally reported by Xu et al. 1 , we show that within this framework, models leveraging explicit molecular substructure encoding—particularly multilayer perceptrons (MLPs) trained on Morgan fingerprints combined with Expert descriptors—consistently achieve the highest predictive accuracy and should serve as essential baselines for the development of new, more sophisticated models. In contrast, some current graph-based models, including AGILE, Chemprop, and KPGT, tend to show comparatively lower accuracy. The presented framework provides a standardized, transparent, and comprehensive benchmarking resource that enables meaningful comparison of emerging architectures and establishes strong baselines for future development of predictive models in lipid-based RNA delivery.
Mehradfar et al. (Tue,) studied this question.