ML-Guided Prioritization in Virtual Screening: Docking-Supervised, Potency-Supervised, and Hybrid Strategies

The expansion of synthetically accessible chemical space has created a major scaling problem for structure-based virtual screening. Modern libraries now contain millions to billions of compounds, making exhaustive molecular docking increasingly expensive and often impractical. In this setting, the central challenge is no longer only how to perform docking, but how to decide which compounds should be docked in the first place. Machine learning (ML) has emerged as a useful solution to this prioritization problem by helping allocate computational effort more selectively across large chemical libraries. This review examines ML-guided prioritization strategies used around molecular docking workflows. We organize the field primarily by source of supervision, distinguishing docking-supervised approaches trained on docking-derived outputs, potency-supervised approaches trained on experimental activity measurements such as IC₅₀ or pIC₅₀, and hybrid strategies that combine both evidence types. We also consider workflow placement as a separate dimension, including pre-docking filtering, active-learning-guided docking, docking-integrated scoring, and post-docking rescoring. Across these settings, ML can compress search space, improve screening efficiency, and enrich candidate selection, but its performance remains constrained by docking bias, assay quality, domain shift, and limited prospective validation. Overall, the literature suggests that the most effective role of ML in virtual screening is not to replace molecular docking, but to guide where docking and experimental effort should be concentrated within increasingly large chemical libraries.