Identification of non-hydroxamate histone deacetylase 8 inhibitors using deep learning-based screening of two-dimensional molecular images

Histone deacetylase 8 (HDAC8) is a zinc-dependent enzyme that deacetylates proteins and is involved in cancer development, making its inhibition a potential anticancer strategy. Most reported HDAC8 inhibitors contain a hydroxamate zinc-binding group; however, hydroxamate moieties have been associated with potential mutagenicity, motivating the search for non-hydroxamate inhibitors. In this study, we applied a two-dimensional (2D) molecular image-based deep learning approach to identify non-hydroxamate HDAC8 inhibitors. We first assessed DEEPScreen, a reported convolutional neural network-based platform that uses a 2D molecular image model; however, the original architecture exhibited limited predictive performance for HDAC8 (F1-score < 0.7). Incorporating a ResNet architecture into DEEPScreen improved the classification accuracy of HDAC8 inhibitors (F1-score = 0.84) and outperformed conventional descriptor-based methods. Using the trained ResNet model, we screened the Osaka University library and experimentally assessed the resulting virtual hit compounds. This workflow identified a non-hydroxamate HDAC8 inhibitor hit S29 (61.6% inhibition at 30 μM; IC 50 = 14.1 μM). Subsequent straightforward structural optimization revealed that the α-methyl substituent on the ester moiety was not essential for HDAC8 inhibition and resulted in the identification of compound 1 , a more potent non-hydroxamate HDAC8 inhibitor (72% inhibition at 30 μM; IC 50 = 7.8 μM). This study demonstrates that 2D image-based deep learning—integrated with experimental validation and structural optimization—can facilitate the discovery of non-hydroxamate HDAC8 inhibitors and expand the accessible chemical space for this target. • A ResNet-based deep learning model enabled 2D image–based virtual screening for HDAC8 inhibitors. • Screening of the Osaka University library with the trained deep learning model identified S29 , a non-hydroxamate hit compound that showed 61.6% HDAC8 inhibition at 30 μM. • Straightforward structural optimization of the hit S29 yielded a more potent non-hydroxamate HDAC8 inhibitor 1 with an IC 50 of 7.8 μM.