Distinguishing normal and abnormal tissues in nonclinical toxicity studies using unsupervised representation learning

Nonclinical toxicity studies are essential components of the drug development process and are based on labor-intensive visual examination of large numbers of glass slides under the microscope by pathologists. In this work, we present an artificial intelligence (AI)-based solution using an unsupervised representation learning model with a Bidirectional Generative Adversarial Network (BiGAN) for triaging normal and abnormal tissues and a machine learning-based severity grade classifier (SGC) for severity grade estimation on whole-slide images (WSIs) of rat liver. The BiGAN model was trained solely on vehicle control WSIs to learn normal histology and computed a tile-level similarity error, representing the deviation from normalcy, which was used to identify tile- and slide-level abnormalities. Our BiGAN model demonstrated an abnormality discrimination of receiver operating characteristic (ROC) area under the curve (AUC) of 0.77 with highest sensitivity for high-grade abnormalities. We utilized the similarity error data from BiGAN to train the SGC model and predicted slide-level severity grades with ROC AUC values ranging from 0.68 to 0.96 (minimal to marked) with most errors within ±1 grade, reflecting real-world pathologist grading variability. In addition, our heatmap visualizations of tile-level abnormalities revealed good to fair agreement with ground truth abnormalities in 70% of evaluated WSIs. By training the BiGAN on normal tissue slides, including minimal grade background abnormalities from vehicle control animals, our workflow eliminates annotation requirements and enables rapid adaptation to new tissues and species, paving the way for scalable AI models that streamline nonclinical pathology with minimal pathologist input.