Deep learning enables pathologist-like scoring of NASH models

Figures

• 

  Figure 1. Overview of the workflow for the automated scoring of fibrosis, ballooning inflammation and steatosis, the features correlated with non-alcoholic fatty liver disease (NAFLD)/non-alcoholic steatohepatitis (NASH). A whole slide scan stained with Masson’s trichrome is cut into tiles of two scales. Low magnification tiles are analyzed by a convolutional neural network (CNN) to recognize fibrosis. High magnification tiles are analyzed by three separated CNNs to recognize ballooning, inflammation and steatosis. The probability-like output of the CNNs per tile results in the distribution of the histopathological features in the liver. The distributions are aggregated to obtain one continuous quantity per liver sample for each of the features. The continuous features can be used directly or mapped to discrete pathologist-like scores.

• 

  Figure 2. Examples of the classes used to train the four convolutional neural networks (CNN) to recognize relevant features of the histopathological features in the Kleiner score. First row: ballooning, with classes 0 (no-ballooning), 1 (ballooning cells, see arrows) and ignore (insufficient liver tissue visible on a tile). Second row: inflammation with classes 0 (no inflammation), 1 (moderate inflammation), 2 (severe inflammation with clear inflammatory foci) and ignore. Third row: steatosis with classes 0 (<5% area coverage of vacuoles), 1 (>=5% and <33%), 2 (>=33% and <66%), 3 (>=66%) and ignore. Fourth row: fibrosis with classes 0, 1 (perisinosiodal or periportal fibrosis), 2 (perisinosiodal and periportal fibrosis), 3 (bridging fibrosis), 4 (cirrhosis) and ignore (not shown). Scale bars in first three rows (high magnification tiles): 50 µm, last row (low magnification tiles): 100 µm.

• 

  Figure 3. Examples of class activation maps for the four convolutional neural network (CNN) models. The color-coded maps highlight regions which were most discriminative for a certain class (for example a ballooning cell or a fibrotic band [CNN model and the selected class are shown on top]). Numbers on the top left in the activation map images are output probabilities for the respective classes.

• 

  Figure 4. Conversion of individual class scores to the weighted class per tile using the fibrosis model as an example. A whole slide image (WSI) of a liver section stained with Masson’s trichrome (top left) and the corresponding distribution of the normalized neural network output confidences for the fibrosis classes 0, 1, 2, 3 and 4. In this example, the majority of tiles are classified as fibrosis class 3. The output confidences are summarized as weighted class per tile. The insets show four tiles of a region with neighboring bridging fibrosis and an almost unaffected area. Numbers correspond to the weighted class per tile.

• 

  Figure 5. Deep learning-based average class per liver section (horizontal) compared to the pathologist’s score (vertical) on (A) ballooning, (B) inflammation, (C) steatosis, and (D) fibrosis. Dots are individual liver sections. Pathologist’s scores on the vertical axis have added jitter for visualization purposes. To obtain the discrete pathologist scores used in research and the clinic, thresholds are applied to the deep learning-based scores. Optimized thresholds were found using a Monte Carlo algorithm. Gray shaded areas show the resulting intervals.