Plants meet machines: Prospects in machine learning for plant biology

Machine learning approaches are affecting all aspects of modern society, from autocorrect applications on cell phones to self-driving cars to facial recognition, personalized medicine, and precision agriculture. Although machine learning has a long history, drastic improvements in these application areas recently have been driven by improvements to computational infrastructure; increased computing power; increased ability to collect, manage, and store very large amounts of data; and algorithmic advances. Multiple types of machine learning have been developed, each with its own techniques, strengths, and weaknesses, making certain approaches better matches for certain problems than others. Supervised machine learning and the use of neural networks (e.g., deep learning; Table 1) underlie much of the recent accelerated application of machine learning to many biological problems, including those across a range of scientific questions in plant science. For example, deep learning technologies have recently achieved impressive performance on a variety of predictive tasks, such as species identification (Unger et al., 2016; Carranza-Rojas et al., 2017), plant species distribution modeling (e.g., Zhang and Li, 2017; Botella et al., 2018), weed detection (Yu et al., 2019), and mercury damage to herbarium specimens (Schuettpelz et al., 2017). They are also being applied to questions of comparative genomics (e.g., Xu and Jackson, 2019) and gene expression (Mochida et al., 2018) and to conduct high-throughput phenotyping (e.g., Singh et al., 2016; Ubbens and Stavness, 2017) for agricultural and ecological research. Moreover, novel approaches are poised to revolutionize studies of plant phenology (e.g., Pearson et al., 2020) and functional traits through application to more than 30 million images of herbarium specimens now available at iDigBio (http://www.idigbio.org) as well as other digital repositories. The application of machine learning methods to extract data from herbarium specimens has grown and diversified in a few short years, beginning with species identification in a specific geographic region (e.g., Unger et al., 2016). Subsequent attempts to use deep learning to tackle the difficult taxonomic task of identifying species in large collections of herbarium specimens showed that convolutional neural networks trained on thousands of digitized herbarium sheets are able to learn highly discriminative patterns (e.g., Carranza-Rojas et al., 2017). These results are very promising for extracting a broad range of accurate annotations in a fully automated way. Such approaches are also being applied to identification of plant phenophase (i.e., bud, flower, fruit), which is important for assessing the effects of climate change on plant growth and reproduction and for comparing plant responses with those of pollinators, migratory birds, and other species that rely on plants for food and/or nesting sites (see, e.g., Lorieul et al., 2019; Pearson et al., 2020; Brenskelle et al., 2020; Goëau et al., 2020). Likewise, other evolutionary or ecological traits, such as leaf shape and size, leaf margins, and flower color, could also potentially be scored from images of herbarium specimens. However, despite the promise of applying deep learning to herbarium specimen images to address a range of questions, this emerging field also raises challenging methodological questions about how to avoid any bias and misleading conclusions when analyzing the produced data. Indeed, as for any statistical learning method, convolutional neural networks are sensitive to bias issues, including the way in which the training data sets are built. Moreover, as good as the prediction might be on average, the quality of the produced annotations can be very heterogeneous from one sample to another, depending on various factors such as the morphology of the species, the storage conditions in which the specimen was preserved, and the age of the specimen when imaged. Given both the opportunities and challenges, additional research into the application of machine learning approaches to herbarium specimen images is needed to enable greater applicability to a broad range of scientific questions. The field of machine learning is moving rapidly, with the development of alternative approaches that may be best suited to specific questions, data sources, and analytical techniques. This special collection of articles in Applications in Plant Sciences presents 16 papers, published across two issues of the journal, that explore methods and applications of machine learning to studies of plant ecology, morphology, genomics, and agriculture. The first issue comprises eight papers and focuses on applications to images of herbarium specimens, on topics from phenology to herbivory. The second issue includes papers that address a broader range of topics, data, and biological scale. We summarize the content of both issues here. Plant phenological research has seen major advances in recent years through the use of herbarium specimens (Willis et al., 2017). Herbarium specimens collected over the past three centuries provide insight into flowering, leaf-out, and fruit timing globally and across plant phylogeny (Davis et al., 2015). A major hurdle, however, is that to harness the full power of herbarium specimens for phenological research requires counting reproductive structures, which can be time consuming. Thus, automated recognition of reproductive structures on herbarium specimens is a key goal in current phenological research (Lorieul et al., 2019; Pearson et al., 2020). Two papers in this special issue address plant reproductive phenology. To make use of the extensive volume of herbarium specimens for examining angiosperm reproductive phenology, Goëau et al. (2020) applied a state-of-the-art segmentation approach (mask R-CNN) to automate locating, segmenting, and counting reproductive structures on images of herbarium specimens of Streptanthus tortuosus Kellogg (Brassicaceae). Phenological stages (i.e., buds, flowers, immature fruits, mature fruits) are distinct in S. tortuosus, and specimens were scored for phenophase. Evaluation of the performance of the method indicated that it shows particular promise in identifying the number of reproductive structures (accuracy was nearly 80%), but the accuracy of the results varied with respect to the training annotations, the type of reproductive structures scored, and the size of the reproductive structures. Although promising, these results suggest that further refinement is needed, and it is unclear how well the approach will scale to other species with different floral morphologies and perhaps less well-differentiated phenophases. To train machine learning algorithms to do this, however, will require massive input data to data-hungry machine learning algorithms. In this issue, Brenskelle et al. (2020) assess the conditions needed for volunteers to help gather these data. The authors test for the effects of training type (in person or online), career stage, plant taxon, and phenological stage scored on the accuracy of volunteer-provided phenological data from herbarium specimens. Regardless of expertise and training method, users provided highly accurate data, although data from people trained in person were more accurate than those trained online. This study provides a best practices guide for collecting annotation data. Importantly, the authors also demonstrate that online citizen science platforms might be able to provide accurate annotation data that can then be used downstream to train machine learning algorithms to recognize phenological stages. Morphological variation, coupled with variation in the quality of herbarium specimens, leads to noise and potential bias in automated coding of characters from specimen images. Image segmentation is a computer vision algorithm that groups together pixels of an image that have similar attributes and generates a mask for each focal object in the image, such as a flower in an image of a herbarium specimen. Application of masks, such as those applied to plant phenophases, can help to reduce noise and bias. White et al. (2020) developed a workflow to apply segmentation masks to plant images using deep learning. Focusing on ferns, they generated a model that could segment herbarium images automatically, efficiently, and accurately across the morphological diversity of this clade. Although their study was restricted to ferns, the workflow is generalizable to all herbarium images and, with modification, may be applicable to other clades of plants with highly different morphologies. Plants and insects have been interacting for 400 million years, and these interactions have likely driven diversification of both clades. The fossil record shows evidence of herbivory, providing a glimpse into long-term patterns of plant–herbivore interactions and evolution. However, how herbivory changes over shorter timescales and geography is much less clear. Despite the fact that botanists generally attempt to collect specimens that are free of herbivore damage, herbarium specimens offer a view of plant–herbivore interactions over the past three or four centuries, with the potential to infer spatial and temporal patterns of herbivory, including response to climate change (Meineke and Davies, 2018; Meineke et al., 2018). However, manual scoring of insect damage to herbarium specimens is extremely laborious, and the possibility of applying machine learning to quantify the patterns and extent of insect damage to plant specimens is appealing. Meineke et al. (2020) initiated machine learning methods to explore their ability to classify multiple types of herbivory (and its absence) across a pair of divergent plant species. Although herbivory could not always be classified with high accuracy, the use of hand-drawn boxes to locate areas of potential herbivory increased the accuracy of herbivory classification to 81.5%. The authors further identify ways to expand the accuracy of the models in future applications, potentially paving the way for exploring patterns of herbivory in relation to climate change, invasive species, and more. The contributions of machine learning to the plant sciences, especially for automated species identification from images of digitized herbarium specimens, is showing great promise (Schuettpelz et al., 2017; Wäldchen and Mäder, 2018). This is especially true for genera with only slight morphological variation among species, particularly when compounded by hybridization and the presence of infraspecific taxa. Pryer et al. (2020) have built on this work with Equisetum L., a distinctive genus with 15 extant species complicated by morphological plasticity and frequent hybridization events that have resulted in a disproportionately high number of misidentified herbarium specimens. Equisetum includes two relatively distinct species (E. hyemale L. and E. laevigatum A. Braun) and a widespread, sexually sterile hybrid between them (E. ×ferrissii Clute) (Rutz and Farrar, 1984; Des Marais et al., 2003). The challenges faced here result from the cylindrical nature of the stem, which results in dramatic differences in specimen images due to factors such as the geometry of the flattened stems, the number of stems included on a single sheet, stem colors, and imaging parameters. Compounding the variations among images is the fact that accurate identification has more to do with the appearance of stem nodes and strobili than other features. Through successive testing of several models, Pryer and colleagues discovered that, out of 30 test images, 27 were classified correctly. Although the number of specimens is probably too small to be broadly generalizable, E. hyemale images were correctly classified in nine of 10 cases, E. ×ferrissii images in eight of 10 cases, and E. laevigatum images were never confused, resulting in an accuracy of 90%. These results suggest strong potential for machine learning's impact on the accurate determination of closely similar taxa. In their contribution, Ott et al. (2020) outline the development and output of GinJinn, object-detection software designed to extract leaf images from herbarium specimens based on the TensorFlow (Abadi et al., 2016) object-detection application programming interface (API), an API designed to make supervised deep learning object detection accessible for plant scientists. Although GinJinn makes heavy use of TensorFlow's API, the authors maintain that GinJinn is not merely a wrapper for the API; it also provides data preprocessing, project set up, pretrained model download, simple model exporting, and the use of trained networks for the extraction of bounding boxes from newly acquired data. GinJinn was tested on a data set of 286 JPEG images of preserved plant herbarium specimens provided by the herbarium of the Botanic Garden and Botanical Museum Berlin-Dahlem, Berlin, Germany. The images were annotated using the free open-source tool LabelImg version 1.8.1 (https://github.com/tzutalin/labelImg), resulting in a total of 889 annotated intact leaves within 243 images of herbarium specimens of two species of Leucanthemum Mill. (the diploid L. vulgare Lam. and the tetraploid L. ircutianum DC.) known for their high variability in leaf shape. The task is complicated by the rare occurrence of intact leaves versus non-intact leaves in these species. Using 183 specimens as the training data set, the GinJinn pipeline extracted one or more intact leaves in 95% of 61 test images. A major challenge to cataloging and describing plant diversity lies in the development of high-throughput technologies that facilitate rapid discovery of new taxa hidden in the backlog of still-to-be processed herbarium specimens. The 400,000 plant species currently known to science have required more than 250 years to name and classify, and as many as 70,000 flowering plant species are likely yet to be discovered (Joppa et al., 2011). Many of these may well be among the estimated one million specimens currently backlogged in herbaria. From Little et al.'s (2020) perspective, this renders herbaria largely untapped resources for the new and rapidly developing use of artificial intelligence (AI) in taxonomic research (Wäldchen and Mäder, 2018). To capitalize on this enthusiasm and encourage an increasing number of AI specialists to devote attention to algorithms that can produce species identifications, these authors mounted a Kaggle competition platform to crowdsource effective machine learning algorithms for analyzing plant specimen images. The competition data set included 46,469 images representing 683 species of the family Melastomataceae (Tan et al., 2019). In just two months, 254 models were developed that automatically identified the taxa among these digital representations, with the top four models identifying specimens to species with >88% accuracy. Trait extraction from herbarium specimens can be laborious and time consuming, making the process an excellent candidate for the application of high-throughput machine learning protocols and algorithms. Here, Weaver et al. (2020) describe and test LeafMachine, an automated, open-source software tool for recognizing and measuring leaf dimensions from herbarium specimens and single leaf images across a wide range of largely woody taxa (trees, shrubs, lianas), although some herbaceous taxa were also included. The tests show varying results based on image resolution, specimen presentation, leaf condition, and whether leaf clumping was present. Of ~1000 images containing measurable leaves as confirmed through assessment, LeafMachine produced morphometric information for at least one leaf in 82.0% of high-resolution images and 60.8% of low-resolution images, suggesting positive results to the researchers but with a need for enhancement as machine learning technologies advance. The second set of papers explores a broad range of topics, beginning with application of machine learning approaches to agriculture. The use of herbicides to control weeds in agricultural fields is costly both economically and environmentally, and alternatives are needed, especially for organic farming. Possible solutions include the use of targeted application of small doses of herbicide precisely on weeds via a robotic detector and application system and non-herbicide methods of removal such as electrocution. However, such approaches to precision agriculture require highly accurate methods of detection and identification of weeds in agricultural fields. Champ et al. (2020) applied an instance segmentation convolutional neural network to robotically generated images of agricultural field plots to detect individual plants and then identify them as crops or weeds. Using this mask R-CNN approach, the authors were able to correctly identify individual maize and bean crop plants at average precision values of 0.85 and 0.59, respectively; identification of weeds was generally more difficult, with average precision values as high as 0.73 for Brassica nigra W. D. J. Koch but less than 0.5 for the other weeds studied. Using these detection results, up to 60% of weeds could be removed, and plant centroids were more precisely located than with alternative bounding box approaches. Refinement of the models to account for plant species, plant size, plant position, and possible crop–weed interactions could improve accuracy for greater automated weed removal with fewer possibilities of confusion with crops. Plant–insect interactions are biodiverse (Forister et al., 2015) and can be highly consequential for agricultural productivity (Sharma, 2014) and ecosystem function (Kurz et al., 2008). As a result, quantifying plant traits associated with resistance to insects is of broad interest in the natural sciences. One such type of defense against insect herbivores are trichomes, small hairs that serve as mechanical defenses that discourage insect herbivore feeding, oviposition, and movement. Like many such leaf traits, counting the trichomes required to address a given research hypothesis can be a Herculean task. Mirnezami et al. (2020) make advances toward automating quantification of trichome densities by capturing images of leaves, making the leaves transparent through a clearing process, and applying novel semi-automatic and automatic methods for counting trichomes. They then compare results from these novel methods to manual counting and determine that the most accurate novel method was semi-automatic (requiring input from the user) and was 90% accurate at estimating trichome densities on leaf surfaces. Although fully automated trichome counting has not yet been achieved, this study represents an important and detailed description of a major step forward in automated defense trait phenotyping for plants. Given the ability to automate the estimation of plant traits, a follow-on question would be whether the plant traits extracted could be reliably used for plant species identification. Furthermore, could the most informative traits for species identification be determined using machine learning approaches? Almeida et al. (2020) investigate the use of decision trees for plant identification using trait databases as well as identifying the most informative traits distinguishing between species. Using the TRY Plant Trait Database (Kattge et al., 2011, 2020) and a collection of species that spanned trees, herbs, grasses, and other taxa, they were able to correctly identify plant species with up to 90% accuracy in cross-validation. Traits such as leaf shape, fruit type, and flower color were identified as being some of the most informative. As more plant trait data are collected (including by automated methods as mentioned above), the type of approach presented in this paper can be used to guide and inform the data collection process. Acquiring high-resolution images of plant root architecture for use in downstream analysis and machine learning algorithms has proved a challenging endeavor. Most current methods use techniques that are destructive to root architecture (e.g., Trachsel et al., 2011); involve ex situ imaging under controlled conditions, often using aboveground rhizotrons (chambers with windows into the soil of plants under cultivation); incorporate intrusive methods through which cameras are inserted into the ground (Johnson et al., 2001), sometimes by soil coring (Wu et al., 2018), with the tendency to disturb soil and roots; or use non-intrusive methods such as ground-penetrating radar for trees and woody plants with roots ≥1 cm in diameter or X-ray computed tomography (Tabb et al., 2018) or magnetic imaging et al., 2017) for plants with root et al. (2020) on to improve the of these images by two state-of-the-art deep learning the et al., 2016) and the method is designed to high-resolution output from low-resolution images to not by a of these demonstrate that these models the when trained with data Supervised machine learning methods are the methods most used when applied to plant science. machine learning approaches are used to automate or reduce the and time needed to that were by These of well to supervised approaches. machine learning approaches also provide for data and of collected data. et al. (2020) and the use of an in discovery of species The authors to analysis and determine that can to and in some problems and data recognition and are in a range of applications including the automated of specimen and Thus, the automated recognition of scientific can be particularly for some Little (2020) and developed an open-source approach for scientific name recognition using artificial neural The method on an network approach that can recognize scientific across a range of (e.g., with high and precision and at of Plant are generally large and with and high amounts of over plant now published et al., 2018), many more and both and resources available for thousands of other plant species (e.g., et al., et al., 2019), data are now available for comparative analysis of plant across Although methods for identifying are currently for gene function and other attributes of plant require further Machine learning approaches are being applied to a range of problems in plant genomics, and et al. (2020) the promise of these They on supervised machine learning for gene function from information as well as data. gene function may and within a plant and have or effects on functional prediction a of at gene expression and and the authors machine learning methods at each of these problems as well as those designed to information across and biological these the authors identify current to more models and suggest possible Many machine learning methods have been developed for and as more and more methods are being developed and to such as X-ray computed ground-penetrating and and et al., 2016; et al., 2018). et al. (2020) developed a segmentation and approach for leaf using X-ray The approach by the authors a small number of image to automate segmentation over more than with of greater than 90%. The approach is on leaf manual but the possibilities of being able to apply machine learning methods to automate the analysis of a wide variety of data and image The application of machine learning to questions in plant is in its yet the promise of these methods to a broad range of problems is clear. From to of plant morphology, and and from assessing ecological interactions of plants with herbivores and their to use in new approaches machine learning have the potential to change how study plants and the questions can with fields from to ecosystem all by new machine learning will further enable new in plant However, as the contributions to this special issue have methods with high accuracy for application are under development and may require extensive in training data Thus, despite the promise and of machine learning certain problems may not be of in the model or the data needed for training sets are not available or not We that the papers presented in this collection encourage further on the emerging applications of machine learning to plant This collection of papers was by with from the