Latent Space Phenotyping: Automatic Image-Based Phenotyping for Treatment Studies

Modern phenomics to empower holistic crop science, agronomy, and breeding research

Crop phenomics enables the collection of diverse plant traits for a large number of samples along different time scales, representing a greater data collection throughput compared with traditional measurements. Most modern crop phenomics use different sensors to collect reflective, emitted, and fluorescence signals, etc., from plant organs at different spatial and temporal resolutions. Such multi-modal, high-dimensional data not only accelerates basic research on crop physiology, genetics, and whole plant systems modeling, but also supports the optimization of field agronomic practices, internal environments of plant factories, and ultimately crop breeding. Major challenges and opportunities facing the current crop phenomics research community include developing community consensus or standards for data collection, management, sharing, and processing, developing capabilities to measure physiological parameters, and enabling farmers and breeders to effectively use phenomics in the field to directly support agricultural production.

Yield prediction in a peanut breeding program using remote sensing data and machine learning algorithms

Peanut is a critical food crop worldwide, and the development of high-throughput phenotyping techniques is essential for enhancing the crop's genetic gain rate. Given the obvious challenges of directly estimating peanut yields through remote sensing, an approach that utilizes above-ground phenotypes to estimate underground yield is necessary. To that end, this study leveraged unmanned aerial vehicles (UAVs) for high-throughput phenotyping of surface traits in peanut. Using a diverse set of peanut germplasm planted in 2021 and 2022, UAV flight missions were repeatedly conducted to capture image data that were used to construct high-resolution multitemporal sigmoidal growth curves based on apparent characteristics, such as canopy cover and canopy height. Latent phenotypes extracted from these growth curves and their first derivatives informed the development of advanced machine learning models, specifically random forest and eXtreme Gradient Boosting (XGBoost), to estimate yield in the peanut plots. The random forest model exhibited exceptional predictive accuracy (R2 = 0.93), while XGBoost was also reasonably effective (R2 = 0.88). When using confusion matrices to evaluate the classification abilities of each model, the two models proved valuable in a breeding pipeline, particularly for filtering out underperforming genotypes. In addition, the random forest model excelled in identifying top-performing material while minimizing Type I and Type II errors. Overall, these findings underscore the potential of machine learning models, especially random forests and XGBoost, in predicting peanut yield and improving the efficiency of peanut breeding programs.

Data-driven approaches to improve water-use efficiency and drought resistance in crop plants

With the increasing population, there lies a pressing demand for food, feed and fibre, while the changing climatic conditions pose severe challenges for agricultural production worldwide. Water is the lifeline for crop production; thus, enhancing crop water-use efficiency (WUE) and improving drought resistance in crop varieties are crucial for overcoming these challenges. Genetically-driven improvements in yield, WUE and drought tolerance traits can buffer the worst effects of climate change on crop production in dry areas. While traditional crop breeding approaches have delivered impressive results in increasing yield, the methods remain time-consuming and are often limited by the existing allelic variation present in the germplasm. Significant advances in breeding and high-throughput omics technologies in parallel with smart agriculture practices have created avenues to dramatically speed up the process of trait improvement by leveraging the vast volumes of genomic and phenotypic data. For example, individual genome and pan-genome assemblies, along with transcriptomic, metabolomic and proteomic data from germplasm collections, characterised at phenotypic levels, could be utilised to identify marker-trait associations and superior haplotypes for crop genetic improvement. In addition, these omics approaches enable the identification of genes involved in pathways leading to the expression of a trait, thereby providing an understanding of the genetic, physiological and biochemical basis of trait variation. These data-driven gene discoveries and validation approaches are essential for crop improvement pipelines, including genomic breeding, speed breeding and gene editing. Herein, we provide an overview of prospects presented using big data-driven approaches (including artificial intelligence and machine learning) to harness new genetic gains for breeding programs and develop drought-tolerant crop varieties with favourable WUE and high-yield potential traits.

Explainable deep learning in plant phenotyping

The increasing human population and variable weather conditions, due to climate change, pose a threat to the world's food security. To improve global food security, we need to provide breeders with tools to develop crop cultivars that are more resilient to extreme weather conditions and provide growers with tools to more effectively manage biotic and abiotic stresses in their crops. Plant phenotyping, the measurement of a plant's structural and functional characteristics, has the potential to inform, improve and accelerate both breeders' selections and growers' management decisions. To improve the speed, reliability and scale of plant phenotyping procedures, many researchers have adopted deep learning methods to estimate phenotypic information from images of plants and crops. Despite the successful results of these image-based phenotyping studies, the representations learned by deep learning models remain difficult to interpret, understand, and explain. For this reason, deep learning models are still considered to be black boxes. Explainable AI (XAI) is a promising approach for opening the deep learning model's black box and providing plant scientists with image-based phenotypic information that is interpretable and trustworthy. Although various fields of study have adopted XAI to advance their understanding of deep learning models, it has yet to be well-studied in the context of plant phenotyping research. In this review article, we reviewed existing XAI studies in plant shoot phenotyping, as well as related domains, to help plant researchers understand the benefits of XAI and make it easier for them to integrate XAI into their future studies. An elucidation of the representations within a deep learning model can help researchers explain the model's decisions, relate the features detected by the model to the underlying plant physiology, and enhance the trustworthiness of image-based phenotypic information used in food production systems.

"Canopy fingerprints" for characterizing three-dimensional point cloud data of soybean canopies

Advances in imaging hardware allow high throughput capture of the detailed three-dimensional (3D) structure of plant canopies. The point cloud data is typically post-processed to extract coarse-scale geometric features (like volume, surface area, height, etc.) for downstream analysis. We extend feature extraction from 3D point cloud data to various additional features, which we denote as 'canopy fingerprints'. This is motivated by the successful application of the fingerprint concept for molecular fingerprints in chemistry applications and acoustic fingerprints in sound engineering applications. We developed an end-to-end pipeline to generate canopy fingerprints of a three-dimensional point cloud of soybean [Glycine max (L.) Merr.] canopies grown in hill plots captured by a terrestrial laser scanner (TLS). The pipeline includes noise removal, registration, and plot extraction, followed by the canopy fingerprint generation. The canopy fingerprints are generated by splitting the data into multiple sub-canopy scale components and extracting sub-canopy scale geometric features. The generated canopy fingerprints are interpretable and can assist in identifying patterns in a database of canopies, querying similar canopies, or identifying canopies with a certain shape. The framework can be extended to other modalities (for instance, hyperspectral point clouds) and tuned to find the most informative fingerprint representation for downstream tasks. These canopy fingerprints can aid in the utilization of canopy traits at previously unutilized scales, and therefore have applications in plant breeding and resilient crop production.

Focusing the GWAS Lens on days to flower using latent variable phenotypes derived from global multienvironment trials

Adaptation constraints within crop species have resulted in limited genetic diversity in some breeding programs and areas where new crops have been introduced, for example, for lentil (Lens culinaris Medik.) in North America. An improved understanding of the underlying genetics involved in phenology-related traits is valuable knowledge to aid breeders in overcoming limitations associated with unadapted germplasm and expanding their genetic diversity by introducing new, exotic material. We used a large, 18 site-year, multienvironment dataset phenotyped for phenology-related traits across nine locations and over 3 yr along with accompanying latent variable phenotypes derived from a photothermal model and principal component analysis (PCA) of days from sowing to flower (DTF) data for a lentil diversity panel (324 accessions), which has also been genotyped with an exome capture array. Genome-wide association studies (GWAS) on DTF across multiple environments helped confirm associations with known flowering-time genes and identify new quantitative trait loci (QTL), which may contain previously unknown flowering time genes. Additionally, the use of latent variable phenotypes, which can incorporate environmental data such as temperature and photoperiod as both GWAS traits and as covariates, strengthened associations, revealed additional hidden associations, and alluded to potential roles of the associated QTL. Our approach can be replicated with other crop species, and the results from our GWAS serve as a resource for further exploration into the complex nature of phenology-related traits across the major growing environments for cultivated lentil.

Classification of high-throughput phenotyping data for differentiation among nutrient deficiency in common bean

The development of automated, image-based, high-throughput plant phenotyping enabled the simultaneous measurement of many plant traits. Big and complex phenotypic datasets require advanced statistical methods which enable the extraction of the most valuable traits when combined with other measurements, interpretation, and understanding of their (eco)physiological background. Nutrient deficiency in plants causes specific symptoms that can be easily detected by multispectral imaging, 3D scanning, and chlorophyll fluorescence measurements. Screening of numerous image-based phenotypic traits of common bean plants grown in nutrient-deficient solutions was conducted to optimize phenotyping and select the most valuable phenotypic traits related to the specific nutrient deficit. Discriminant analysis was used to compare the efficiency of groups of traits obtained by high-throughput phenotyping techniques (chlorophyll fluorescence, multispectral traits, and morphological traits) in discrimination between nutrients [nitrogen (N), phosphorus (P), potassium (K), magnesium (Mg), and iron (Fe)] at early and prolonged deficiency. Furthermore, a recursive partitioning analysis was used to select variables within each group of traits that show the highest accuracy for assigning plants to the respective nutrient deficit treatment. Using the entire set of measured traits, the highest classification success by discriminant function was achieved using multispectral traits. In the subsequent measurements, chlorophyll fluorescence and multispectral traits achieved comparably high classification success. Recursive partitioning analysis was able to intrinsically identify variables within each group of traits and their threshold values that best separate the observations from different nutrient deficiency groups. Again, the highest success in assigning plants into their respective groups was achieved based on selected multispectral traits. Selected chlorophyll fluorescence traits also showed high accuracy for assigning plants into control, Fe, Mg, and P deficit but could not correctly assign K and N deficit plants. This study has shown the usefulness of combining high-throughput phenotyping techniques with advanced data analysis to determine and differentiate nutrient deficiency stress.

Comparing Deep Learning Approaches for Understanding Genotype × Phenotype Interactions in Biomass Sorghum

We explore the use of deep convolutional neural networks (CNNs) trained on overhead imagery of biomass sorghum to ascertain the relationship between single nucleotide polymorphisms (SNPs), or groups of related SNPs, and the phenotypes they control. We consider both CNNs trained explicitly on the classification task of predicting whether an image shows a plant with a reference or alternate version of various SNPs as well as CNNs trained to create data-driven features based on learning features so that images from the same plot are more similar than images from different plots, and then using the features this network learns for genetic marker classification. We characterize how efficient both approaches are at predicting the presence or absence of a genetic markers, and visualize what parts of the images are most important for those predictions. We find that the data-driven approaches give somewhat higher prediction performance, but have visualizations that are harder to interpret; and we give suggestions of potential future machine learning research and discuss the possibilities of using this approach to uncover unknown genotype × phenotype relationships.

Machine Learning for Image Analysis: Leaf Disease Segmentation

Plant phenomics field has seen a great increase in scalability in the last decade mainly due to technological advances in remote sensors and phenotyping platforms. These are capable of screening thousands of plants many times throughout the day, generating massive amounts of data, which require an automated analysis to extract meaningful information. Deep learning is a branch of machine learning that has revolutionized many fields of research. Deep learning models are able to extract autonomously the underlying features within the dataset, providing a multi-level representation of the data. Our intention is to show the feasibility and effectiveness of using deep learning and low-cost technology for automated phenotyping. In this methods chapter, we describe how to train a deep neural network to segment leaf images and extract the pixels related to the disease.

High-throughput field crop phenotyping: current status and challenges

In contrast to the rapid advances made in plant genotyping, plant phenotyping is considered a bottleneck in plant science. This has promoted high-throughput plant phenotyping (HTP) studies, resulting in an exponential increase in phenotyping-related publications. The development of HTP was originally intended for use as indoor HTP technologies for model plant species under controlled environments. However, this subsequently shifted to HTP for use in crops in fields. Although HTP in fields is much more difficult to conduct due to unstable environmental conditions compared to HTP in controlled environments, recent advances in HTP technology have allowed these difficulties to be overcome, allowing for rapid, efficient, non-destructive, non-invasive, quantitative, repeatable, and objective phenotyping. Recent HTP developments have been accelerated by the advances in data analysis, sensors, and robot technologies, including machine learning, image analysis, three dimensional (3D) reconstruction, image sensors, laser sensors, environmental sensors, and drones, along with high-speed computational resources. This article provides an overview of recent HTP technologies, focusing mainly on canopy-based phenotypes of major crops, such as canopy height, canopy coverage, canopy biomass, and canopy stressed appearance, in addition to crop organ detection and counting in the fields. Current topics in field HTP are also presented, followed by a discussion on the low rates of adoption of HTP in practical breeding programs.

Multi-dimensional leaf phenotypes reflect root system genotype in grafted grapevine over the growing season

BACKGROUND

Modern biological approaches generate volumes of multi-dimensional data, offering unprecedented opportunities to address biological questions previously beyond reach owing to small or subtle effects. A fundamental question in plant biology is the extent to which below-ground activity in the root system influences above-ground phenotypes expressed in the shoot system. Grafting, an ancient horticultural practice that fuses the root system of one individual (the rootstock) with the shoot system of a second, genetically distinct individual (the scion), is a powerful experimental system to understand below-ground effects on above-ground phenotypes. Previous studies on grafted grapevines have detected rootstock influence on scion phenotypes including physiology and berry chemistry. However, the extent of the rootstock's influence on leaves, the photosynthetic engines of the vine, and how those effects change over the course of a growing season, are still largely unknown.

RESULTS

Here, we investigate associations between rootstock genotype and shoot system phenotypes using 5 multi-dimensional leaf phenotyping modalities measured in a common grafted scion: ionomics, metabolomics, transcriptomics, morphometrics, and physiology. Rootstock influence is ubiquitous but subtle across modalities, with the strongest signature of rootstock observed in the leaf ionome. Moreover, we find that the extent of rootstock influence on scion phenotypes and patterns of phenomic covariation are highly dynamic across the season.

CONCLUSIONS

These findings substantially expand previously identified patterns to demonstrate that rootstock influence on scion phenotypes is complex and dynamic and underscore that broad understanding necessitates volumes of multi-dimensional data previously unmet.

R. Patiranage D, Johansen K, Schmöckel SM, Bertero D, Oakey H, Colque-Little C, Afzal I, Raubach S, Miller N, Streich J, Amby DB, Emrani N, Warmington M, Mousa MAA, Wu D, Jacobson D, Andreasen C, Jung C, Murphy K, Bazile D, Tester M. Quinoa Phenotyping Methodologies: An International Consensus

Quinoa is a crop originating in the Andes but grown more widely and with the genetic potential for significant further expansion. Due to the phenotypic plasticity of quinoa, varieties need to be assessed across years and multiple locations. To improve comparability among field trials across the globe and to facilitate collaborations, components of the trials need to be kept consistent, including the type and methods of data collected. Here, an internationally open-access framework for phenotyping a wide range of quinoa features is proposed to facilitate the systematic agronomic, physiological and genetic characterization of quinoa for crop adaptation and improvement. Mature plant phenotyping is a central aspect of this paper, including detailed descriptions and the provision of phenotyping cards to facilitate consistency in data collection. High-throughput methods for multi-temporal phenotyping based on remote sensing technologies are described. Tools for higher-throughput post-harvest phenotyping of seeds are presented. A guideline for approaching quinoa field trials including the collection of environmental data and designing layouts with statistical robustness is suggested. To move towards developing resources for quinoa in line with major cereal crops, a database was created. The Quinoa Germinate Platform will serve as a central repository of data for quinoa researchers globally.

Hyperspectral reflectance-based phenotyping for quantitative genetics in crops: Progress and challenges

Many biochemical and physiological properties of plants that are of interest to breeders and geneticists have extremely low throughput and/or can only be measured destructively. This has limited the use of information on natural variation in nutrient and metabolite abundance, as well as photosynthetic capacity in quantitative genetic contexts where it is necessary to collect data from hundreds or thousands of plants. A number of recent studies have demonstrated the potential to estimate many of these traits from hyperspectral reflectance data, primarily in ecophysiological contexts. Here, we summarize recent advances in the use of hyperspectral reflectance data for plant phenotyping, and discuss both the potential benefits and remaining challenges to its application in plant genetics contexts. The performances of previously published models in estimating six traits from hyperspectral reflectance data in maize were evaluated on new sample datasets, and the resulting predicted trait values shown to be heritable (e.g., explained by genetic factors) were estimated. The adoption of hyperspectral reflectance-based phenotyping beyond its current uses may accelerate the study of genes controlling natural variation in biochemical and physiological traits.

Robotic Technologies for High-Throughput Plant Phenotyping: Contemporary Reviews and Future Perspectives

Phenotyping plants is an essential component of any effort to develop new crop varieties. As plant breeders seek to increase crop productivity and produce more food for the future, the amount of phenotype information they require will also increase. Traditional plant phenotyping relying on manual measurement is laborious, time-consuming, error-prone, and costly. Plant phenotyping robots have emerged as a high-throughput technology to measure morphological, chemical and physiological properties of large number of plants. Several robotic systems have been developed to fulfill different phenotyping missions. In particular, robotic phenotyping has the potential to enable efficient monitoring of changes in plant traits over time in both controlled environments and in the field. The operation of these robots can be challenging as a result of the dynamic nature of plants and the agricultural environments. Here we discuss developments in phenotyping robots, and the challenges which have been overcome and others which remain outstanding. In addition, some perspective applications of the phenotyping robots are also presented. We optimistically anticipate that autonomous and robotic systems will make great leaps forward in the next 10 years to advance the plant phenotyping research into a new era.

Deep learning-based high-throughput phenotyping can drive future discoveries in plant reproductive biology

Advances in deep learning are providing a powerful set of image analysis tools that are readily accessible for high-throughput phenotyping applications in plant reproductive biology. High-throughput phenotyping systems are becoming critical for answering biological questions on a large scale. These systems have historically relied on traditional computer vision techniques. However, neural networks and specifically deep learning are rapidly becoming more powerful and easier to implement. Here, we examine how deep learning can drive phenotyping systems and be used to answer fundamental questions in reproductive biology. We describe previous applications of deep learning in the plant sciences, provide general recommendations for applying these methods to the study of plant reproduction, and present a case study in maize ear phenotyping. Finally, we highlight several examples where deep learning has enabled research that was previously out of reach and discuss the future outlook of these methods.

Machine learning in plant science and plant breeding

Technological developments have revolutionized measurements on plant genotypes and phenotypes, leading to routine production of large, complex data sets. This has led to increased efforts to extract meaning from these measurements and to integrate various data sets. Concurrently, machine learning has rapidly evolved and is now widely applied in science in general and in plant genotyping and phenotyping in particular. Here, we review the application of machine learning in the context of plant science and plant breeding. We focus on analyses at different phenotype levels, from biochemical to yield, and in connecting genotypes to these. In this way, we illustrate how machine learning offers a suite of methods that enable researchers to find meaningful patterns in relevant plant data.

Multi-dimensional machine learning approaches for fruit shape phenotyping in strawberry

BACKGROUND

Shape is a critical element of the visual appeal of strawberry fruit and is influenced by both genetic and non-genetic determinants. Current fruit phenotyping approaches for external characteristics in strawberry often rely on the human eye to make categorical assessments. However, fruit shape is an inherently multi-dimensional, continuously variable trait and not adequately described by a single categorical or quantitative feature. Morphometric approaches enable the study of complex, multi-dimensional forms but are often abstract and difficult to interpret. In this study, we developed a mathematical approach for transforming fruit shape classifications from digital images onto an ordinal scale called the Principal Progression of k Clusters (PPKC). We use these human-recognizable shape categories to select quantitative features extracted from multiple morphometric analyses that are best fit for genetic dissection and analysis.

RESULTS

We transformed images of strawberry fruit into human-recognizable categories using unsupervised machine learning, discovered 4 principal shape categories, and inferred progression using PPKC. We extracted 68 quantitative features from digital images of strawberries using a suite of morphometric analyses and multivariate statistical approaches. These analyses defined informative feature sets that effectively captured quantitative differences between shape classes. Classification accuracy ranged from 68% to 99% for the newly created phenotypic variables for describing a shape.

CONCLUSIONS

Our results demonstrated that strawberry fruit shapes could be robustly quantified, accurately classified, and empirically ordered using image analyses, machine learning, and PPKC. We generated a dictionary of quantitative traits for studying and predicting shape classes and identifying genetic factors underlying phenotypic variability for fruit shape in strawberry. The methods and approaches that we applied in strawberry should apply to other fruits, vegetables, and specialty crops.

Convolutional Neural Networks for Image-Based High-Throughput Plant Phenotyping: A Review

Plant phenotyping has been recognized as a bottleneck for improving the efficiency of breeding programs, understanding plant-environment interactions, and managing agricultural systems. In the past five years, imaging approaches have shown great potential for high-throughput plant phenotyping, resulting in more attention paid to imaging-based plant phenotyping. With this increased amount of image data, it has become urgent to develop robust analytical tools that can extract phenotypic traits accurately and rapidly. The goal of this review is to provide a comprehensive overview of the latest studies using deep convolutional neural networks (CNNs) in plant phenotyping applications. We specifically review the use of various CNN architecture for plant stress evaluation, plant development, and postharvest quality assessment. We systematically organize the studies based on technical developments resulting from imaging classification, object detection, and image segmentation, thereby identifying state-of-the-art solutions for certain phenotyping applications. Finally, we provide several directions for future research in the use of CNN architecture for plant phenotyping purposes.