1
|
Love NLR, Bonnet P, Goëau H, Joly A, Mazer SJ. Machine Learning Undercounts Reproductive Organs on Herbarium Specimens but Accurately Derives Their Quantitative Phenological Status: A Case Study of Streptanthus tortuosus. Plants (Basel) 2021; 10:plants10112471. [PMID: 34834835 PMCID: PMC8623300 DOI: 10.3390/plants10112471] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 10/20/2021] [Revised: 11/10/2021] [Accepted: 11/12/2021] [Indexed: 06/13/2023]
Abstract
Machine learning (ML) can accelerate the extraction of phenological data from herbarium specimens; however, no studies have assessed whether ML-derived phenological data can be used reliably to evaluate ecological patterns. In this study, 709 herbarium specimens representing a widespread annual herb, Streptanthus tortuosus, were scored both manually by human observers and by a mask R-CNN object detection model to (1) evaluate the concordance between ML and manually-derived phenological data and (2) determine whether ML-derived data can be used to reliably assess phenological patterns. The ML model generally underestimated the number of reproductive structures present on each specimen; however, when these counts were used to provide a quantitative estimate of the phenological stage of plants on a given sheet (i.e., the phenological index or PI), the ML and manually-derived PI's were highly concordant. Moreover, herbarium specimen age had no effect on the estimated PI of a given sheet. Finally, including ML-derived PIs as predictor variables in phenological models produced estimates of the phenological sensitivity of this species to climate, temporal shifts in flowering time, and the rate of phenological progression that are indistinguishable from those produced by models based on data provided by human observers. This study demonstrates that phenological data extracted using machine learning can be used reliably to estimate the phenological stage of herbarium specimens and to detect phenological patterns.
Collapse
Affiliation(s)
- Natalie L. R. Love
- Department of Ecology, Evolution, and Marine Biology, University of California, Santa Barbara, CA 93106, USA;
- Biological Sciences Department, California Polytechnic State University, San Luis Obispo, CA 93407, USA
| | - Pierre Bonnet
- Botany and Modeling of Plant Architecture and Vegetation (AMAP), French Agricultural Research Centre for International Development (CIRAD), French National Centre for Scientific Research (CNRS), French National Institute for Agriculture, Food and Environment (INRAE), Research Institute for Development (IRD), University of Montpellier, 34398 Montpellier, France; (P.B.); (H.G.)
| | - Hervé Goëau
- Botany and Modeling of Plant Architecture and Vegetation (AMAP), French Agricultural Research Centre for International Development (CIRAD), French National Centre for Scientific Research (CNRS), French National Institute for Agriculture, Food and Environment (INRAE), Research Institute for Development (IRD), University of Montpellier, 34398 Montpellier, France; (P.B.); (H.G.)
| | - Alexis Joly
- ZENITH Team, Laboratory of Informatics, Robotics and Microelectronics-Joint Research Unit, Institut National de Recherche en Informatique et en Automatique (INRIA) Sophia-Antipolis, CEDEX 5, 34095 Montpellier, France;
| | - Susan J. Mazer
- Department of Ecology, Evolution, and Marine Biology, University of California, Santa Barbara, CA 93106, USA;
| |
Collapse
|
2
|
Ta QB, Kim JT. Monitoring of Corroded and Loosened Bolts in Steel Structures via Deep Learning and Hough Transforms. Sensors (Basel) 2020; 20:s20236888. [PMID: 33276512 PMCID: PMC7731320 DOI: 10.3390/s20236888] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 11/09/2020] [Revised: 11/28/2020] [Accepted: 11/29/2020] [Indexed: 11/16/2022]
Abstract
In this study, a regional convolutional neural network (RCNN)-based deep learning and Hough line transform (HLT) algorithm are applied to monitor corroded and loosened bolts in steel structures. The monitoring goals are to detect rusted bolts distinguished from non-corroded ones and also to estimate bolt-loosening angles of the identified bolts. The following approaches are performed to achieve the goals. Firstly, a RCNN-based autonomous bolt detection scheme is designed to identify corroded and clean bolts in a captured image. Secondly, a HLT-based image processing algorithm is designed to estimate rotational angles (i.e., bolt-loosening) of cropped bolts. Finally, the accuracy of the proposed framework is experimentally evaluated under various capture distances, perspective distortions, and light intensities. The lab-scale monitoring results indicate that the suggested method accurately acquires rusted bolts for images captured under perspective distortion angles less than 15° and light intensities larger than 63 lux.
Collapse
|
3
|
Davis CC, Champ J, Park DS, Breckheimer I, Lyra GM, Xie J, Joly A, Tarapore D, Ellison AM, Bonnet P. A New Method for Counting Reproductive Structures in Digitized Herbarium Specimens Using Mask R-CNN. Front Plant Sci 2020; 11:1129. [PMID: 32849691 PMCID: PMC7411132 DOI: 10.3389/fpls.2020.01129] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/12/2020] [Accepted: 07/09/2020] [Indexed: 05/29/2023]
Abstract
Phenology-the timing of life-history events-is a key trait for understanding responses of organisms to climate. The digitization and online mobilization of herbarium specimens is rapidly advancing our understanding of plant phenological response to climate and climatic change. The current practice of manually harvesting data from individual specimens, however, greatly restricts our ability to scale-up data collection. Recent investigations have demonstrated that machine-learning approaches can facilitate this effort. However, present attempts have focused largely on simplistic binary coding of reproductive phenology (e.g., presence/absence of flowers). Here, we use crowd-sourced phenological data of buds, flowers, and fruits from >3,000 specimens of six common wildflower species of the eastern United States (Anemone canadensis L., A. hepatica L., A. quinquefolia L., Trillium erectum L., T. grandiflorum (Michx.) Salisb., and T. undulatum Wild.) to train models using Mask R-CNN to segment and count phenological features. A single global model was able to automate the binary coding of each of the three reproductive stages with >87% accuracy. We also successfully estimated the relative abundance of each reproductive structure on a specimen with ≥90% accuracy. Precise counting of features was also successful, but accuracy varied with phenological stage and taxon. Specifically, counting flowers was significantly less accurate than buds or fruits likely due to their morphological variability on pressed specimens. Moreover, our Mask R-CNN model provided more reliable data than non-expert crowd-sourcers but not botanical experts, highlighting the importance of high-quality human training data. Finally, we also demonstrated the transferability of our model to automated phenophase detection and counting of the three Trillium species, which have large and conspicuously-shaped reproductive organs. These results highlight the promise of our two-phase crowd-sourcing and machine-learning pipeline to segment and count reproductive features of herbarium specimens, thus providing high-quality data with which to investigate plant responses to ongoing climatic change.
Collapse
Affiliation(s)
- Charles C. Davis
- Department of Organismic and Evolutionary Biology, Harvard University Herbaria, Harvard University, Cambridge, MA, United States
| | - Julien Champ
- LIRMM, Inria, University of Montpellier, Montpellier, France
| | - Daniel S. Park
- Department of Organismic and Evolutionary Biology, Harvard University Herbaria, Harvard University, Cambridge, MA, United States
| | - Ian Breckheimer
- Department of Organismic and Evolutionary Biology, Harvard University Herbaria, Harvard University, Cambridge, MA, United States
| | - Goia M. Lyra
- Department of Organismic and Evolutionary Biology, Harvard University Herbaria, Harvard University, Cambridge, MA, United States
- Universidade Federal da Bahia (UFBA), Salvador, Brazil
| | - Junxi Xie
- Department of Organismic and Evolutionary Biology, Harvard University Herbaria, Harvard University, Cambridge, MA, United States
| | - Alexis Joly
- LIRMM, Inria, University of Montpellier, Montpellier, France
| | - Dharmesh Tarapore
- Department of Computer Science, Boston University, Boston, MA, United States
| | - Aaron M. Ellison
- Harvard Forest, Harvard University, Petersham, MA, United States
| | - Pierre Bonnet
- CIRAD, UMR AMAP, Montpellier, France
- AMAP, Univ Montpellier, CIRAD, CNRS, INRAE, IRD, Montpellier, France
| |
Collapse
|
4
|
Goëau H, Mora‐Fallas A, Champ J, Love NLR, Mazer SJ, Mata‐Montero E, Joly A, Bonnet P. A new fine-grained method for automated visual analysis of herbarium specimens: A case study for phenological data extraction. Appl Plant Sci 2020; 8:e11368. [PMID: 32626610 PMCID: PMC7328656 DOI: 10.1002/aps3.11368] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/04/2019] [Accepted: 02/02/2020] [Indexed: 05/26/2023]
Abstract
PREMISE Herbarium specimens represent an outstanding source of material with which to study plant phenological changes in response to climate change. The fine-scale phenological annotation of such specimens is nevertheless highly time consuming and requires substantial human investment and expertise, which are difficult to rapidly mobilize. METHODS We trained and evaluated new deep learning models to automate the detection, segmentation, and classification of four reproductive structures of Streptanthus tortuosus (flower buds, flowers, immature fruits, and mature fruits). We used a training data set of 21 digitized herbarium sheets for which the position and outlines of 1036 reproductive structures were annotated manually. We adjusted the hyperparameters of a mask R-CNN (regional convolutional neural network) to this specific task and evaluated the resulting trained models for their ability to count reproductive structures and estimate their size. RESULTS The main outcome of our study is that the performance of detection and segmentation can vary significantly with: (i) the type of annotations used for training, (ii) the type of reproductive structures, and (iii) the size of the reproductive structures. In the case of Streptanthus tortuosus, the method can provide quite accurate estimates (77.9% of cases) of the number of reproductive structures, which is better estimated for flowers than for immature fruits and buds. The size estimation results are also encouraging, showing a difference of only a few millimeters between the predicted and actual sizes of buds and flowers. DISCUSSION This method has great potential for automating the analysis of reproductive structures in high-resolution images of herbarium sheets. Deeper investigations regarding the taxonomic scalability of this approach and its potential improvement will be conducted in future work.
Collapse
Affiliation(s)
- Hervé Goëau
- AMAPUniversity of MontpellierCIRADCNRSINRAEIRDMontpellierFrance
- CIRADUMR AMAPMontpellierFrance
| | - Adán Mora‐Fallas
- School of ComputingCosta Rica Institute of TechnologyCartagoCosta Rica
| | - Julien Champ
- Institut national de recherche en informatique et en automatique (INRIA) Sophia‐Antipolis, ZENITH teamLaboratory of InformaticsRobotics and Microelectronics–Joint Research Unit, 34095MontpellierCEDEX 5France
| | - Natalie L. Rossington Love
- Department of Ecology, Evolution, and Marine BiologyUniversity of California, Santa BarbaraSanta BarbaraCalifornia93106USA
| | - Susan J. Mazer
- Department of Ecology, Evolution, and Marine BiologyUniversity of California, Santa BarbaraSanta BarbaraCalifornia93106USA
| | | | - Alexis Joly
- Institut national de recherche en informatique et en automatique (INRIA) Sophia‐Antipolis, ZENITH teamLaboratory of InformaticsRobotics and Microelectronics–Joint Research Unit, 34095MontpellierCEDEX 5France
| | - Pierre Bonnet
- AMAPUniversity of MontpellierCIRADCNRSINRAEIRDMontpellierFrance
- CIRADUMR AMAPMontpellierFrance
| |
Collapse
|