1
|
Li Y, Tang Y, Liu Y, Zheng D. Semi-supervised Counting of Grape Berries in the Field Based on Density Mutual Exclusion. PLANT PHENOMICS (WASHINGTON, D.C.) 2023; 5:0115. [PMID: 38033720 PMCID: PMC10684290 DOI: 10.34133/plantphenomics.0115] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 06/26/2023] [Accepted: 10/29/2023] [Indexed: 12/02/2023]
Abstract
Automated counting of grape berries has become one of the most important tasks in grape yield prediction. However, dense distribution of berries and the severe occlusion between berries bring great challenges to counting algorithm based on deep learning. The collection of data required for model training is also a tedious and expensive work. To address these issues and cost-effectively count grape berries, a semi-supervised counting of grape berries in the field based on density mutual exclusion (CDMENet) is proposed. The algorithm uses VGG16 as the backbone to extract image features. Auxiliary tasks based on density mutual exclusion are introduced. The tasks exploit the spatial distribution pattern of grape berries in density levels to make full use of unlabeled data. In addition, a density difference loss is designed. The feature representation is enhanced by amplifying the difference of features between different density levels. The experimental results on the field grape berry dataset show that CDMENet achieves less counting errors. Compared with the state of the arts, coefficient of determination (R2) is improved by 6.10%, and mean absolute error and root mean square error are reduced by 49.36% and 54.08%, respectively. The code is available at https://github.com/youth-tang/CDMENet-main.
Collapse
Affiliation(s)
- Yanan Li
- School of Computer Science and Engineering, School of Artificial Intelligence,
Wuhan Institute of Technology, Wuhan 430205, China
- Hubei Key Laboratory of Intelligent Robot,
Wuhan Institute of Technology, Wuhan 430073, China
| | - Yuling Tang
- School of Computer Science and Engineering, School of Artificial Intelligence,
Wuhan Institute of Technology, Wuhan 430205, China
- Hubei Key Laboratory of Intelligent Robot,
Wuhan Institute of Technology, Wuhan 430073, China
| | - Yifei Liu
- School of Computer Science and Engineering, School of Artificial Intelligence,
Wuhan Institute of Technology, Wuhan 430205, China
- Hubei Key Laboratory of Intelligent Robot,
Wuhan Institute of Technology, Wuhan 430073, China
| | - Dingrun Zheng
- School of Computer Science and Engineering, School of Artificial Intelligence,
Wuhan Institute of Technology, Wuhan 430205, China
- Hubei Key Laboratory of Intelligent Robot,
Wuhan Institute of Technology, Wuhan 430073, China
| |
Collapse
|
2
|
Du W, Liu P. Instance Segmentation and Berry Counting of Table Grape before Thinning Based on AS-SwinT. PLANT PHENOMICS (WASHINGTON, D.C.) 2023; 5:0085. [PMID: 37654806 PMCID: PMC10465307 DOI: 10.34133/plantphenomics.0085] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/16/2023] [Accepted: 08/08/2023] [Indexed: 09/02/2023]
Abstract
Berry thinning is one of the most important tasks in the management of high-quality table grapes. Farmers often thin the berries per cluster to a standard number by counting. With an aging population, it is hard to find adequate skilled farmers to work during thinning season. It is urgent to design an intelligent berry-thinning machine to avoid exhaustive repetitive labor. A machine vision system that can determine the number of berries removed and locate the berries removed is a challenge for the thinning machine. A method for instance segmentation of berries and berry counting in a single bunch is proposed based on AS-SwinT. In AS-SwinT, Swin Transformer is performed as the backbone to extract the rich characteristics of grape berries. An adaptive feature fusion is introduced to the neck network to sufficiently preserve the underlying features and enhance the detection of small berries. The size of berries in the dataset is statistically analyzed to optimize the anchor scale, and Soft-NMS is used to filter the candidate frames to reduce the missed detection of densely shaded berries. Finally, the proposed method could achieve 65.7 APbox, 95.0 A P 0.5 box , 57 A P s box , 62.8 APmask, 94.3 A P 0.5 mask , 48 A P s mask , which is markedly superior to Mask R-CNN, Mask Scoring R-CNN, and Cascade Mask R-CNN. Linear regressions between predicted numbers and actual numbers are also developed to verify the precision of the proposed model. RMSE and R2 values are 7.13 and 0.95, respectively, which are substantially higher than other models, showing the advantage of the AS-SwinT model in berry counting estimation.
Collapse
Affiliation(s)
- Wensheng Du
- Shandong Agricultural Equipment Intelligent Engineering Laboratory; Shandong Provincial Key Laboratory of Horticultural, Machinery and Equipment; College of Mechanical and Electronic Engineering,
Shandong Agricultural University, Tai’an 271000, China
- School of Construction Machinery,
Shandong Jiaotong University, Jinan 250357, China
| | - Ping Liu
- Shandong Agricultural Equipment Intelligent Engineering Laboratory; Shandong Provincial Key Laboratory of Horticultural, Machinery and Equipment; College of Mechanical and Electronic Engineering,
Shandong Agricultural University, Tai’an 271000, China
| |
Collapse
|
3
|
Rançon F, Keresztes B, Deshayes A, Tardif M, Abdelghafour F, Fontaine G, Da Costa JP, Germain C. Designing a Proximal Sensing Camera Acquisition System for Vineyard Applications: Results and Feedback on 8 Years of Experiments. SENSORS (BASEL, SWITZERLAND) 2023; 23:847. [PMID: 36679645 PMCID: PMC9865571 DOI: 10.3390/s23020847] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 11/30/2022] [Revised: 12/30/2022] [Accepted: 01/05/2023] [Indexed: 06/17/2023]
Abstract
The potential of image proximal sensing for agricultural applications has been a prolific scientific subject in the recent literature. Its main appeal lies in the sensing of precise information about plant status, which is either harder or impossible to extract from lower-resolution downward-looking image sensors such as satellite or drone imagery. Yet, many theoretical and practical problems arise when dealing with proximal sensing, especially on perennial crops such as vineyards. Indeed, vineyards exhibit challenging physical obstacles and many degrees of variability in their layout. In this paper, we present the design of a mobile camera suited to vineyards and harsh experimental conditions, as well as the results and assessments of 8 years' worth of studies using that camera. These projects ranged from in-field yield estimation (berry counting) to disease detection, providing new insights on typical viticulture problems that could also be generalized to orchard crops. Different recommendations are then provided using small case studies, such as the difficulties related to framing plots with different structures or the mounting of the sensor on a moving vehicle. While results stress the obvious importance and strong benefits of a thorough experimental design, they also indicate some inescapable pitfalls, illustrating the need for more robust image analysis algorithms and better databases. We believe sharing that experience with the scientific community can only benefit the future development of these innovative approaches.
Collapse
Affiliation(s)
- Florian Rançon
- IMS Laboratory, CNRS UMR 5218, University of Bordeaux, Talence Campus, F-33400 Talence, France
- Bordeaux Sciences Agro, F-33175 Gradignan, France
| | - Barna Keresztes
- IMS Laboratory, CNRS UMR 5218, University of Bordeaux, Talence Campus, F-33400 Talence, France
| | - Aymeric Deshayes
- IMS Laboratory, CNRS UMR 5218, University of Bordeaux, Talence Campus, F-33400 Talence, France
| | - Malo Tardif
- IMS Laboratory, CNRS UMR 5218, University of Bordeaux, Talence Campus, F-33400 Talence, France
| | - Florent Abdelghafour
- INRAE, Institut Agro, ITAP, University of Montpellier, F-34196 Montpellier, France
| | - Gael Fontaine
- IMS Laboratory, CNRS UMR 5218, University of Bordeaux, Talence Campus, F-33400 Talence, France
| | - Jean-Pierre Da Costa
- IMS Laboratory, CNRS UMR 5218, University of Bordeaux, Talence Campus, F-33400 Talence, France
- Bordeaux Sciences Agro, F-33175 Gradignan, France
| | - Christian Germain
- IMS Laboratory, CNRS UMR 5218, University of Bordeaux, Talence Campus, F-33400 Talence, France
- Bordeaux Sciences Agro, F-33175 Gradignan, France
| |
Collapse
|
4
|
Kierdorf J, Weber I, Kicherer A, Zabawa L, Drees L, Roscher R. Behind the Leaves: Estimation of Occluded Grapevine Berries With Conditional Generative Adversarial Networks. Front Artif Intell 2022; 5:830026. [PMID: 35402903 PMCID: PMC8990779 DOI: 10.3389/frai.2022.830026] [Citation(s) in RCA: 6] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/06/2021] [Accepted: 02/28/2022] [Indexed: 11/30/2022] Open
Abstract
The need for accurate yield estimates for viticulture is becoming more important due to increasing competition in the wine market worldwide. One of the most promising methods to estimate the harvest is berry counting, as it can be approached non-destructively, and its process can be automated. In this article, we present a method that addresses the challenge of occluded berries with leaves to obtain a more accurate estimate of the number of berries that will enable a better estimate of the harvest. We use generative adversarial networks, a deep learning-based approach that generates a highly probable scenario behind the leaves exploiting learned patterns from images with non-occluded berries. Our experiments show that the estimate of the number of berries after applying our method is closer to the manually counted reference. In contrast to applying a factor to the berry count, our approach better adapts to local conditions by directly involving the appearance of the visible berries. Furthermore, we show that our approach can identify which areas in the image should be changed by adding new berries without explicitly requiring information about hidden areas.
Collapse
Affiliation(s)
- Jana Kierdorf
- Remote Sensing Group, Institute of Geodesy and Geoinformation, University of Bonn, Bonn, Germany
| | - Immanuel Weber
- Application Center for Machine Learning and Sensor Technology, University of Applied Sciences Koblenz, Koblenz, Germany
| | - Anna Kicherer
- Julius Kühn-Institut (JKI), Federal Research Centre for Cultivated Plants, Institute for Grapevine Breeding Geilweilerhof, Siebeldingen, Germany
| | - Laura Zabawa
- Geodesy Group, Institute of Geodesy and Geoinformation, University of Bonn, Bonn, Germany
| | - Lukas Drees
- Remote Sensing Group, Institute of Geodesy and Geoinformation, University of Bonn, Bonn, Germany
| | - Ribana Roscher
- Remote Sensing Group, Institute of Geodesy and Geoinformation, University of Bonn, Bonn, Germany
| |
Collapse
|
5
|
Grape Berry Detection and Size Measurement Based on Edge Image Processing and Geometric Morphology. MACHINES 2021. [DOI: 10.3390/machines9100233] [Citation(s) in RCA: 7] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/17/2022]
Abstract
Counting grape berries and measuring their size can provide accurate data for robot picking behavior decision-making, yield estimation, and quality evaluation. When grapes are picked, there is a strong uncertainty in the external environment and the shape of the grapes. Counting grape berries and measuring berry size are challenging tasks. Computer vision has made a huge breakthrough in this field. Although the detection method of grape berries based on 3D point cloud information relies on scanning equipment to estimate the number and yield of grape berries, the detection method is difficult to generalize. Grape berry detection based on 2D images is an effective method to solve this problem. However, it is difficult for traditional algorithms to accurately measure the berry size and other parameters, and there is still the problem of the low robustness of berry counting. In response to the above problems, we propose a grape berry detection method based on edge image processing and geometric morphology. The edge contour search and the corner detection algorithm are introduced to detect the concave point position of the berry edge contour extracted by the Canny algorithm to obtain the best contour segment. To correctly obtain the edge contour information of each berry and reduce the error grouping of contour segments, this paper proposes an algorithm for combining contour segments based on clustering search strategy and rotation direction determination, which realizes the correct reorganization of the segmented contour segments, to achieve an accurate calculation of the number of berries and an accurate measurement of their size. The experimental results prove that our proposed method has an average accuracy of 87.76% for the detection of the concave points of the edge contours of different types of grapes, which can achieve a good edge contour segmentation. The average accuracy of the detection of the number of grapes berries in this paper is 91.42%, which is 4.75% higher than that of the Hough transform. The average error between the measured berry size and the actual berry size is 2.30 mm, and the maximum error is 5.62 mm, which is within a reasonable range. The results prove that the method proposed in this paper is robust enough to detect different types of grape berries.
Collapse
|
6
|
Ghiani L, Sassu A, Palumbo F, Mercenaro L, Gambella F. In-Field Automatic Detection of Grape Bunches under a Totally Uncontrolled Environment. SENSORS (BASEL, SWITZERLAND) 2021; 21:3908. [PMID: 34198844 PMCID: PMC8201373 DOI: 10.3390/s21113908] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 05/07/2021] [Revised: 05/31/2021] [Accepted: 06/02/2021] [Indexed: 12/02/2022]
Abstract
An early estimation of the exact number of fruits, flowers, and trees helps farmers to make better decisions on cultivation practices, plant disease prevention, and the size of harvest labor force. The current practice of yield estimation based on manual counting of fruits or flowers by workers is a time consuming and expensive process and it is not feasible for large fields. Automatic yield estimation based on robotic agriculture provides a viable solution in this regard. In a typical image classification process, the task is not only to specify the presence or absence of a given object on a specific location, while counting how many objects are present in the scene. The success of these tasks largely depends on the availability of a large amount of training samples. This paper presents a detector of bunches of one fruit, grape, based on a deep convolutional neural network trained to detect vine bunches directly on the field. Experimental results show a 91% mean Average Precision.
Collapse
Affiliation(s)
- Luca Ghiani
- Department of Agricultural Sciences, University of Sassari, Viale Italia 39 a, 07100 Sassari, Italy; (L.G.); (A.S.); (L.M.)
| | - Alberto Sassu
- Department of Agricultural Sciences, University of Sassari, Viale Italia 39 a, 07100 Sassari, Italy; (L.G.); (A.S.); (L.M.)
| | - Francesca Palumbo
- Intelligent System DEsign and Applications (IDEA) Group, Department of Chemistry and Pharmacy, University of Sassari, Via Muroni 23/A, 07100 Sassari, Italy;
| | - Luca Mercenaro
- Department of Agricultural Sciences, University of Sassari, Viale Italia 39 a, 07100 Sassari, Italy; (L.G.); (A.S.); (L.M.)
| | - Filippo Gambella
- Department of Agricultural Sciences, University of Sassari, Viale Italia 39 a, 07100 Sassari, Italy; (L.G.); (A.S.); (L.M.)
| |
Collapse
|