Progress report for GNE22-293
Project Information
Crop load management is the single most important yet difficult management strategy that determines the annual profitability of apple orchards. Pollination and thinning are the two aspects that affect greatly on effective crop load management. Environmental conditions interfere with the natural pollination process thus causing huge uncertainty in achieving optimal pollination. Later after the pollination process, to apply appropriate amount of thinning remains a challenge: If thinning is inadequate and too many fruits remain on the tree, fruit size will be small, whilefruit quality will be poor. Over thinning also carries economic perils since yield and crop value the year of application will be reduced and fruit size will be excessively large with reduced fruit quality due to reduced flesh firmness, reduced color and a much-reduced post-harvest life. Thus, in this study, we propose to develop a robotic apple crop load management system to achieve high yield and quality of fruit crops resulting in a substantial economic benefit to the tree fruit industry. The proposed robotic system consists of two major components; i) a well-developed machine vision system that can identify the location of apple flower cluster and king flowers, generate flower density map, and communicate with robotic manipulator automatically; and ii) a spraying system that is mounted on a robotic manipulator to reach target position. It is expected that our prototype and field validation will provide sufficient information for companies to develop and commercialize a robotic crop load management system for the growers.
The primary goal of this proposed project is to develop an autonomous apple crop load management system that is capable of localization of flower clusters and king blossom on apple canopies to perform precision targeted spraying. The project will be focused on the following objectives:
- Establish a flower cluster image dataset throughout the flowering growing stage
The purpose of this objective is to collect adequate dataset of apple flowers for the lateral training and testing for the Deep Learning network. Images will be taken on the Gala and Honeycrisp apple trees during different periods of flowering stages. Two ZED 2 stereo cameras (Stereo Labs) will be used to cover whole tree canopy. The manually-counted number of king blossom will be served as the ground truth to be compared with the results of vision system.
- Flower Cluster Localization and King Blossom Identification
In this objective, a deep learning-based machine vision system will be developed for individual apple flower segmentation. With some further image processing process, separate flower clusters will be localized. King blossom will be identified within each flower cluster.
- Develop a path sequencing algorithm to support the decision making for chemical suspension
An optimized trajectory leads to higher efficiency for the robotic system. The most well-known problem of searching for the optimal sequence is the Traveling Salesman Problem (TSP). The goal of the TSP is to find a minimal-cost cyclic tour through a set of points such that every point is visited once.
- Integrate the vision system and robotic manipulator to perform lab evaluation on precision apple flower pollination
A UR-5e robotic manipulator will be used to integrate general spraying nozzle along with the vision system developed in previous objective. The manipulator and sensors will be mounted at different locations and orientations in a reconfigurable frame to make it suitable to fruiting-wall canopy architectures.
Insect pollination has traditionally been the important ecosystem service for the productivity of apples. However, evidence suggests that supplies of pollination services, both from honeybees and wild pollinators, do not match the increasing demands (Ramirez et al., 2013). Due to the Colony Collapse Disorder (CCD), honeybees around the world have been dying off at alarming rates. Also, environmental conditions have a direct effect on when and how insect pollinators are active. Factors include heat, humidity and wind. Even localized micro-climates can have an effect on the work done by these insect pollinators. Under such conditions, farmers around the world constantly face a large number of risks and uncertainties when considering the pollination of their crops.
To minimize these risks, farmers have been always searching for alternative methods to gain more control over the pollination of apples, thus increasing food security and often increasing crop yield. Artificial pollination by applying pollens directly to the flowers is among them. The proposed research is designed to deliver a robotic system to address this challenge by creating a strategy to accurately identify flowers during the blooming period, track flower growth, and finally precisely apply pollens when the flower development is at the right stage.
One of the major challenges in robotic pollination is to develop a robust and accurate blossom detection algorithm that can detect king flowers in varying background and illumination conditions. Deep learning techniques, such as Faster Region-Based Convolutional Neural Network (Faster R-CNN) and Semantic Segmentation Network (SSN), have been proven to be robust, accurate and reliable in various vision-based object detection tasks (Dias et al., 2018), while there are many improvements need to be addressed before they can be integrated for robotic pollination. For example, the algorithm is able to identify the tentative area of blossoms but could not pinpoint exact location. The algorithm also suffers from the false positive detection of flowers for objects such as branches and leaves. This proposed project will focus on developing a Deep Learning-based vision system that precisely identify and locate king flowers in the tree canopies, which lead to efficient and reproducible pollination of apples and thus help increase the yield of high-quality apples.
In addition, thinning is a particular challenge for apple growers among Pennsylvania. Apple blossom thinning is a procedure that has been in practice for hundreds of years. The act of thinning or removal of blossoms is critical to efficient apple tree fruit production. The absence of blossom thinning can result in poor quality fruit, diminished fruit size, breakage of limbs, exhaustion of tree reserves, and ultimately reduction of tree life. The thinning practice is a complex, time sensitive procedure that reduces fruit branch loading, resulting in a higher quality, larger sized product. Currently, hand-thinning is still a necessary and costly practice in apple production, and an incalculable number of crops are lost annually due to lack of labor and the resulting non-thinned crops (NIFA, 2009).
Research
Method 1: Flower cluster dataset establishment
Deep Learning requires a large set of images to train the model and detect objects accurately. Among the Gala and Honeycrisp canopies planted in the orchard in the Fruit Research Extension Center located in Biglerville, PA, 20 random canopies will be picked from each variety. At each flowering stage defined by horticultural scientist, 40 images will be taken from each canopy from each variety. A total of 800 (40*20) images will be collected for lateral image processing.
Images will be pre-processed using the method of image augmentation. Four regular types of augmentation will be implemented: geometric transformation, flipping, cropping and rotation. As a result, 3200 images will serve as the experiment dataset.
Among all of the images, 2000 images will be randomly selected as the training set, while from the rest 1200 images, 800 images will be randomly chosen as testing set for the developed detection algorithm. The dataset will be input to a Faster R-CNN network with a pre-trained ResNet_101 model.
Images will be labeled manually with respect to five classes: Flower, Flower buds, Flower occluded by flower buds, Flower occluded by leaves, Flower occluded by other flowers. The output images are expected to be detected flowers with contours around them, and different color of the contours represents true positive, false positive and false negative.
During the image collection process, the counted number of King Flowers at different stages will also be recorded as the ground truth. The number is to be compared with the detection results of the vision system to test the accuracy of the algorithm. A margin of error around 5%-8% is expected.
Method 2: Flower Cluster Localization and King Blossom Identification
While previous approach on detection of apple flowers implemented semantic segmentation, instance segmentation has become the state-of-art approach in the area of machine vision research. It aims to predict the object class-label and the pixel-specific object instance mask, which identify specific objects within each single class variety (Hafiz & Bhat, 2020). Instance segmentation involves object detection using a box followed by object-box segmentation. One of the most successful approaches regarding object detection task can be Mask R-CNN. Mask R-CNN outperforms the Faster R-CNN detection algorithm using a relatively simple mask predictor which leads to higher accuracy and faster computation. Instance segmentation approaches based on Mask R-CNN have shown outstanding results in recent instance segmentation challenges (Hafiz & Bhat, 2020). Figure 1 illustrates the Mask R-CNN model used for detecting flowers in this study. The output from apple flower segmentation is the colored masks of individual flowers. These masks contain information about each detected flower including the size (number of pixels) and the location (coordinates).
Every fruiting spur on an apple tree produces a cluster of five to six buds: the central blossom is often known as the King Blossom, which is the critical target of precision pollination. In order to precisely identify the King Blossom within each flower cluster, the first step is to separately localize each flower cluster in the apple canopies. In this proposed study, a novel approach of flower cluster identification method is proposed using flower density 2D mapping process. With the output from the Mask R-CNN based instance segmentation described in previous paragraph, the masks of each detected apple flower can be easily separated out from the background as shown in figure 2.
The next step is to divide the image of flower masks into equally-sized super pixels based on the size of the original image. In this application, the size of image is 5184×3456, with the size of super pixel being chosen as 576×384, a total of 81 super pixels were obtained carrying different sections of flower masks as the results shown in Figure 3.
For each individual super pixel, the flower density is calculated as the percentage of flower mask pixels (white-colored pixels) over the total pixels, which is shown in the following equation:
Based on the calculated density value within each super pixel, a colormap is assigned to present whether the super pixel contains more flower mask pixels. The brighter the color is, more flower mask pixels are concentrated in the super pixel. A flower density mapping sample is shown in Figure 4.
Since flower clusters are more likely to be located at the place where has higher flower density than the surroundings, this algorithm is coded to output the coordinates of those brighter super pixels which represent the location of flower cluster with higher probability. With a further step, the density map is converted into a 3D mesh map, in Figure 4, for a clearer view of each possible flower cluster locations. Each output coordinate is treated as the center of a circle with a fixed radius. With appropriately chosen radius, the flowers included within each circle are considered as the detected flower cluster. From each separated flower cluster, the flower that located at the most centered position is considered as the king blossom in that cluster. The image dataset acquired in the Method 1 will be used to train and evaluate the flower cluster detection and king flower identification algorithms.
Method 3: Develop a path sequencing algorithm to support the decision making for chemical suspension
The path sequencing problem, or multi-goal path planning, is often mentioned in research aiming to improve the productivity for industrial robots (Alatartsev et al., 2015). An optimized trajectory leads to higher efficiency for the robotic system. The Traveling Salesman Problem (TSP) is the most widely implemented algorithm for path sequencing task, which allows the manipulator to reach every targeted point in a minimal-cost cyclic tour and return to its starting position. The TSP algorithm works fine in the application of chemical blossom thinning, however, is limited in apple flower pollination since there is only one point (flower) existed between paths, shown in the left image of Figure. 5. Thus, an advanced version of TSP will be introduced, which is referred as Generalized TSP (GTSP), shown in the right image of Figure. 5. The goal of GTSP is to find the minimal-cost cyclic tour that contains a point from every set. The advanced algorithm is expected to fulfill the application of both blossom thinning and pollination that each flower cluster (set) usually contains 5-6 flowers (points).
Method 4: Integrate the vision system and robotic manipulator to perform field evaluation (chemical thinning as an example)
Along with the developed flower density mapping vision system (Method #2), and the optimized path sequencing approach (Method #3), a series of performance tests will be conducted in the laboratory environment. The prototype of the proposed robotic pollination system consists of two ZED 2 cameras mounted on a UR-5e robotic arm with a chemical spraying end-effector located at the last joint of the manipulator. A 1.6 meters artificial tree will be decorated with artificial apple flowers to simulate apple canopies in the orchard (Figure. 6). Firstly, the vision system will be tested again to see if it successfully detects every flower within the canopy and generate the density map. With coordinates provided by the vision system, the path will be planned by the Robot Operation System (ROS) using inverse kinematic process. According to the GTSP algorithm, the sequence of path is optimized if each of the spraying targets is visited once, while the manipulator returns to its starting position at the end of the process. The outcomes from the lab study are expected to be implemented in the orchards for targeted pollination or thinning.
The success of the proposed project will be assessed using the following measures:
- The efficacy of the flower detection platform will be determined by comparing actual flower counts with those system detects within replicated flowering limbs of apples. A positive detection rate over 80% is expected for king flower and lateral flowers.
- Spraying target detection will be assessed by comparing spraying rate between our robotic system and mechanical spraying (air-blast spraying). The metric for success is less than 10% difference between manual and robotic systems.
- In the proposed field evaluation, an entire row of apple canopies will be reserved to be applied with three different treatments: targeted spraying, air-blast spraying, and un-thinned control (UTC). Three treatments will be randomly distributed among all the canopies. The usage of chemical will be compared between the proposed targeted spraying and the traditional air-blast spraying to see how much chemical will be saved by applying this robotic system. Later after the flowering stage, the green fruits will be counted to measure the effectiveness of thinning.
Objective 1: Flower Cluster Detection
Figure 1 shows an example of the detection results obtained with the developed Mask R-CNN with instance segmentation approach. The original output of the instance segmentation model contained pixel-leveled masks together with bounding box indicating that flower clusters were detected. In this study, since the most critical information needed from the vision system was the pair of x-coordinates of each flower cluster, which can be directly extracted from the bounding box, the visualization code was modified so that it only showed the bounding box in output images as shown in Figure 1.
Figure 1. Mask R-CNN based instance segmentation of apple flower clusters
The performance of developed flower clusters detection model was satisfying. Based on the Precision, Recall, and F1 score, along with field test observation, most of the flower clusters were correctly detected. The recorded data showed that there were 216 flower clusters counted in total on the 18 sample trees for CTS. The quantitative output from the vision system reported that 201 of them were detected, which led to an accuracy of 93.06%. An accurate vision system helps the overall system to locate the spraying targets more precisely and therefore lead to the better performance of blossom thinning.
Table 1. Precision, Recall, and F1-score for flower clusters detection model
Objective 2: Apple Blossom Thinning Test
The proposed cartesian target sprayer (CTS) was compared with an air-blast sprayer (ABS) and a boom-typed sprayer (BTS) in the application of apple blossom thinning. Three measurements were recorded during the field test: time used for spraying, usage of chemical (lime-sulfur), and the number of green fruit set before June Drop. It should be noted that the Cartesian target sprayer took 4.66 s to finish spraying one out of four sections of trellis wire apple trees. To unify all the data for comparison, the recorded time for Cartesian target sprayer was multiplied by four to estimate the total time it took to complete the spraying for the whole tree. It can be recognized from Table 2 that the processing time was not the advantage of the proposed spraying system. The Cartesian target spraying was six to nine times slower than boom-typed sprayer and air-blast sprayer. The main reason for the slowness was because the speed of the linear actuators on the Cartesian manipulator was set to one mph, while the driving speed of boom-typed sprayer and air-blast sprayer were around 3-3.5 mph. Another significant reason was the processing route: After finishing spraying for one limb, the Cartesian manipulator was required to return back to its home position before it can start reaching the next target limb, which doubled the distance the manipulator need to travel.
Despite of CTS’s longer processing time, it did an outstanding work to save the use of chemical thinner. As shown in the 3rd column of Table 2, the amount of chemical sprayer was decreased by 66.7% and 45.5% compared to ABS and BTS respectively.
Table 2. Comparison of Three Different Sprayers for the spraying time and chemical usage
Another major metric to assess the effectiveness of thinning was the number of green fruit set before June Drop. From the ANOVA analysis image in Figure 2, it can not be said that the number of green fruits on the tree thinned by CTS was significantly less than the other two treatments, but it was clear that the average number of green fruits was the least among these three treatments. The Cartesian target sprayer was able to achieve relatively the same effectiveness of thinning while reducing the usage of chemical thinner by 45.5% to 66.7%.
Figure 2. One-way ANOVA mean analysis of green fruit counting among three sprayers
Another interesting fact was found during the manual green fruits counting. Among the sample trees for the Cartesian target sprayer, researchers observed a few clusters that contain five to six green fruits, which meant no thinning was applied. The reason of that is all of these un-thinned clusters located at the back of the trellis wire, the Cartesian target sprayer was not able to either detect the flower clusters that were hidden or reach those locations with the current mechanical design. Such shortcoming needs urgent solutions to improve the robustness of the overall system.
Education & Outreach Activities and Participation Summary
Participation Summary:
The ultimate output of the research will be a prototype of robotic system for optimal apple crop load management. The system will include the proposed machine vision system and targeted chemical spraying system. The overall system will be demonstrated to growers and stakeholders and knowledge and information will be shared through various medium. During the period of this project, we will expand our dissemination and outreach activities by providing: (i) technical presentations in professional meetings, (ii) presentations during stakeholder meetings and public events, (iii) demos and presentations during field day events, and (iv) through research and extension publications. Below are a few detailed plans for outreach activities.
- The results of this project will be shared with researchers through publications, conference presentations, posters, and seminars. Project PIs will participate in various domestic conferences, including the Annual International Meeting of American Society of Agricultural and Biological Engineers (ASABE), Northeast Agricultural and Biological Engineering Conference (NABEC). The results will be presented at conferences through oral presentations and exhibition/posters at those scientific and professional society meetings to get feedback from other researchers. Major funding will also be published at some flag scientific journals, such as the Transactions of the ASABE, Computers and Electronics in Agriculture, among others.
- The outcomes from the project will be disseminated to stakeholders, especially tree fruit growers to gain engagement with the tree fruit industry. The PIs will present the research findings to stakeholders during the annual Penn State Fruit Research and Extension Center (FREC) field day and Mid-Atlantic Meetings with 50-100 participants. We could also present the research findings to broad grower community through events such as winter and spring grower meetings organized by Penn State Extension. Meanwhile, we will publish our research outcomes in the extension newsletters, such as Fruit Times, which is a widespread article in the Pennsylvania State.
- The PIs will also showcase the developed sensing technology and robotic pruning system to the general public, such as STEM students at Penn State and 4-H robotic groups. The results or the robotic system will be exhibiting or posting at Penn State Gamma Sigma Delta Research exposition and Penn State Ag Progress Day.
In summary, this project focuses on machine vision-based precision apple flower cluster and king blossom identification, thus to provide guideline information for development of a precision and robotic solution for apple crop load management. Upon the success of the project, it is our plan to carry out more outreach activities through Penn State extension for practical trials and application.
Project Outcomes
The ultimate output of the research will be a prototype of robotic system for optimal apple crop load management. The system will include the proposed machine vision system and targeted chemical spraying system. The overall system will be demonstrated to growers and stakeholders and knowledge and information will be shared through various medium. During the period of this project, we will expand our dissemination and outreach activities by providing: (i) technical presentations in professional meetings, (ii) presentations during stakeholder meetings and public
events, (iii) demos and presentations during field day events, and (iv) through research and extension publications. Below are a few detailed plans for outreach activities.
- The results of this project will be shared with researchers through publications, conference presentations, posters, and seminars. Project PIs will participate in various domestic conferences, including the Annual International Meeting of American Society of Agricultural and Biological Engineers (ASABE), Northeast Agricultural and Biological Engineering Conference (NABEC). The results will be presented at conferences through oral presentations and exhibition/posters at those scientific and professional society meetings to get feedback from other researchers. Major funding will also be published at some flag scientific journals, such as the Transactions of the ASABE, Computers and Electronics in Agriculture, among others.
- The outcomes from the project will be disseminated to stakeholders, especially tree fruit growers to gain engagement with the tree fruit industry. The PIs will present the research findings to stakeholders during the annual Penn State Fruit Research and Extension Center (FREC) field day and Mid-Atlantic Meetings with 50-100 participants. We could also present the research findings to broad grower community through events such as winter and spring grower meetings organized by Penn State Extension. Meanwhile, we will publish our research outcomes in the extension newsletters, such as Fruit Times, which is a widespread article in the Pennsylvania State.
- The PIs will also showcase the developed sensing technology and robotic pruning system to the general public, such as STEM students at Penn State and 4-H robotic groups. The results or the robotic system will be exhibiting or posting at Penn State Gamma Sigma Delta Research exposition and Penn State Ag Progress Day.
In summary, this project focuses on machine vision-based precision apple flower cluster and king blossom identification, thus to provide guideline information for development of a precision and robotic solution for apple crop load management. Upon the success of the project, it is our plan to carry out more outreach activities through Penn State extension for practical trials and application.
In this season, a cartesian robotic system was developed, which was based on Mask R-CNN network with instance segmentation model that represents the state-of-art approach for computer vision application. Instead of heavy labor cost, with the help of the robotic system, growers just need to place the system in their apple orchards and leave most of work to the robots. The completed robotic system will automatically localize the targeted flowers to perform precision thinning. In addition, the Cartesian target spraying system reduced the chemical thinner to the trees, comparing to two conventional spraying technologies (boom-typed sprayer and air-blast spray). For chemical blossom thinning, saving chemical (i.e., lime sulfur) not only reduce the cost, but also minimize the environmental impact. The overall system is expected to increase the profitability of the apple production and maintain the sustainability of the apple industry.
Even though initial focus will be in robotic blossom thinning in apples, the technology is expected to be applicable, with minimal modification, to other tree fruit crops, thus benefiting a large number of additional specialty crop producers. In addition, particularly sensing technologies for flower cluster detection and localization, and manipulation technology for targeted spraying end-effector may benefit a range of other applications that rely on robotics in production agriculture and may extend to other cropping systems as well. Therefore, the proposed study is highly relevant and timely with a potential to make a huge positive impact on the competitiveness and economic and social sustainability of the overall tree fruit industry in the country.