Apple blossom density estimation and mapping through UAV-based RGB photogrammetry

Progress report for GNE21-274

Project Type: Graduate Student
Funds awarded in 2021: $14,997.00
Projected End Date: 10/31/2022
Grant Recipient: The Pennsylvania State University
Region: Northeast
State: Pennsylvania
Graduate Student:
Faculty Advisor:
Expand All

Project Information

Project Objectives:

The overall goal of the project is to develop a point cloud processing and mapping algorithm that can assist farmers in the decision-making of precise blossom thinning by allowing convenient blossom density estimation and direct orchard blossom density visualization with drone technology. Specifically, the project objectives include:
1. Reconstruct dense point cloud models of apple trees during blooming using aerial RGB images and photogrammetry software.
2. Develop and evaluate a point cloud-based apple tree blossom count estimation method.
3. Develop a flexible apple tree blossom density mapping algorithm that allows the visualization of blossom density at various tree heights.

Introduction:

Apple (Malus domestica) is one of the most valuable and culturally significant fruits worldwide [1]. For the crop year 2020/2021, global apple production is forecasted to be 76.1 million tons [2]. In US, more than 5,000 farms with almost 85,000 acres of apples are located in the northeastern region, and more than 200,000 tons of apples can be produced in Pennsylvania per year [3]. Despite apple has been a domesticated crop for thousands of years [1], challenges still exist in fruit production management. Apple trees tend to overproduce flowers, which will grow into fruits with inadequate sizes and less marketable values due to insufficient resource allocation. Following the years with large yields of small sized fruits, certain apple varieties usually have low or even no yields at the following years, which is known as the alternate bearing or biennial bearing phenomenon [4]. In order to achieve stable production of apples with high quality year after year, human interventions are needed for trees to have a balanced ratio between canopy photosynthesis and fruit load [5].

Thinning, the action of removing a fraction of blossoms or fruits from trees, is a crucial crop load management practice for farmers to improve fruit size and quality [6] and suppress alternate bearing for consistent yields over the years [5]. Common thinning methods include manual and chemical thinning. Manual thinning is a labor-intensive and time-consuming process, during which fruitlets are manually removed typically after the physiological fruit drop based on their size and proximity to one another [7]. As an alternative approach, chemical substances can be sprayed on trees to prevent flower pollination or fertilization, or to stimulate fruitlets drop [8]. Despite early blossom thinning is considered to be more effective at increasing fruit size, quality and color at harvest and promoting return blooming [9], due to the variability of chemical uptake and tree sensitivity, the unpredictability of chemical thinning has been an issue for apple growers for more than 50 years [10]. Moreover, traditionally blossom thinning requires human experts to visually rate tree blooming intensity prior to chemical application, which can be inaccurate and inefficient. In order to improve the reliability of chemical blossom thinning and reduce its reliance on human, an automated solution that can assist farmers in blossom density monitoring and assessment, thus inform precise chemical spraying and timely management plan adjustment is needed.

Research

Materials and methods:

1. Study Site, Hardware, and Software

Field experiments will be conducted at Penn State Horticulture Research Farm (Pennsylvania Furnace, Pennsylvania, USA). Apple trees in block A1 (40.708765° N, 77.953676° W) (Figure 1) and block CW1 (40.707918° N, 77.954370° W) will be utilized for developing the blossom count estimation methodology and demonstrating the blossom density mapping algorithm respectively. Mavic 2 Zoom (DJI, Shenzhen, China) will be employed for apple blossom RGB data acquisition. Autonomous flights will be conducted using DroneDeploy (San Francisco, USA). PIX4Dmapper (Pix4D, Lausanne, Switzerland) will be used to reconstruct dense 3D point cloud models of apple trees using the collected RGB images.

Figure 1. Apple tree rows in block A1 at Penn State Horticulture Research Farm (Pennsylvania Furnace, Pennsylvania, USA).

2. Blossom Count Estimation

2.1. Data Collection

The development of an accurate blossom density estimation algorithm relies on calibrating digital data with accurate ground truth data, that is, correlating apple blossom counts derived from point clouds with manually counted blossom numbers. However, not only blossom distribution on apple trees can be extremely dense, but also branches of trees can be intertwined due to the orchard tree training systems, making counting all blossoms of a tree accurately an extremely challenging task. To increase the accuracy of manual blossom counting, two tree blocking structures, designed to be lightweight and easy to set up and disassemble, were constructed. Each structure skeleton consists of eight PVC pipes and eight three-way pipe fittings. Two pieces of tarps were wrapped around two sides of the skeleton and fastened with Velcro (Figure 2).

Figure 2. Constructed tree blocking structures.

During manual counting and corresponding RGB data collection, two structures will be placed over the same tree row, however, only the blossoms that are in between the two structures are counted. By changing the locations of the two structures and adjusting the distance between them, blossom counting regions containing different amounts of blossoms can be created (Figure 3). At least 20 “data pairs” of the apple orchard will be collected during the first bloom or full bloom growth stage in 2022 growing season. Each data pair consists of one manual blossom count and one set of RGB images with high overlapping percentages taken around a blossom counting region in between the two tree blocking structures.

Figure 3. Schematic diagram showing the blossom counting region in between the tree blocking structures.

2.2. Point Cloud Processing

Every reconstructed point cloud model initially will include not only the apple trees inside a blossom counting region, but also the tree blocking structures and other apple trees outside the structures. The raw point cloud models will first be cleaned up manually by deleting all points that do not belong to the blossom counting regions. The distinctive shape and color of the tree blocking structures can help easily identify blossom counting regions during this process. After obtaining point cloud models containing only apple trees inside blossom counting regions, a white-color thresholding will be utilized to remove non-flower points. Since the density of point clouds reconstructed by photogrammetry techniques can be influenced by many factors such as lighting condition, distance between camera and object, etc., to make the algorithm more robust against environmental variables, flower point cloud models will be further downsampled using grid averaging to minimize the influence from inconsistent point cloud density. Optimal grid size will be investigated. A linear relationship between the numbers of unique grid-averaged flower points in blossom counting regions and corresponding manual flower counts will be established based on the data pairs and evaluated using root mean square error (RMSE).

3. Blossom Density Mapping

In order to generate apple orchard blossom density maps of user-defined tree height regions, variation of terrain elevation within the orchard needs to be first compensated. An orchard point cloud is downsampled using grid averaging by a customizable grid size. A maximum height map and a minimum height map are generated consequently where each pixel represents the height of the highest or the lowest point respectively. A height difference map is calculated by subtracting the minimum height map from the maximum height map. A terrain map is generated using a four-step procedure based on the minimum height map and the height difference map. First creating a mask to identify terrain regions free of obstacles by thresholding the height difference map with a customizable height value. Second applying the mask onto the minimum height map to remove the obstacle regions. Third, filling the obstacle regions through neighboring pixel interpolation. Lastly, the interpolated terrain map is smoothed to reduce local elevation noise.

As described in Section 2.2, apple blossom points will be extracted from the orchard point cloud using white-color thresholding, whose heights will then be adjusted based on the smoothed terrain map. Blossom counts will be estimated based on the downsampled blossom point counts derived with the optimal grid size. A blossom density map with the same spatial resolution as the terrain map will be generated by counting the number of blossom points within the predefined tree height regions for each terrain pixel and dividing the blossom counts by the volume of tree height regions. The mapping algorithms will be developed in MATLAB. A point cloud model of block CW1 reconducted using RGB image collected on May 1, 2021 will be deployed for mapping algorithm demonstration.

Research results and discussion:

1. Orchard Point Cloud Model

Figure 4 shows a reconstructed dense point cloud model of block CW1 using PIX4Dmapper. The model was generated from 274 RGB images and consists of approximately 200 million points. The generation time of such a point cloud was roughly four hours.

Figure 4. An example of a reconstructed orchard point cloud from PIX4Dmapper.

2. Orchard Terrain Map

Figure 5 shows a terrain map generated using the orchard point cloud model in Figure 4. The map has a 0.1 m resolution and was smoothed using a 11×11 kernel.

Figure 5. An example of an orchard terrain map.

3. Blossom Point Cloud Model

Figure 6 shows the terrain height-adjusted, grid-averaged blossom point cloud model derived from the orchard point cloud model in Figure 4. The grid size used was 0.1 m. As a tiny portion of ground points can have white color too, only white points with adjust heights larger than 0.3 m were considered as blossom points.

Figure 6. An example of an orchard blossom point cloud model.

Participation Summary

Education & Outreach Activities and Participation Summary

Participation Summary:

Education/outreach description:

Aside from one open-access journal article and a presentation at ASABE 2022 Annual International Meeting, the final result of the study will also be shared with apple growers, tree fruit industry, and research community in northeastern US via:

• Presentation at IFTA 65th Annual Conference and Tours
• Field demonstration at IFTA 65th Annual Conference and Tours

Any opinions, findings, conclusions, or recommendations expressed in this publication are those of the author(s) and do not necessarily reflect the view of the U.S. Department of Agriculture or SARE.