Progress report for GNE24-337
Project Information
Apple crop load management is critical for balancing high yield and quality fruit production while ensuring sustainable tree health. Artificial Spur Extinction (ASE) is a new crop management technique that reduces the floral bud density of the tree canopy. This project aims to optimize the ASE process in apple orchards through advanced vision-based robotics, enhancing fruit quality and ensuring regular annual bearing. This technique employs automated, precise bud thinning based on the branch cross-sectional area (BCA) to achieve the desired fruit size. Our study follows three primary objectives: Firstly, developing a vision-based system for assessing bud density at the tree and branch levels enables accurate thinning decisions. Second, a decision support system for bud thinning will be developed based on the bud density and distributions at the branch and tree levels. Lastly, the robotic thinning system undergoes integration and evaluation through comprehensive field testing. Key to the success of this study is the engagement with the local agricultural community through a comprehensive outreach strategy. The technology will be demonstrated on field days targeted at local apple growers, and the findings and methodologies will be shared through extension talks and research publications. This approach facilitates the direct transfer of technology and encourages feedback for further refinement. The expected outcome is a proof of concept to showcase the feasibility and benefits of robotic bud thinning, potentially advancing apple orchard management and significantly boosting profitability through improved crop quality and yield.
This proposed research aims to develop a robotic bud thinning system for early crop load management in orchards, considering branches and tree levels for good-quality apples. The project will be structured around three key objectives:
Objective #1: Develop Deep Learning-Based Algorithms for Apple Bud Detection and Localization in Orchard Environments
Accurate detection and localization of apple buds within diverse orchard environments are foundational to precision agriculture practices, particularly in the context of bud thinning. This objective addresses a critical gap in current agricultural technology by developing sophisticated deep-learning algorithms that analyze visual data for bud density and specific locations. These algorithms are about identifying the buds and understanding the intricate dynamics of crop load at both branch and tree levels. The precision and effectiveness of bud thinning, directly influenced by the accuracy of these detections, can significantly affect the fruit quality and quantity, making this technology pivotal for sustainable and profitable orchard management.
Objective #2: Create a Decision Support System for Bud Thinning Based on the Bud Density and Distributions at the Branch and Tree Levels
Creating a decision support system (DSS) for bud thinning based on a comprehensive analysis of bud density and distribution represents a leap towards intelligent crop management. Such a system moves beyond the conventional one-size-fits-all approach to thinning, offering instead a nuanced, data-driven strategy that considers the unique characteristics of each tree and branch. This approach optimizes resource use and ensures that each action taken contributes to the overall health and productivity of the orchard. The DSS shifts from manual to data-driven decision-making, offering insights that boost thinning efficiency, cut labor costs, and enhance crop yield and quality.
Objective #3: Design and Test a Robotic End-Effector for Effective Bud Removal for Early Apple Crop Load Management
Designing and testing a robotic end-effector for efficient bud removal tackles one of the most labor-intensive and skill-dependent tasks in orchard management. Introducing such a device is not merely an advancement in mechanical design but a fundamental shift towards fully automated, precision agriculture. The effectiveness of bud thinning, crucial for controlling crop load and ensuring the development of high-quality fruit, depends significantly on the precision and gentleness of the removal process. A robotic end-effector that can quickly, accurately, and safely remove unwanted buds without damaging the tree opens the door to unprecedented efficiency and effectiveness in orchard management.
The purpose of this project is to optimize apple orchard productivity through the integration of advanced robotic technology for Artificial Spur Extinction (ASE), targeting the automation of bud thinning to manage early crop loads efficiently. The number of blossoms in commercial apple production is essential for yield, bearing habits, fruit storage potential, and profitability. Thinning is a critical management decision that profoundly influences orchard productivity and crop value over several cycles (Musacchi, 2017). However, traditional thinning practices require a high level of expertise from growers and are time-consuming. Consequently, fruit growers are interested in an automated decision-support method to manage this balance with precision and reliability.
Crop-load management in U.S. orchards operates under established guidelines encompassing tree pruning, flower thinning, and fruitlet thinning (Zhang et al., 2019). A vital component of these guidelines is early thinning, performed before petal fall, which is particularly effective in increasing fruit size (Lakso et al., 1996). This precise regulation of the number of buds on branches is crucial as it directly influences the potential yield and quality of the fruit. Within this framework, the technique known as Artificial Spur Extinction (ASE), or spur pruning, emerges as an alternative method. As demonstrated by Lauri et al. (2002 ; 2011), ASE employs selective bud removal to enhance fruit size and reduce biennial bearing tendencies significantly, optimizing light exposure and the positioning of the spurs, thus underlining the importance of strategic bud management in achieving optimal orchard productivity.
Despite the known benefits of precise crop-load management, implementing these practices in varied and unstructured orchard environments presents significant challenges. Environmental variability has hindered the development of automated systems for these tasks, making the design of effective robotic solutions more complex (Bechar and Vigneault, 2016).
Recent advancements in ASE research have led to significant improvements in crop load management by removing the majority of lateral buds to leave optimal spacing (Tustin et al., 2010; Van Hooijdonk et al., 2014; Breen et al., 2015). Studies have shown that managing the density and positioning of buds, especially terminal buds on older wood, can yield larger fruit sizes, attributing this benefit to enhanced spur leaf area and better light exposure (Volz et al., 1994; Breen et al., 2014). Inspired by these findings, this project is set to apply robotic technology to replicate and enhance the ASE method, targeting the automation of bud thinning to optimize early crop load management efficiently.
The project aims to incorporate robotic technologies that offer efficient methods for removing unwanted buds and achieving the optimal density of buds on each branch. This goal is attainable by developing deep learning algorithms that enable accurate bud density calculations, precisely detecting and locating buds from a machine vision system. Developing an end-effector capable of removing specific buds without harming trees, combined with integrating the robot manipulator and vision system, will facilitate fully automated bud thinning work. Accurate counting apple buds on a specific branch or tree is crucial for effective bud thinning and precision crop load management. A decision support system (DSS) is necessary to identify bud density by combining branch diameter with bud counting and locations. This system will also determine the buds that should be removed. Due to the complex nature of the apple orchard environment, a specialized manipulator suited to orchard operations will be developed. This manipulator will link the robotic system to the environment and direct the end-effector to the precise location of unwanted buds. Integrating the vision system and support algorithms allows for the complete automation of the robot.
The drive for robotic bud thinning is crucial for aligning apple orchard management with early crop load management strategies, which supports Northeast SARE’s outcome statement by promoting sustainable agriculture practices that enhance both the environmental sustainability and economic viability of orchard management. The robotic technology introduced in this project could alleviate labor-intensive tasks and improve orchard productivity by introducing an innovative method for bud thinning that accounts for factors such as size, shape, branch diameter, and bud density. Beyond marking a leap forward in agricultural technology, adopting this automated, precision-based system in apple orchards illustrates a more profound dedication to sustainable farming practices. By automating bud thinning to suit the unique characteristics of each tree, the proposed method aims to significantly improve accuracy and reliability, addressing a long-standing need among apple growers for more efficient management techniques.
Cooperators
- (Educator)
- (Educator)
Research
The proposed structure of this study, shown in Figure 1, suggests that automated technology will revolutionize the rapidly developing precision agriculture sector. This framework provides a strategic approach to apple orchard management that promises increased crop yields and quality because of the development of a robotic bud thinning system. The method consists of three main components: a data-driven decision support system for best thinning techniques, robotic end-effector precision engineering for precise bud removal, and sophisticated deep learning for bud detection.
Figure 1: Sequential Framework of Research Processes: From Detection to Precision Thinning
Approach and Method 1: Develop Deep Learning-Based Algorithms for Apple Bud Detection and Localization in Orchard Environments.
This section describes the approach and techniques used to accomplish Objective 1, which consists of three main activities. The first task involves collecting data on apple buds under various conditions to train the algorithm. The second task involves developing an algorithm capable of determining the locations of apple buds. The third task includes testing the algorithm in the orchard. The anticipated outcome is for the vision system to detect and locate buds successfully within a real-world environment. The flowchart of this study is presented in Figure 2.
Figure 2: Sequential Framework of Developing Deep Learning-Based Algorithms for Apple Bud Detection and Localization in Orchard Environments
Task 1.1: Image data acquisition.
During the initial phase of the study, an extensive data collection and preprocessing effort was undertaken in the spring of 2024, which is essential for developing deep learning algorithms. This process began with capturing thousands of images from various participating orchards, aiming to cover a comprehensive difference in tree structure, bud stages, and lighting environmental conditions to ensure the richness and diversity of the dataset. After capturing images, each was carefully annotated using advanced software tools to mark the exact locations of Apple buds, creating a labelled dataset crucial for training and testing the algorithms. To enhance the model’s resilience against natural variability found in orchard environments—such as differences in lighting, scale, and orientation—the preprocessing regimen included resizing, normalizing, and augmenting the images. This comprehensive data collection and preprocessing approach laid the groundwork for the subsequent development of highly accurate and robust deep-learning models for bud detection and localization.
Task 1.2: Deep learning-based algorithm development.
The model development phase involved critically evaluating various deep learning architectures, including convolutional neural networks (CNNs), recurrent neural networks (RNNs), and hybrids, to determine the most effective bud detection and localization framework. Once the optimal model was identified, the annotated dataset was employed to train it, with careful adjustments to parameters and layers to enhance accuracy while minimizing the risk of overfitting. Techniques such as transfer learning were also utilized, leveraging pre-trained models to boost efficiency further. The dataset was divided into training, validation, and test sets to evaluate the model’s performance rigorously. Performance metrics, including precision, recall, and the F1 score, were applied to gauge the model’s accuracy in detecting and localizing buds, ensuring a robust and reliable system for implementation in orchard management.
Tasks 1.3: Model evaluation through field testing.
In the field testing and iteration phase, the trained model was used in a field-tested mobile device or system to evaluate its performance in various orchard environments. This critical phase assessed the model’s performance outside of controlled situations, emphasizing how well-suited it is to the circumstances seen in real orchards. The input from these field tests was carefully gathered, identifying any flaws or possible places where the system may be strengthened. It was crucial to go through an iterative process of model refinement based on feedback from the real world to improve the model’s ability to handle the complexity and variety of natural surroundings. The model’s resilience and reliability were significantly increased through ongoing modifications and enhancements, guaranteeing its adaptability and efficacy in real-world orchard management.
Approach and Method 2: Create a Decision Support System for Bud Thinning Based on the Bud Density and Distributions at the Branch and Tree Levels.
This objective is divided into two main tasks: developing the decision support system for Bud Thinning. The first task involves data collection for the algorithm from Objective 1, which will be utilized to count the apple buds on each branch and measure their diameters. The second task uses the branch diameter and the count of buds on each branch to develop the decision support system for bud thinning. The outline of this work is illustrated in Figure 3.
Figure 3: Sequential Framework of Creating a Decision Support System for Bud Thinning Based on the Bud Density and Distributions at the Branch and Tree Levels
Task 2.1: Bud counting and branch diameter measurement.
In creating a decision support system (DSS) for bud thinning, an intricate procedure is employed to gather and analyze branch-level bud density data using RGB-D sensors and imaging technologies. This approach captures tree species’ diverse traits and developmental phases. Integral to this process is incorporating techniques from the Artificial Spur Extinction (ASE) method. This novel crop management strategy reduces bud density to enhance the tree’s annual flowering and fruit bud potency. The analysis method utilizes branch diameter measurements and advanced bud detection algorithms to calculate bud density precisely. Crucially, this process involves determining the optimal bud density that aligns with the desired crop load for achieving the targeted fruit size, a decision based on the branch cross-sectional area (BCA). The development and application of machine learning algorithms are informed by this comprehensive analysis and historical yield data, which enable the prediction of optimal thinning strategies for each branch. These algorithms significantly refine bud management, which is central to the DSS. It equips the system to provide targeted, data-driven guidance for effective bud-thinning practices.
Task 2.2: Development of decision support system (DSS) for bud thinning.
A decision support system (DSS), refined through field testing in selected orchards, is meticulously evaluated for its capacity to automatically determine which buds should be removed or retained for optimal early crop load management. This stage assesses the system’s real-world applicability, emphasizing the precision of its automated recommendations against manual decisions. The feedback loop with orchard managers and agronomists is crucial for enhancing the system’s usability and ensuring its recommendations align with practical orchard management needs. Comparative studies then measure the DSS’s influence on thinning practices, labor efficiency, and crop outcomes, using insights from these evaluations to fine-tune the system for broader deployment. This iterative refinement underscores the DSS’s capability to adapt to diverse orchard environments, showcasing its potential to revolutionize crop management by providing targeted, data-driven guidance on bud removal decisions.
Approach and Method 3: Design and Test a Robotic End-Effector for Effective Bud Removal for Early Apple Crop Load Management.
Task 3.1: Design of end-effector for apple bud thinning.
Understanding crop variability is essential to developing an end-effector for apple bud thinning because it determines the best way to remove buds based on their mechanical and physical properties. With this information, an end-effector can be created to remove undesired buds while carefully protecting the surrounding plant tissue and the surviving buds. Therefore, the effectiveness of the end-effector is measured not just by its precision in removing unwanted buds but also by its ability to protect the adjacent plant tissue and buds. This dual requirement emphasizes the delicate balance the end-effector must maintain, essential in optimizing early crop load management.
Task 3.2: Design of robot manipulator for apple early crop load management.
A manipulator will be specifically engineered to navigate the unique architecture of apple orchards, enabling it to access apple buds effectively. Given the distinctive structural complexities of orchards, this task requires the development of a manipulator adept at maneuvering within these environments. Positioning tests will be conducted under laboratory and orchard conditions to validate its functionality, ensuring the manipulator’s performance is consistent across different settings. This manipulator will be integrated with a vision system and a decision-making algorithm. Before cutting action by the end-effector, the vision system and manipulator must work in unison to ensure precise positioning and identification of the targeted buds. This step is essential, as it guarantees the system’s effectiveness in selectively thinning buds, enhancing the overall precision and efficiency of the orchard management process.
Task 3.3: Integration of a robotic bud thinning system with the end-effector, machine vision system, and decision support system.
This task is dedicated to integrating the end-effector, vision system, and robotic manipulator to create a unified framework for managing apple orchards with precision and efficiency. Field tests to evaluate the performance and effectiveness of the developed precision system will be conducted at the research orchards of the Penn State Fruit Research and Extension Center (FREC). This integration is essential because it guarantees that the precise manipulator movements and precise bud detection by the vision system are synchronized with the end-effector’s cutting activities. These features work together to avoid damage to nearby branches and automatically target bud removal. Assembling a system that functions well as a whole, with each part supporting the others to offer the maximum accuracy and efficiency for the automated thinning process, is the first step towards effectively finishing this operation. The effects of the robotic thinning system on early crapload management will be evaluated, including bud detection accuracy, bud removal success rate, and the overall efficiency of the robotic system.
Figure 4 Overall steps of this work after integrating all components, including system deployment, manipulator positioning at the branch junction, branch diameter measurement, YOLO-based object detection displayed on the monitor, and real-time bud detection and tracking as the user manually moves the camera along the branch.
Thinning is a critical practice in apple orchard management, directly influencing crop load and fruit quality. To assist automated crop load management, a machine vision system for apple bud detection was developed to be integrated with robotic platforms. The system employed a Kinect a Kinect Azure sensor for real-time bud detection and branch diameter measurement, utilizing a YOLOv8-based object detection model trained and evaluated across multiple datasets. The evaluation identified the best-performing model by balancing precision, recall, and robustness in the complex and unstructured environments of apple orchards. Among the bud detection models tested, Model 7 achieved the highest precision (86%), while Model 2 recorded the highest recall (68%) and mean average precision (mAP) of 72%. Model 3, trained on a combination of FLIR and Kinect Azure data, demonstrated balanced performance (precision: 68%, recall: 55%, mAP: 59%) and was selected for deployment due to its robustness in handling diverse training data and compatibility with the Kinect Azure sensor for real-world applications. Two proposed imaging methods for branch diameter measurement were validated against manual caliper-based measurements, with statistical analysis revealing no significant differences (p = 0.98). These findings confirm the semi-automated methods as reliable and labor-efficient alternatives for field applications. Additionally, the bud counting algorithm demonstrated accurate tracking and counting of apple buds, effectively avoiding omissions and duplications in real orchard settings. This study underscores the potential of vision systems to revolutionize apple bud thinning, providing a strong foundation for the development of fully automated solutions in precision orchard management.
Education & outreach activities and participation summary
Participation summary:
This outreach program aims to enhance the awareness and adoption of a groundbreaking robotic technology designed to facilitate the implementation of the Artificial Spur Extinction (ASE) technique among apple growers. The heart of this program is to underline the significant economic and environmental gains from utilizing this robotic system, illustrating its effectiveness in enhancing apple quality and yield through the optimal density of flower buds. The centerpiece of this initiative is to demonstrate the extensive economic and environmental advantages of deploying this technology. By achieving an optimal balance of flower buds per branch, the system aims to significantly improve apple quality and yield, establishing a new benchmark in precision agriculture and orchard management.
Plan #1: Promoting the Robotic Technology for ASE Implementation to Apple Growers
The introduction of this novel robotic technology will be published in the Penn State Fruit Times and the Pennsylvania Fruit News, presentations at significant agricultural conferences, and papers in peer-reviewed journals. These efforts aim to reach a broad audience of apple growers, particularly in the Northeast, to generate interest and engagement. The communication will highlight how the robotic system’s innovative capabilities enable growers to make informed decisions for early crop load management. It will detail the system’s distinct advantages over traditional orchard management technologies and outline the benefits of integrating it with the ASE technique, illustrated through results from initial field trials.
Plan #2: Understanding Grower Perspectives on Current Orchard Management Practices
Several apple growers in Pennsylvania engage in thinning practices during the green fruit season. A survey targeting these growers will explore their current use of orchard management technologies, their satisfaction with these methods, and their willingness to adopt robotic bud-thinning system approaches. The insights gathered from this survey will serve as a foundation to assess growers’ familiarity with advanced thinning technologies and pinpoint specific educational gaps. These findings will inform the focus areas for upcoming demonstrations and workshops, ensuring they effectively address the needs and enhance the understanding of participating growers.
Plan #3: Engaging Growers with Demonstrations and Workshops
The centerpiece of the outreach initiative will be a series of live demonstration events and workshops. These sessions showcase the robotic technology’s functionality, aimed at optimizing early crop load management, highlighting its sophisticated design, user-friendly operation, and versatility across various orchard settings. The demonstrations will cover the system’s installation, operation, and maintenance, emphasizing its seamless integration with growers’ existing equipment and practices. Furthermore, workshops will detail the operational benefits, focusing on labor savings, improved crop load management, and consequent fruit quality and yield improvements.
These events will offer practical insights into the technology’s application and serve as forums for growers to ask questions, provide feedback, and discuss the potential impacts on their operations. By offering valuable hands-on experience, the initiative aims to build confidence in this innovative technology and encourage its adoption for a more sustainable and productive future in apple orchard management.