Development of a Machine Vision-Based Robotic Apple Crop Load Management System at Bud Stage

The proposed structure of this study, shown in Figure 1, suggests that automated technology will revolutionize the rapidly developing precision agriculture sector. This framework provides a strategic approach to apple orchard management that promises increased crop yields and quality because of the development of a robotic bud thinning system. The method consists of three main components: a data-driven decision support system for best thinning techniques, robotic end-effector precision engineering for precise bud removal, and sophisticated deep learning for bud detection.

Figure 1: Sequential Framework of Research Processes: From Detection to Precision Thinning

Approach and Method 1: Develop Deep Learning-Based Algorithms for Apple Bud Detection and Localization in Orchard Environments.

This section describes the approach and techniques used to accomplish Objective 1, which consists of three main activities. The first task involves collecting data on apple buds under various conditions to train the algorithm. The second task involves developing an algorithm capable of determining the locations of apple buds. The third task includes testing the algorithm in the orchard. The anticipated outcome is for the vision system to detect and locate buds successfully within a real-world environment. The flowchart of this study is presented in Figure 2.

Figure 2: Sequential Framework of Developing Deep Learning-Based Algorithms for Apple Bud Detection and Localization in Orchard Environments

Task 1.1: Image data acquisition.

During the initial phase of the study, an extensive data collection and preprocessing effort was undertaken in the spring of 2024, which is essential for developing deep learning algorithms. This process began with capturing thousands of images from various participating orchards, aiming to cover a comprehensive difference in tree structure, bud stages, and lighting environmental conditions to ensure the richness and diversity of the dataset. After capturing images, each was carefully annotated using advanced software tools to mark the exact locations of Apple buds, creating a labelled dataset crucial for training and testing the algorithms. To enhance the model’s resilience against natural variability found in orchard environments—such as differences in lighting, scale, and orientation—the preprocessing regimen included resizing, normalizing, and augmenting the images. This comprehensive data collection and preprocessing approach laid the groundwork for the subsequent development of highly accurate and robust deep-learning models for bud detection and localization.

Task 1.2: Deep learning-based algorithm development.

The model development phase involved critically evaluating various deep learning architectures, including convolutional neural networks (CNNs), recurrent neural networks (RNNs), and hybrids, to determine the most effective bud detection and localization framework. Once the optimal model was identified, the annotated dataset was employed to train it, with careful adjustments to parameters and layers to enhance accuracy while minimizing the risk of overfitting. Techniques such as transfer learning were also utilized, leveraging pre-trained models to boost efficiency further. The dataset was divided into training, validation, and test sets to evaluate the model’s performance rigorously. Performance metrics, including precision, recall, and the F1 score, were applied to gauge the model’s accuracy in detecting and localizing buds, ensuring a robust and reliable system for implementation in orchard management.

Tasks 1.3: Model evaluation through field testing.

In the field testing and iteration phase, the trained model was used in a field-tested mobile device or system to evaluate its performance in various orchard environments. This critical phase assessed the model’s performance outside of controlled situations, emphasizing how well-suited it is to the circumstances seen in real orchards. The input from these field tests was carefully gathered, identifying any flaws or possible places where the system may be strengthened. It was crucial to go through an iterative process of model refinement based on feedback from the real world to improve the model’s ability to handle the complexity and variety of natural surroundings. The model’s resilience and reliability were significantly increased through ongoing modifications and enhancements, guaranteeing its adaptability and efficacy in real-world orchard management.

Approach and Method 2: Create a Decision Support System for Bud Thinning Based on the Bud Density and Distributions at the Branch and Tree Levels.

This objective is divided into two main tasks: developing the decision support system for Bud Thinning. The first task involves data collection for the algorithm from Objective 1, which will be utilized to count the apple buds on each branch and measure their diameters. The second task uses the branch diameter and the count of buds on each branch to develop the decision support system for bud thinning. The outline of this work is illustrated in Figure 3.

Figure 3: Sequential Framework of Creating a Decision Support System for Bud Thinning Based on the Bud Density and Distributions at the Branch and Tree Levels

Task 2.1: Bud counting and branch diameter measurement.

In creating a decision support system (DSS) for bud thinning, an intricate procedure is employed to gather and analyze branch-level bud density data using RGB-D sensors and imaging technologies. This approach captures tree species’ diverse traits and developmental phases. Integral to this process is incorporating techniques from the Artificial Spur Extinction (ASE) method. This novel crop management strategy reduces bud density to enhance the tree’s annual flowering and fruit bud potency. The analysis method utilizes branch diameter measurements and advanced bud detection algorithms to calculate bud density precisely. Crucially, this process involves determining the optimal bud density that aligns with the desired crop load for achieving the targeted fruit size, a decision based on the branch cross-sectional area (BCA). The development and application of machine learning algorithms are informed by this comprehensive analysis and historical yield data, which enable the prediction of optimal thinning strategies for each branch. These algorithms significantly refine bud management, which is central to the DSS. It equips the system to provide targeted, data-driven guidance for effective bud-thinning practices.

Task 2.2: Development of decision support system (DSS) for bud thinning.

A decision support system (DSS), refined through field testing in selected orchards, is meticulously evaluated for its capacity to automatically determine which buds should be removed or retained for optimal early crop load management. This stage assesses the system’s real-world applicability, emphasizing the precision of its automated recommendations against manual decisions. The feedback loop with orchard managers and agronomists is crucial for enhancing the system’s usability and ensuring its recommendations align with practical orchard management needs. Comparative studies then measure the DSS’s influence on thinning practices, labor efficiency, and crop outcomes, using insights from these evaluations to fine-tune the system for broader deployment. This iterative refinement underscores the DSS’s capability to adapt to diverse orchard environments, showcasing its potential to revolutionize crop management by providing targeted, data-driven guidance on bud removal decisions.

Approach and Method 3: Design and Test a Robotic End-Effector for Effective Bud Removal for Early Apple Crop Load Management.

Task 3.1: Design of end-effector for apple bud thinning.

Understanding crop variability is essential to developing an end-effector for apple bud thinning because it determines the best way to remove buds based on their mechanical and physical properties. With this information, an end-effector can be created to remove undesired buds while carefully protecting the surrounding plant tissue and the surviving buds. Therefore, the effectiveness of the end-effector is measured not just by its precision in removing unwanted buds but also by its ability to protect the adjacent plant tissue and buds. This dual requirement emphasizes the delicate balance the end-effector must maintain, essential in optimizing early crop load management.

Task 3.2:  Design of robot manipulator for apple early crop load management.

A manipulator will be specifically engineered to navigate the unique architecture of apple orchards, enabling it to access apple buds effectively. Given the distinctive structural complexities of orchards, this task requires the development of a manipulator adept at maneuvering within these environments. Positioning tests will be conducted under laboratory and orchard conditions to validate its functionality, ensuring the manipulator’s performance is consistent across different settings. This manipulator will be integrated with a vision system and a decision-making algorithm. Before cutting action by the end-effector, the vision system and manipulator must work in unison to ensure precise positioning and identification of the targeted buds. This step is essential, as it guarantees the system’s effectiveness in selectively thinning buds, enhancing the overall precision and efficiency of the orchard management process.

Task 3.3: Integration of a robotic bud thinning system with the end-effector, machine vision system, and decision support system.

This task is dedicated to integrating the end-effector, vision system, and robotic manipulator to create a unified framework for managing apple orchards with precision and efficiency. Field tests to evaluate the performance and effectiveness of the developed precision system will be conducted at the research orchards of the Penn State Fruit Research and Extension Center (FREC). This integration is essential because it guarantees that the precise manipulator movements and precise bud detection by the vision system are synchronized with the end-effector’s cutting activities. These features work together to avoid damage to nearby branches and automatically target bud removal. Assembling a system that functions well as a whole, with each part supporting the others to offer the maximum accuracy and efficiency for the automated thinning process, is the first step towards effectively finishing this operation. The effects of the robotic thinning system on early crapload management will be evaluated, including bud detection accuracy, bud removal success rate, and the overall efficiency of the robotic system.