Development of a Robotic Harvesting system for Sustainable Button Mushroom Production

As mentioned in the introduction, the mushroom industry has a labor shortage. To tackle this challenge, an autonomous mushroom robot is proposed to increase not only the efficiency of production but also the quality of harvested mushrooms. Figure 1, indicating the different states of the project and depicting the approach to picking mushrooms.

Figure 1. The illustration of picking mushrooms automatically.

Approach and Methods 1: Prepare a mushroom dataset and develop deep learning algorithms utilizing CNN models to detect mushrooms, assess their maturity through RGB-D imagery, and analyze their spatial arrangements.

To accomplish this objective, several interrelated tasks were undertaken, including mushroom dataset preparation; training convolutional neural network (CNN) models using RGB-D data; extraction of 2D and 3D features; and mushroom maturity assessment based on 2D pixel information and 3D geometric features, including cap diameter and surface curvature. These tasks are outlined as follows:

Task 1.1: The dataset preparation and image processing techniques utilized for 2D applications.

Button mushrooms were spawned and grown in tubs at the Mushroom Research Center (MRC) at Pennsylvania State University (University Park, PA). From May 15 to 25, 2024, images were automatically captured every two hours using two ZED X Mini Stereo cameras (StereoLabs, San Francisco, USA), each with a horizontal field of view (HFOV) of up to 110^o, a vertical field of view (VFOV) of up to 80^o, and a 2.2 mm focal length lens. The cameras were securely mounted 26 cm above two mushroom-growing tubs, facing directly downward (Figure 2-A) under uniform lighting conditions. The images were then uploaded to OneDrive cloud storage after each capture. An aggregate of 165 RGB-D images was obtained with a resolution of 1920×1200 and resized for training and testing to 640×640. Images were uploaded to the Roboflow website (Des Moines, Iowa) and annotated around 23,000 mushrooms using bounding boxes (Figure 2-B) for YOLOv5s format.

The dataset was randomly split into training, validation, and test datasets with a 70/20/10 ratio. The training set consisted of 115 images. However, by applying the Roboflow augmentation feature to the original images, such as horizontal flipping and noise addition, the number of images increased to 365.

(A)	(B)

Figure 2. (A) Imaging setup for dataset collection. The ZED X Mini stereo camera is mounted above the mushroom growing tubs in a top-down orientation., (B) Annotated button mushroom (Agaricus bisporus) mushroom image.

The training and testing were conducted using the YOLOv5s model with Python on a computing platform equipped with an NVIDIA RTX A2000 12 GB graphics card, a 12th Gen Intel(R) Core ™ i9-12900K CPU, and 128 GB of RAM, running a 64-bit Windows 10 OS and the PyTorch deep learning framework. The hyperparameters specified are 300 epochs and a batch size of 16. The training was initiated using COCO-pretrained weights (YOLOv5s.pt). The optimizer used was stochastic gradient descent (SGD) with a learning rate of 0.01 and a weight decay of 0.0005. Data augmentation techniques included HSV adjustments (hsv_h = 0.015, hsv_s = 0.7, hsv_v = 0.4), geometric transformations such as translation (0.1), scaling (0.5), and horizontal flipping (fliplr = 0.5), as well as mosaic augmentation. The evaluation metrics for this training are precision (P), recall (R), and mean average precision (mAP) with the following equations:

Where TP, FP, and FN are the number of true positives, false positives, and false negatives.

Task 1.2: Maturity assessment using the cap size in 2D images.

The mushroom cap curvature can indicate the mushroom maturity stage (Lee et al., 2020). However, this study utilized mushroom cap size as a proxy for maturity, with a possibility of picking slightly early or marginally late. Normally, a mature button mushroom has a 2-5 cm cap diameter (Zied & Arturo, 2017). To identify maturity, the mushroom cap size was estimated using a printed checkerboard with a 2×2 cm grid cell size. Images were captured at 11 cm to 35 cm camera height to the checkerboard with a 3 cm increment, and the pixels in multiple grid cells were counted. Therefore, a mushroom size can be estimated by calculating the number of pixels at the mushroom cap diameter and the camera’s height. A pixel threshold was set to differentiate mature and immature mushrooms. Figure 3 illustrates the checkerboard imaging setup.

Figure 3. Checkerboard set up to count the number of 2×2 cm grid cell pixels in different camera heights to identify the pixel-based maturity threshold.

Task 1.3: Experiment samples for the bending distance in 2D

According to Huang et al. (2021a), the mushroom detachment angle for bending motion is 13.6 ± 6.7^o. To achieve the bending distance (i.e., the shifted distance when a mushroom is bent to detach), a series of mushrooms with a variety of cap sizes and total heights were picked and measured (Figure 4-A). In addition, Figure 4-B presents the geometric schematic used to estimate the bending distance. Mushrooms were modeled as rigid during bending, where bending occurs about a fixed pivot at the stem-casing interface. The mushroom caps were approximated as circular, and the stems were straight. The following equation illustrates the bending distance based on the mushroom height and detachment angle:

where b is the bending distance,  h is the mushroom total height, θ is the detachment angle, and α is the internal angle formed in the bending triangle. However, since measuring the cap diameter (d) is more practical than estimating mushroom height from 2D image analysis, a proportional relationship, referred to as the bending coefficient, between bending distance and cap diameter was established. The bending coefficient is defined as:

Therefore, the bending distance can be estimated by multiplying the bending coefficient and the cap diameter:

where d is the mushroom cap diameter and k is the bending coefficient. Table 1 shows the basic parameters of the test, which were taken into consideration for geometric calculations of the bending distance.

(A)	(B)

Figure 4. (A) Cap diameter, total height, and bending distance measurements of the mushroom, (B) Geometric schematic of the bending distance

Table 1. Basic information about tested mushrooms. Values are mean ± standard deviation.

Task 1.4: Spatial arrangement of mature mushrooms

The YOLOv5s model not only detected the mushrooms but also provided a text file for each image, consisting of the location of the bounding boxes and their dimensions. This information corresponded to each mushroom image; the x and y locations of the center of each individual mushroom and their diameter (minimum value between width and height) were extracted. However, some small mushrooms were detected by the YOLOv5s model and formed noise. This noise increased the algorithm computation time. Therefore, a pixel-based diameter threshold was set to filter out irrelevant small mushrooms, and another pixel-based maturity threshold was applied to single out mature mushrooms. Subsequently, for each mature mushroom, the algorithm considered a hypothetical circle with a radius of 1.5 multiply the mushroom diameter to identify all possible neighboring mushrooms (Figure 5-A). The choice of a 1.5 radius stems from the fact that identifying neighboring mushrooms is based on the coordination of their points. If the radius of the hypothetical circle is too small and the neighboring mushrooms have large caps, the algorithm may fail to recognize them as vicinity. Nevertheless, their caps are still present and can lead to errors in the bending process (Mushroom A). However, some small distant mushrooms were also detected as vicinity mushrooms under this circumstance. To address this problem, the center-to-center distance between the targeted mushroom and each vicinity mushroom was calculated and subtracted from the radii of both mushrooms. As a result, the mushrooms edge-to-edge distance was measured and compared with the bending distance threshold. When the edge-to-edge distance was less than the specified value, the mushroom was considered as vicinity. Otherwise, it was excluded from the vicinity mushrooms (Mushroom B). The angular arrangement of the vicinity mushrooms was conducted concerning the targeted mushroom. The angles were identified through the tangent inverse of the angle  (Figure 5-B).

where (x,y)  is the center point of the targeted mushroom, (x',y')  is the center point of the vicinity mushroom, and θ is the angle of the vicinity mushroom regarding the x-axis. Figure 5-C indicates the angular locations of the vicinity mushrooms positioned around the targeted mushroom. The information on all mushrooms in the vicinity was then added to a list for each targeted mushroom.

(A)	(B)	(C)

Figure 5. Vicinity mushrooms detection and angular arrangement, (A) targeted mushroom with a hypothetical circle with a radius of 1.5 multiply its diameter to determine the neighboring mushrooms with less than edge-to-edge distance threshold (mushroom A), (B) to calculate the angle of vicinity mushroom in respect to the targeted mushroom and x-axis, (C) the angular arrangement of vicinity mushrooms regarding the targeted mushroom.

Task 1.5: The dataset preparation and image processing techniques utilized for instance segmentation in 3D applications.

The dataset prepared for 2D applications was also utilized for 3D applications, with minor modifications to the CNN training procedure. A collection of 188 RGB-D images at a resolution of 1920×1200 was acquired and subsequently resized to 800×800 for training and testing. Images were uploaded to the Roboflow platform (Des Moines, Iowa), and annotations were generated using the Smart Polygon feature powered by the Segment Anything Model (SAM) AI model. The dataset was randomly partitioned into train, validation, and test sets at a 70/20/10 ratio. The train set comprised 126 images, which were augmented to 378 images through the implementation of Roboflow augmentation features, including flipping and noise addition.

Training and testing were conducted utilizing the YOLOv8m instance segmentation model with Python on a computing platform equipped with an NVIDIA RTX A2000 12 GB graphics card, a 12th Gen Intel(R) Core ™ i9-12900K CPU, and 128 GB of RAM, operating under a 64-bit Windows 10 operating system and the PyTorch deep learning framework. The specified hyperparameters encompass 120 epochs, a batch size of 16, and an image size of 800. The training process commenced utilizing COCO-seg pretrained weights (yolov8m-seg.pt). The optimizer employed was stochastic gradient descent (SGD), configured with a learning rate of 0.01 and a weight decay of 0.0005. Data augmentation techniques included HSV adjustments (hsv_h = 0.015, hsv_s = 0.7, hsv_v = 0.4), geometric transformations such as translation (0.1), scaling (0.5), and horizontal flipping (fliplr = 0.5), as well as mosaic augmentation. The metrics for evaluating training performance include precision (P), recall (R), and mean average precision (mAP), as outlined in task 1.1.

Task 1.6: Post-processing of mushroom cap segmentation masks and point cloud data (PCD) extraction

A mask for each individual mushroom cap was detected by the YOLOv8m instance segmentation model and subsequently overlaid on the captured image to extract point cloud data (PCD) from different mushroom tubs (Figure 6-A). However, the detected masks may have exhibited coarse and jagged boundaries, along with overlapping and noisy regions. This is attributable to discrepancies between the training and inference image resolutions, the up-sampling process of binary masks, and the quantized polygon representation. Therefore, it was necessary to apply post-processing to smooth boundaries using morphological operations, such as erosion and dilation, and to remove mask overlaps and noise (Figure 6-B).

(A)	(B)

Figure 6. (A) Detected mushroom cap masks with red overlaps and noise, (B) Smoothed mushroom cap masks with overlap and noise removal.

Each individual 2D mask ([Height, Width]) was converted into a 4D tensor ([Batch, Channel, Height, Width]) to conform with the input format required by the PyTorch interpolation function. Both Batch and Channel dimensions were set to 1, as masks were processed sequentially and each pixel contained a single label. Given that the output masks possessed a lower resolution compared to the original image, they were resized to the original image size using nearest-neighbor interpolation to ensure the preservation of discrete mask values. The resized masks were employed to determine the pixel coordinates associated with each instance ID. These coordinates were subsequently mapped to the corresponding 3D coordinates (voxels) of the PCD. Instances with invalid values were excluded from subsequent analysis.

Task 1.7. PCD depth accuracy evaluation and noise removal

To evaluate the depth accuracy of the extracted point cloud, a checkerboard was positioned at five different locations relative to the camera on a flat surface with a consistent vertical offset of 45cm. One image was collected at each position. For each captured image, the distances from the center of each of the five designated checkerboard cells to the camera were measured manually utilizing a measuring tape and subsequently compared with the corresponding depth values estimated by the algorithm. The root mean square error (RMSE) was then computed between the measured and estimated values to quantify the point cloud depth accuracy.

To remove irrelevant voxels from PCD, a series of post-processing procedures was implemented. Initially, an iterative RANSAC (RANdom Sample Consensus) algorithm was utilized to assist in identifying the geometric shape of mushroom caps. RANSAC randomly sampled data points, fitted a model, and assessed its performance based on the number of inliers. The model was iteratively refined to maximize the number of inliers, resulting in a representation of the mushroom caps surface. Following RANSAC, a statistical outlier removal (SOR) method was employed to eliminate isolated voxels deviating from their local neighborhood. This method calculated the mean distance of each point to its nearest neighbors and identified and removed points whose distance exceeded the global mean and standard deviation.

In instances where mask misalignment resulted from interpolation, alongside false-positive (FP) detections, residual noise—such as curved or planar shapes—persisted despite the utilization of RANSAC and SOR methods. To mitigate these issues, the DBSCAN (Density-Based Spatial Clustering of Applications with Noise) algorithm was employed to identify and eliminate low-density noise regions. Ultimately, the PCD was down-sampled to decrease computational costs and improve processing efficiency.

Task 1.8. Mushroom maturity assessment employing mushroom cap diameter and surface curvature measures

Mushroom cap curvature serves as an indicator of its maturity stage. Mature mushroom caps tend to be flatter and more open (Lee, 2020). In addition, the typical diameter of a mature mushroom cap ranges from 2 to 5cm (Zied & Arturo, 2017). This study aimed to assess mushroom maturity by quantifying mushroom cap diameter and surface-curvature measures utilizing the principal curvatures, κ₁ and κ₂ . These represent the maximum and minimum values of all the normal curvatures (eigenvalues of the shape) at a specific point, where κ₁ = 1/r₁ and κ₂ = 1/r₂ (Figure 7). When κ₁ = κ₂, the point is umbilic, corresponding to a locally spherical or flat region (Callens & Zadpoor, 2018).

Figure 7. Principal curvatures κ₁ and κ₂ (Callens & Zadpoor, 2018)

To facilitate the calculation of the κ₁ , κ₂ ,  and the diameter of each mushroom cap, a local frame surrounding an anchor point  P was estimated. For a set of neighboring 3D points {p _i}_i=1^N, the local centroid μ and the corresponding 3 × 3 covariance matrix  were computed as:

where N is the number of points. The anchor point P coincides with the local centroid C providing a stable and symmetric reference. Principal component analysis (PCA) of  is obtained by solving the following equation:

which yields a set of eigenpairs (λ_j, ν_j) with λ₁ ≥ λ₂ ≥ λ₃. The eigenvectors establish an orthonormal local frame: two tangent directions t₁ and t₂ (the largest variances λ₁ , λ₂) and the surface normal n (smallest variance λ₃). Each neighbor p_i is projected from the global coordinate system to the local coordinate frame centered at the anchor point P:

The surface in the vicinity of the anchor point P can be approximated using a quadratic height function on the tangent plane at this point:

where a, b, c, d, e, f are coefficients obtained by least squares on {(x_i , y_i , z_i) }. To evaluate the curvature at the anchor point P, the local gradient equals zero. Therefore, the residual slope coefficients are d = e = 0. The principal curvatures are determined by the eigenvalues of the 2×2 Hessian matrix:

The combination of the principal curvatures produces different surface-curvature measures, including mean curvature H, curvedness C, and anisotropy r (Table 2).

Table 2. Surface-curvature measures produced by principal curvatures.

The mean curvature H and the magnitude of the curvedness C denote the extent to which the mushroom cap is domed. However, when the mushroom cap is deformed or partially occluded, one principal curvature collapses (κ₂-› 0), resulting in the visible mushroom cap appearing cylindrical. In such cases, H and C can be low even for an immature mushroom cap. A high value of anisotropy r flags these cases.

To determine the diameter of the mushroom cap, the distance between the anchor point P and each point in the mushroom PCD was calculated by projecting its translation vectors (P_i = p_i - μ) onto the local tangent plane by t₁ and t₂ (Eq. 10). The equivalent diameter of the mushroom cap was determined by doubling the robust 95%-percentile estimator of the surface points in order to mitigate the effects of noise and incomplete point coverage (Eq. 11):

where u_i and v_i are the projections of the translation vector on the t₁ and t₂ tangent plane, respectively, and r_i is the in-plane radial distance of each point from the anchor point, and D_i is the equivalent mushroom cap diameter.

To evaluate the accuracy of the estimated diameter of the mushroom cap, the algorithm measurements were compared with the actual diameters of 100 mushroom caps, which were measured using a caliper. In addition, the diameter and three surface curvature measures were computed for 117 mushroom caps, which were manually classified into two categories (40 immature and 77 mature) by a human annotator. The extracted measures were employed as inputs to a linear support vector machine (SVM) classifier, with a cost parameter C = 1.0, a 75/25 train-test split, and balanced class weighting. Precision, recall, and F1-score were employed to evaluate the classification performance. To minimize the computational cost of the pipeline and enhance mushroom maturity assessment precision, mushrooms with cap diameters exceeding 4 cm were directly classified as mature, whereas those with diameters less than 3 cm were classified as immature. The assessment of mushroom maturity was subsequently conducted using diameter-based decision rules in conjunction with an SVM classifier. This methodology was employed on a total of 576 mushroom caps, comprising 198 immature and 378 mature samples. A confusion matrix was constructed to evaluate the mushroom maturity assessment pipeline results, and precision, recall, and F1-score were employed to quantify its overall performance.

Approach and Methods 2: Develop a decision-making algorithm to assign the picking sequence and the bending direction of mushrooms for the picking process.

Task 2.1: Decision-making and bending direction strategies in 2D

As the first step of picking, the decision of bending direction for a mushroom was made with respect to the targeted mushroom spatial configuration (Figure 8). When the mushroom was isolated (Figure 8-A), the algorithm arbitrarily opted for one bending direction in the span of 0^o and 360^o. Yet, in the case of single-vicinity mushroom, the bending direction was 180^o opposite its angle of the vicinity mushroom (Figure 8-B). For multi-vicinity mushrooms with a broad free area on the side, the detachment direction was the bisector of the reflex angle between mushrooms (Figure 8-C).

(A)	(B)	(C)

Figure 8. Arrangements of vicinity mushrooms: (A) an isolated mushroom with a random bending direction, (B) the bending direction is opposite to the vicinity mushroom angular arrangement, (C) the bending direction is the bisector of the reflex angle between mushrooms.

When three or more vicinity mushrooms were distributed around the targeted mushroom, the area between mushrooms would be calculated later, provided the broadest middle angle was more than 90^o (Figure 9). To proceed, a block arch (required area for bending the mushroom) starting from one vicinity mushroom to another would be considered (blue and green areas). The width of the arc was equal to the bending distance. It should be highlighted that the arch had some intersections with the vicinity mushroom circles (blue area), and the algorithm was set up to verify if intersections were below a certain threshold. If that was the case, the area between mushrooms was determined to be sufficiently spacious for bending a mushroom. Subsequently, the bending direction would be the bisector of the middle angle between the vicinity mushrooms.

Figure 9. Illustration of a block arch with the bending distance width between the center points of vicinity mushrooms with over 90^o middle angle. If the aggregate intersections (blue area) were below a certain threshold, the area between mushrooms (green area) would be sufficient for bending mushrooms toward.

Task 2.2. Clustering and picking sequence in 2D.

To obtain the clustering distribution in the mushroom tubs, a DBSCAN (density-based spatial clustering for applications with noise) (Ester et al., 1996) was implemented to position mature mushroom clusters and controlled by the eps (ε) and minPts parameters. The eps (ε)  is the maximum distance between two points to be considered as neighbors, and the minPts is the minimum number of points required to form a dense region. In addition, as mushrooms have diverse cap sizes, the corresponding diameter for each mushroom was attributed as a weight to each point to adjust and fine-tune the clustering. Also, since the number of clusters varied from tub to tub, the performance of clustering was evaluated by implementing the silhouette coefficient metric with the following equation:

where S is the silhouette coefficient, X_i is the sample, a is cluster cohesion, b is cluster separation, and n is the number of samples. The range of the silhouette coefficient is between -1 and 1. A higher value of this coefficient is attributed to a better coherent cluster (Belyadi & Haghighat, 2021). To set the eps (ε) parameter, the Euclidean distance (d_k) from each mature mushroom center (x_i) to its 3rd-nearest neighbor (k = 3) was computed and the values {d_k(x_i)} were sorted to obtain the k-distance curve (Figure 10). The curve is flat for dense interior mushrooms and rises sharply (forming an elbow) for isolated mushrooms. The elbow  was estimated by the 95^th percentile of the sorted distances, εˆ=P₉₅({d_k}). It means about 95% of mushrooms have at least three neighbors within εˆ. The search for eps (ε) was limited to a band around the elbow εˆ:

The value δ was selected conservatively. For each mushroom tub, the minPts parameter was varied from 1 to 5, and eps (ε) within the band. The pair was selected based on the highest value of the silhouette coefficient. The average and standard deviation of the silhouette coefficient were calculated for 29 mushroom tubs with different numbers of mushrooms and arrangements. The clustering procedure involved partitioning mushrooms into separate clusters, structuring the ordering step, and enhancing the efficiency of decision-making and picking sequence. Subsequently, the algorithm followed the cluster orders based on the average Euclidean distance of each cluster from the origin to select mature mushrooms.

Figure 10. Example of k-distance curve for the 3rd-nearest neighbor for each mature mushroom point. The sharp increase (elbow) indicates the density break and is used to select the DBSCAN eps (ε) parameter with minPts = 4.

Following the mushrooms spatial arrangement in each cluster, the algorithm put targeted mushrooms in a sequence according to the Euclidean distance of the mushrooms to the origin (0, 0). So, mushrooms closer to the origin were set to be picked first (Figure 11). The picking sequence and number of vicinity mushrooms were updated right after a decision was made for a targeted mushroom. Therefore, the algorithm followed the picking sequence given that the previously identified mushroom no longer exists. Figure 12 illustrates the picking sequence and bending direction. In this cluster, seven mature mushrooms are identified and need to be picked in a sequence. It is expected that these mushrooms are picked without collisions with their neighbors.

Figure 11. Assigning of picking sequence in a cluster based on the Euclidean distance of the mature mushrooms to the origin (0,0).

Figure 12. The picking sequence and bending directions of mature mushrooms from image A to image H, respectively.

Task 2.3. 2D decision-making algorithm validation

Several mushrooms, according to the preceding vicinity mushroom arrangements (Figure 8), were selected to provide ground truth for the model. The number of evaluated mushrooms is represented and distinguished according to each arrangement in Table 3. Subsequently, mushrooms within different arrangements were fed to the detection and decision-making algorithms, respectively, to estimate the related bending directions and to what extent the algorithm could outline the correct directions in each arrangement.

Table 3. The number of tested mushrooms within different arrangements (illustrated in Figure 8).

The algorithm's decisions regarding bending direction and picking sequence performance for the entire mushroom bed needed to be evaluated to determine the mushroom-picking strategy. To establish this, 29 images were taken from 29 tubs, with each containing an average of 64 mature mushrooms. Subsequently, the mushrooms in the images were detected, and the picking strategies and bending direction were obtained from the decision-making algorithm for all tubs. To test the feasibility of the algorithm, 90 randomly selected mushrooms were picked by a human picker following the bending direction and picking sequence obtained from the algorithm.

Task 2.3: 3D bending direction determination

The detection of mushroom PCD, along with measurements of their cap diameter and an assessment of their maturity, has facilitated the development of the mushroom picking sequence and bending determination. To identify the vicinity arrangement surrounding each targeted mushroom, proximity thresholds were established in both horizontal (xy) plane and the vertical (z) direction. Initially, the center-to-center distance between two mushrooms i and j was calculated as:

Subsequently, the maximum permissible distance τ between their centers was established as follows:

where D_i and D_j represent the diameters of the two mushrooms, and  is a user-tunable proximity coefficient. When k = 1, the envelope represents idealized circular mushroom caps that only contact at their boundaries. However, due to partial occlusion of mushroom caps, their irregular shapes, and the diameter estimates derived from noisy PCD, a k>1 was employed to expand the envelope and enhance robustness against shape variability, measurement noise, and segmentation uncertainty. Furthermore, a marginally larger value of k permitted the algorithm to accommodate the bending distance of mushrooms and to prevent potential collisions during the bending action. A mushroom was classified as a vicinity if  d_ij ≤ τ_ij (Figure 13).

Figure 13. Schematic of vicinity mushroom detection on the horizontal plane. The mushrooms center-to-center distance d_ij was compared with the adaptive envelope using the maximum allowable distance threshold τ_ij, defined as τ_ij = k(r_i + r_j). 

To further evaluate the vertical threshold, the disparity between the height components of two mushrooms was calculated, normalized by a scaling constant z_scale, and compared with a user-defined maximum vertical gap z_{gap_max}. The z_scale compressed the vertical axis to mitigate the impact of slight height variation, because mushrooms primarily block each other laterally rather than vertically. Consequently, the vicinity region surrounding each mushroom was visualized as an oblate spheroid compressed along z axis and elongated within the xy plane.

However, there was a slight deviation between how the open-space region was identified in this 3D methodology. In 2D appraoch, the open-space region angle was exclusively determined based on the angular position of the center-to-center lines from the targeted mushroom to vicinity mushrooms A and B (i.e., the reflex angle between C_A and C_B). This study refined the calculation by incorporating the physical sizes of the vicinity mushrooms (r_A and r_B). As illustrated in Figure 14, the available open-space region was identified by taking the angular range between C_A and C_B, and subtracting the occupied regions defined by θ_A and θ_B. The bisector of this refined open-space region provided a more realistic and precise determination of the bending direction.

Figure 14. Refined calculation of the targeted mushroom bending direction by incorporating the sizes of the vicinity mushrooms (r_A and r_B) and their respective occupied angular spans (θ_A and θ_B).

Task 2.4. 3D picking sequence algorithm

Determining an appropriate picking sequence is a vital strategy for reducing the travel distance of the robotic end-effector during mushroom harvesting and enhancing the probability of successfully detaching each mushroom from the bed. In the proposed algorithm, once the PCD of the mature mushroom caps and their vicinity arrangements were identified, the mushrooms were grouped into clusters solely to structure the harvesting strategy.

Specifically, as illustrated in Figure 15, if mushroom A was within the vicinity of mushroom B, and B was within the vicinity of C, the algorithm classified them within the same cluster (Cluster 1), even in the absence of direct adjacency. In contrast, mushrooms separated by significant gaps, such as mushroom D, were treated as independent clusters (Cluster 2).

The algorithm identified the initial cluster for harvesting cluster (Cluster 1) by evaluating and selecting the shorter Euclidean distances to the centroid of the nearest mushroom in each cluster (mushrooms A and D). The mature mushrooms within the identified cluster were assigned a picking sequence according to their Euclidean distance from the origin and the available open-space region suitable for bending.

Figure 15. Example of grouping mushrooms into clusters based on the detection of the vicinity arrangement of mature mushrooms. The initial harvest cluster (Cluster 1) was selected according to the centroid of the mushroom with the closest Euclidean distance to the origin (mushrooms A). The green lines indicate mushroom vicinity arrangement; the red arrows denote the Euclidean distance to the origin. Cluster 1: Mushrooms A, B, C. Cluster 2: Mushroom D.

Task 2.5. 3D mushroom bending direction and picking sequence validation

To validate the determined bending directions and picking sequence by the developed pipeline, a total of 268 mushrooms within different vicinity arrangements were examined across three mushroom tubs: 37 in the isolated arrangement, 117 in the single-vicinity arrangement, and 114 in the multi-vicinity arrangement. Mushrooms were harvested according to the picking sequence determined by the pipeline. For isolated mushrooms, the bending was arbitrary, as the pipeline also determined a free direction.

For the other mushroom vicinity arrangements, wooden skewers were inserted into the center of each targeted mushroom and the corresponding vicinity mushrooms in order to physically represent their angular arrangements (Figure 16-A). For the single-vicinity arrangement, the skewers were placed into the middle slot of a corner-angle finder. The targeted mushroom was subsequently bent, while the skewer followed a direction opposite to that of the vicinity mushrooms towards the determined pipeline direction (Figure 16-B).

In the case of a multi-vicinity arrangement, the open-space region angle was determined utilizing a digital angle finder and subsequently compared with those determined by the algorithm. (Figure 16-C). During the bending process, the skewer of the targeted mushrooms was positioned in the central slot of the corner angle finder, while the skewers of the vicinity mushrooms were placed in the side components (Figure 16-D). This configuration ensured that the targeted mushroom was oriented toward the bisector of the measured reflex angle, thereby facilitating comparison with the algorithm-predicted bending direction.

Figure 16. Experimental setup for validating the algorithm-predicted bending direction. (A) Wooden skewer inserted into the center of the mushroom cap of the targeted mushroom and its vicinity mushrooms, (B) For single-vicinity arrangement, the wooden skewers inserted into the mushrooms were positioned within the central slot of a corner-angle finder. The targeted mushroom skewer was guided in the direction opposite to the vicinity mushroom, (C) Determination of the available open-space region angle using a digital angle finder for multi-vicinity arrangement, (D) For multi-vicinity arrangement, the targeted mushroom skewer is positioned in the central slot, while the vicinity mushroom skewers are placed against the side components of a corner-angle finder. This arrangement is intended to guide the central skewer toward the bisector of the determined open-space region angle.

The entire pipeline was applied to 378 mature mushrooms of varying sizes and arrangements to determine the bending direction and picking sequence for each mushroom cap. Additionally, the procedure aimed to evaluate the number of remaining unassigned mushrooms.

Approach and Method 3: Develop an electrical and control system to move the robot's manipulator and a soft end-effector and pick mushrooms by bending and twisting mechanisms.

This objective has two main sub-sections to accomplish.

Task 3.1: To develop a manipulator and soft end-effector for mushroom picking.

For this project, the SCARA manipulator is proposed to pick mushrooms (Figure 17). This mechanism has parallel-axis joints with six degrees of freedom (DoF), two linked arms, and human arms. The manipulator revolves around each joint to cover a vast area or mushroom tray, and the far-end ones do the twisting motion for picking mushrooms. Furthermore, the vertical motion happens at the end joint connected to the soft end-effector to make the picking motion. Also, for the bending motion, two different scenarios are suggested here. The first is to add another joint (stepper motor) at the far end to tilt the end-effector and bend the mushrooms, followed by twisting and lifting. This scenario guarantees bending even with mushrooms with different poses but adds complexity to the system and increases the number of DoF. The other scenario is to move the distant arm toward any direction and expect the end-effector to cause bending on mushrooms. This mechanism is more straightforward and faster. The performance of each should be investigated through experiments. In addition, the camera will be located at one of the arms between joints to have the best view over mushroom trays.

Figure 17. A SCARA manipulator (Torres-Del Carmen et al., 2020) that can be used for mushroom picking robot

The intended soft end-effector for this project is a vacuum cup. Figure 18 illustrates how the suction cup grips the mushroom cap to pick mushrooms. This cup is connected to a pump, and once it gets close to the targeted mushroom, it will stick to the caps to pick it up.

Figure 18. A soft end-effector is used to pick mushrooms by bending them (Huang, Jiang, et al., 2021).

An electronic system controls these systems. The information obtained by the machine vision section is sent to the controller to plan the path, move the manipulator, and locate the end-effector exactly above the targeted mushroom.

Task 3.2: To develop a stem trimming end-effector for harvested mushrooms.

Once the mushroom is picked, the soiled stem portion should be cut to remove the basal attachment of the casing soil. This task will be conducted using a stem-trimming system, which will be equipped with a photoelectric sensor and a knife. Once the harvested mushroom is held in front of the sensor using the vacuum cup, the knife is activated. The knife moves forward and trims the stem end. Finally, the end-effector will put the mushroom in the dedicated box.