Progress report for GNE24-338
Project Information
Agaricus bisporus, commonly known as button mushroom, should be harvested selectively by hand to preserve its high quality and delivered to the fresh market owing to its delicate skin, prone to bruise and drop in quality. However, the mushroom industry has confronted high costs and a shortage of skilled workers due to the labor-intensive environment of picking mushrooms. Studies have shown that harvesting mushrooms by bending and twisting individual mushrooms is effective in picking motions, as is the human harvesting approach. In the meantime, the pose and vicinity of each mushroom play crucial roles in picking them. This study aims to develop an autonomous robot to pick mushrooms using a vacuum cup-type end-effector with bending and twisting motion. Meanwhile, a machine vision system with a stereo camera will be developed for mushroom detection and decision-making on individual mushroom picking. Finally, a harvesting robot will be tested and evaluated by integrating the picking end-effector and the machine vision system. The outreach strategy includes the field demonstration of the developed robot to Pennsylvania growers and carrying out surveys, workshops, webinars, etc., to evaluate the current satisfaction and knowledge of the growers and educate them regarding the benefits of applying developed technology. The target of this proposal is to alleviate the labor shortage issue, which imposes high costs on the mushroom industry, and increase the quality and productivity of mushrooms.
The primary goal of this project is to develop a robotic harvesting system for picking button mushrooms in greenhouses. This research project will be focused on three objectives listed below:
- Develop image processing and deep learning algorithms using YOLOv CNN methods to identify the location, maturity, and estimate of mushrooms’ pose.
The coupled image processing and deep learning algorithms provide a robust system to detect and segment mushrooms from their background in real-time. Of different deep learning algorithms, we will be using YOLOv convolutional neural networks to accomplish this objective accurately. In the meantime, the system could find the individual mushroom’s pose in 2D and 3D, maturity, center point, and circle. This information will be sent to the
decision-making section to determine the next step.
- Develop a decision-making algorithm to assign the picking sequence and the bending direction of mushrooms for the picking process.
By acquiring information from the last stage to carry on the decision-making process, our developed algorithm will be able to identify the sequence of picking mushrooms using image processing techniques such as morphology characteristics. In addition, as the targeted mushrooms will be picked by bending and twisting, and because they grow in clusters, we will identify the bending direction so as not to injure nearby mushrooms.
- Develop an electrical and control system to move the robot's manipulator and a soft end-effector and pick mushrooms by bending and twisting mechanisms.
The collected information from the last objectives is fed to the control system to set the robot joints' rotation degrees, approaching orientation, plan to pick mushrooms, etc. As a result, the manipulation occurs, and a vacuum cup picks each mushroom, transfers it to the stem trimmer, and puts it in the
specified location.
The purpose of this project is to develop robotic harvesting to alleviate the labor shortage problem in the mushroom industry and pave the way for sustainable mushroom production. Pennsylvania produces around two-thirds of the Agaricus mushroom of the USA, about 408 out of 645 million pounds produced button mushrooms, a 945-million-dollar business, in 2022-23; of which 85% delivered to the fresh market and the remaining to processed production (USDA, 2023). However, historically, this industry produced approximately 827 million pounds (557 in Pennsylvania), of which 92% went to the fresh market in the 2018-19 season (USDA, 2019). Therefore, the stats show that mushroom production and fresh market supply experienced about 20% and 7% falls within the past four years, respectively. Also, the National Agricultural Statistics Service (NASS) reported that from 2008-09 to 2017-18, the number of growers in the US fell by about 20% (USDA, 2019). However, it gave up reporting as of the 2019-20 report (USDA, 2020). Other than that, the provided mushrooms for the fresh market are harvested selectively by hand due to the sensitivity of the mushrooms’ skin, longer shelf-life, appearance, and multiple harvesting flushes because of the different growing speeds of mushrooms. Therefore, high costs in the mushroom industry for skillful workers who work intensively are imposed because of the nature of picking mushrooms. In addition, there are problems with picking by workers, including the substantial skill of pickers to distinguish mature mushrooms appropriately, variation among pickers for harvesting mushrooms, health safety of workers, and constraints on laborers’ working hours, which generally comprise about 30% of mushroom production costs (Cunha Zied & Pardo-Giménez, 2017). In addition, based on the information provided by the USDA's Economic Research Service (ERS), the farm wage was substantially less than that of nonfarm in 2022, 40% lower, and farm growers report that finding workers was more difficult than usual (ERS, 2023). Hence, labor shortages are a significant issue in the mushroom industry. To address this, mechanical and robotic systems come into play.
Since the late 1970s, the introduction of mechanical mushroom harvesters resulted in higher efficiency, harvesting massively within minutes compared to manual harvesters taking days at the same crop area (Azoyam, 2004). Van den Top, a Dutch company, is an excellent example of a mechanical harvester cutting mushroom stems and transferring them with a conveyor roll (Vandentop, 2024). However, these types of harvesting require a specific mushroom growing shelf. Also, they pick mushrooms regardless of size and maturity; most are damaged and have low-quality products. To that end, the produced mushrooms are primarily suitable for processed consumption (Cunha Zied & Pardo-Giménez, 2017). For this reason, an accurate robotic system that can harvest mushrooms selectively is a better option to reduce production costs and maintain the high quality of mushrooms.
The first attempts to evaluate the viability of automatic picking mushrooms have been reported since the 1990s. The result showed that robotic harvesting for high-quality fresh market mushrooms is economically feasible and deducts between 15% and 30% of mushroom production costs (Cunha Zied & Pardo-Giménez, 2017). Lack of damage, contamination of casing soil, uniformity of size, and state of development define the high-quality mushrooms required by the fresh market. To meet these criteria, a robotic harvesting system should address some challenges, including identifying an appropriate size, location, grading, selecting, mechanism of picking so as not to damage or contaminate the mushroom or its neighbors, trimming the stem, and gently putting mushrooms into a container (Reed et al., 2001).
Tillett and Batchelor (1991) intended to characterize mushrooms visually and conducted the first image processing algorithms for mushroom detection and location identification (Tillett & Batchelor, 1991). However, Reed and Tillett conducted a preliminary robotic mushroom harvesting test in 1994. After localizing the mushrooms, they used their developed end-effector, a vacuum cup, to pick mushrooms. They discovered that bending as a detachment may enhance the picking performance compared to twisting (Reed, 1994). In 1997, Noble et al. built a more robust robot. They devised a sequence of bending, twisting, and picking mushrooms as a proper harvesting strategy, causing lower damage to mushrooms and increasing the picking success by over 80% (Noble et al., 1997). However, automatic mushroom harvesting systems have not gained popularity in the mushroom industry, and innovations have been released slowly (Cunha Zied & Pardo-Giménez, 2017).
Research
As mentioned in the introduction, the mushroom industry has a labor shortage. To tackle this challenge, an autonomous mushroom robot is proposed to increase not only the efficiency of production but also the quality of harvested mushrooms. Figure 1, indicating the different states of the project and depicting the approach to picking mushrooms.
Figure 1. The illustration of the picking mushroom automatically.
Approach and Methods 1: Develop image processing and deep learning algorithms using YOLOv CNN methods to identify the location, segment, maturity, and estimate mushrooms’ pose.
To accomplish this objective, we should perform four different tasks involving preparing the imaging structure, training the deep learning algorithm, feature extraction, such as mushroom center point and maturity, and 3D image processing. To begin with, button mushrooms (Agaricus bisporus) are spawned and grown in tubs at the Penn State Mushroom Research Center (MRC) (University Park, PA). The tasks are as follows:
Task 1.1: To develop a machine vision system for mushroom detection.
We will build a dismantled aluminum extrusion profile structure to mount the imaging and computing system and light source and put it precisely above the mushroom tubs at a specific distance. Owing to the margin between the mushroom growing bed and the shelf above about 40 cm, we selected a short-range camera with high accuracy in depth perception. Our imaging system comprised two ZED X Mini Stereo Cameras (StereoLabs*, San Francisco, USA) ranging from 0.1 to 8 m and a compatible computing unit, one ZED Box as a computing system (StereoLabs*, San Fransisco, USA) (Figure 2-a). This system automatically captured 165 RGB-D images over 11 days with tow-hour intervals. Images were uploaded to the OneDrive storage cloud with 1920x1200 resolution in real-time. Following that, images were annotated on the Roboflow website (Des Moines, Iowa, USA) using bounding boxes (Figure 2-b) and resized to 640x640 resolution for YOLOv5 format.
 |
 |
| (a) | (b) |
Figure 2. (a) Dataset preparation through capturing images from a mushroom tub, (b) Annotated Agaricus bisporus mushroom image.
The built dataset was divided into training (115 images), validation (33 images), and test (17 images) with a ratio of 70/20/10. Applying augmentation features such as flip and noise embedded in Roboflow increased the training dataset to 365 images. Training and testing were carried out in YOLOv5s with Python using a computing platform equipped with NVIDIA RTX A2000 12GB graphic card, 12th Gen Intel(R) Core ™ i9-12900K CPU, and 128GB RAM and run a 64-bit Windows 10 OS and PyTorch deep learning framework. The hyperparameters specified are 300 epochs, batch size 16. The evaluation metrics for this training are precision (P), recall (R), and mean average precision (mAP) with the following equations:
Where TP, FP, and FN are the number of true positives, false positives, and false negatives.
Task 1.2: To detect mushroom cap circle, center point, maturity, and pose estimation in 2D images.
Pixel-based maturity detection
Normally, the cap diameter of a mature button mushroom mushroom ranges from 2-5 cm. In this study, multiple images were captured from a checkerboard with 2x2 cm grid cells using a ZED X mini camera to detect size-based maturity. Starting from 11 cm, the camera height was incrementally increased by 3 cm up to 35 cm, with one image taken at each height. Grid cell pixels corresponding to each height were counted, representing the number of cap diameter pixels at different heights. Thus, a pixel threshold was set to differentiate mature and immature mushrooms. Figure 3 illustrates the checkerboard imaging setup.
Figure 3. Checkerboard setup to count the number of 2x2 cm grid cell pixels in different camera heights to identify the pixel-based maturity threshold
Also, in this task, images from individual mushrooms will first be taken in the lab with different orientations. We aim to detect mushroom stems and use deep learning and mushroom morphology to estimate each mushroom's pose.
Task 1.3: To identify the spatial location and pose estimation in 3D environment for each mushroom.
Experiment samples for bending distance
According to Huang et al. (2021), the mushroom detachment angle for bending motion is 13.6 ± 6.7 degrees. To achieve the bending distance (i.e., the shifted distance when a mushroom is bent to detach), a series of mushrooms with a variety of cap sizes and total heights were picked and measured (Figure 4). Table 1 shows the basic parameters of the test, which were taken into consideration for geometric calculations of the bending distance.
Figure 4. Cap diameter, total height, and bending distance measurements of mushrooms.
Table 1. Basic information about tested mushrooms. Values are mean ± standard deviation.
|
Total counts |
Cap diameter (mm) |
Total height (mm) |
|
39 |
43.2±4.8 |
38.3±3.4 |
In addition to this, the stereo camera will provide RGB-D and point cloud images. Integrating 2D images and depth information gives spatial information regarding individual mushrooms, which will be sent to the electronic system to pick assigned mushrooms. In addition, point cloud images also provide voxels (pixels in 3D images). There are several techniques to remove noises in point cloud images, such as RANSAC (RANdom Sample Consensus). Then, using some segmentation techniques in 3D, such as a watershed transform or Faster R-CNN, can help us detect each mushroom cloud. Finally, by calculating the average of eigenvectors of each voxel corresponding to each mushroom cap, we can detect the pose of each mushroom.
Approach and Methods 2: Develop a decision-making algorithm to assign the picking sequence and the bending direction of mushrooms for the picking process.
This objective has one main task to meet.
Task 2.1: To identify the sequence of picking mushrooms and bending direction.
Button mushrooms usually grow in clusters with different growing rates. For this reason, picking mature mushrooms should be done selectively through bending, twisting, and lifting. However, all mushrooms cannot be arbitrarily harvested with these motions. There are three scenarios: (i) there is no mushroom adjacent to the targeted mushroom so that it can be bent in either direction; (ii) mature mushrooms are in a dense cluster, and they must be picked through a specific picking sequence by bending and twisting (Figure 5 (a)) (orange, blue, and green mushrooms respectively), (iii) immature mushrooms surround the targeted mushrooms and must be picked up upward through solely twisting (Figure 5 (b)) (green mushroom).
We suggest analyzing mushrooms' spatial arrangement and morphology obtained in the first objective to do the task. Given that the algorithm starts from a specific location, we could identify the closest mushrooms to the center or the edge of each mushroom in each image. Then, the mushrooms with the most vacant space could be bent and twisted against nearby mushrooms. In contrast, if the mushroom is mature with no available space around it, it will only be twisted and lifted at last. This algorithm updates after picking some mushrooms to renew the decisions that have been made.
 |
 |
| (a) |
(b) |
Figure 5. The sequence of picking mushrooms in clusters using bending and/or twisting motions (a) to pick mushrooms in green, the nearby mushrooms must be picked in order of orange and blue, (b) only twist motion must be executed to pick green mushroom bordered by orange immature ones.
Approach and Method 3: Develop an electrical and control system to move the robot's manipulator and a soft end-effector and pick mushrooms by bending and twisting mechanisms.
This objective has two main sub-sections to accomplish.
Task 3.1: To develop a manipulator and soft end-effector for mushroom picking.
For this project, the SCARA manipulator is proposed to pick mushrooms (Figure 6). This mechanism has parallel-axis joints with six degrees of freedom (DoF), two linked arms, and human arms. The manipulator revolves around each joint to cover a vast area or mushroom tray, and the far-end ones do the twisting motion for picking mushrooms. Furthermore, the vertical motion happens at the end joint connected to the soft end-effector to make the picking motion. Also, for the bending motion, two different scenarios are suggested here. The first is to add another joint (stepper motor) at the far end to tilt the end-effector and bend the mushrooms, followed by twisting and lifting. This scenario guarantees bending even with mushrooms with different poses but adds complexity to the system and increases the number of DoF. The other scenario is to move the distant arm toward any direction and expect the end-effector to cause bending on mushrooms. This mechanism is more straightforward and faster. The performance of each should be investigated through experiments. In addition, the camera will be located at one of the arms between joints to have the best view over mushroom trays.
Figure 6. A SCARA manipulator (Torres-Del Carmen et al., 2020) that can be used for mushroom picking robot
The intended soft end-effector for this project is a vacuum cup. Figure 7 illustrates how the suction cup grips the mushroom cap to pick mushrooms. This cup is connected to a pump, and once it gets close to the targeted mushroom, it will stick to the caps to pick it up.
Figure 7. A soft end-effector is used to pick mushrooms by bending them (Huang, Jiang, et al., 2021).
An electronic system controls these systems. The information obtained by the machine vision section is sent to the controller to plan the path, move the manipulator, and locate the end-effector exactly above the targeted mushroom.
Task 3.2: To develop a stem trimming end-effector for harvested mushrooms.
Once the mushroom is picked, the soiled stem portion should be cut to remove the basal attachment of the casing soil. This task will be conducted using a stem-trimming system, which will be equipped with a photoelectric sensor and a knife. Once the harvested mushroom is held in front of the sensor using the vacuum cup, the knife is activated. The knife moves forward and trims the stem end. Finally, the end-effector will put the mushroom in the dedicated box.
Training result analysis
Following the section presented in the past, the performance of the YOLOv5s model for Agaricus bisporus mushroom detection was evaluated through standard metrics of precision, recall, and mean average precision (mAP) for 300 epochs. The recorded recall and precision were 94% and 92%, respectively. Also, the mAP was 97% at an IoU threshold of 0.5 (Table 2).
Table 2. YOLOv5s training result
|
Network model |
Precision |
Recall |
Mean Average Precision (mAP) |
|
YOLOv5s |
92% |
94% |
97% |
The results indicate that this model effectively detected mushrooms of different sizes in mushroom tubs, similar to the results of Chen, et al. (2023). The model was able to detect most mushrooms partially obstructed by others in clusters and discern stems from mushroom caps. Consequently, the outcome of this model can be proclaimed reliable for further experiments.
Maturity detection assessment
Upon image collection from different camera heights, the pixel numbers were counted for 2-cm grid cells, and the graph indicated in Figure 8 was concluded. The corresponding equation to the graph is as follows:
P = -80.1 ln(H) + 322.97 Eqn. 4
Where P represents the number of pixels for 2 cm, H camera height; thus, through understanding the camera height, the number of calculated pixels represents the specific maturity threshold for the mushroom. Moreover, the calculated R2-score was 97%. The farther the checkerboard was from the camera, the fewer pixels were in the grid cell.
Figure 8. The relationship between the number of pixels for a 2-cm-grid-cell checkerboard vs. ZED X mini camera height.
The obtained R2-score represents the effectiveness and reliability of the method. In addition, according to Lee (2020), mushroom cap curvature estimation using point clouds is another pivotal maturity indicator. Pairing 2D and 3D information from stereo cameras might improve mushroom detection and maturity estimation. In addition, owing to the camera’s lens distortion, the mushrooms near the image edges underwent a mild curve. Therefore, the maturity estimation of border mushrooms may have contained some discrepancies.
Bending distance result
Bending distance is a pivoting facet of Agaricus bisporus mushrooms for the feasibility of harvesting. To determine this distance, the cap size and height of 39 mushrooms were measured, and the geometric calculations were conducted for the longest bending distance of the lowest and highest detachment angles, 6.9 and 20.7 degrees, respectively. The results revealed that the mushroom bending distance hovers around 0.21±0.09 x the mushroom’s cap diameter, which is enough distance for bending to ensure sufficient space for mushrooms to be detached. Additionally, as the pose of mushrooms might vary during their growth stage (Baisa & Al-Diri, 2022; Retsinas et al., 2023), estimating their pose can fine-tune the estimation of the bending distance. Table 3 presents examples of bending distance for mushrooms with variable cap diameter and height.
Table 3. Examples of the bending distance for different detachment angles
|
Cap diameter (mm) |
Height (mm) |
Detachment angle |
Bending distance (mm) |
Fraction of cap diameter |
| 44 | 42 |
6.9 |
5 |
0.11 |
|
13.6 |
10 |
0.22 |
||
|
20.7 |
15 |
0.33 |
||
| 40.5 | 36 |
6.9 |
4 |
0.10 |
|
13.6 |
8 |
0.21 |
||
|
20.7 |
13 |
0.31 |
||
| 37 | 35 |
6.9 |
4 |
0.11 |
|
13.6 |
8 |
0.22 |
||
|
20.7 |
12 |
0.33 |
||
| 47 | 37 |
6.9 |
4 |
0.09 |
|
13.6 |
9 |
0.19 |
||
|
20.7 |
13 |
0.27 |
||
| 51 | 43 |
6.9 |
5 |
0.10 |
|
13.6 |
10 |
0.20 |
||
|
20.7 |
15 |
0.30 |
||
|
|
Average |
0.21±0.09 |
Education & Outreach Activities and Participation Summary
Participation Summary:
This study's outreach program will improve farmers' performance in mushroom production by utilizing robotic solutions. This system brings many economic benefits for mushroom growers because they can tackle the problem of labor shortages, especially skilled workers. Also, it is advantageous for the workers because of the less intensive work and the more health benefits. As a result, this industry will gain advantages from sustainable production with high quality and efficiency.
Plan #1: Introduction of the mushroom harvesting robot to Pennsylvania growers
An article introducing the robotic harvesting system for button mushrooms will be published in the extension article, renowned conferences, and refereed journals. These professional newsletters are widely distributed to mushroom growers in Pennsylvania and surrounding states and will help generate interest in this system before the demonstration field day. The article will highlight the technology that distinguishes this system from other “mushroom-picking” technologies on the market and review the results of research conducted in other countries using these systems. Inviting growers and stakeholders to attend the field demonstration event will also be included.
Plan #2: Assess growers’ knowledge about the robotic harvesting system.
Kennett Square, Pennsylvania, is known as the “Mushroom capital of the world” because more than half of the United States' mushrooms are produced here. Therefore, we will assess growers' satisfaction with the current situation and evaluate the growers’ community knowledge about the novelty of mushroom harvesting robots. To do so, there will be a survey to collect and understand their familiarity with technology and educational demands, which will be addressed by demonstration and workshop.
Plan #3: Demonstration, workshop, and post-research adoption survey.
The field demonstration will address the following four issues: (i) System design: What components are needed to evaluate and facilitate the robotic harvesting system, adjust the robot's location, and replace components with the existing equipment? (ii) Ease of use: How can the system be maintained, and how can components be replaced as necessary? (iii) Management of in-field operations: How does the system adapt to terrain and various mushroom clusters? (4) Operator requirements: What knowledge is required to operate the system effectively?
The workshop will address the issues listed above in greater detail and will include a detailed analysis of the robotic mushroom-picking results. It will also provide a simple toolkit for evaluating the costs associated with investing in the system and the return on investment associated with a robotic harvesting system with reduced worker costs and improved mushroom quality. Materials produced for the workshop will be available online, including the economic evaluation toolkit, tutorials on configuring, operating, and calibrating the system, videos of system operation, safety, and maintenance, and all publications from the research activities.