| Citation: |
Ji J T, Du S C, Li M S, Zhu X F, Zhao K X, Zhang S H, et al. Design and experiment of a picking robot for Agaricus bisporus based on machine vision. Int J Agric & Biol Eng, 2024; 17(3): 1–10. DOI: 10.25165/j.ijabe.20241703.7740
|
Harvesting represents the crucial stage in the cultivation process of Agaricus bisporus mushrooms. An important way for the production process of Agaricus bisporus to reduce costs and increase income is to ensure timely harvest of Agaricus bisporus, reduce harvesting costs, and improve harvesting efficiency. There are many disadvantages in manual picking, such as high labor intensity, time-consuming work and high cost. In this study, a set of mushroom picking platform including climbing mechanism, picking robot, and control system was designed and developed. The picking robot consisted of a truss mechanism, an image acquisition device, a mushroom collection device, and a picking actuator. The profile picking actuator could realize the function of constant force clamping. An online size detection algorithm for Agaricus bisporus based on deep image processing was proposed. The algorithm included removal of abnormal noise points, background segmentation, coordinate conversion, and diameter detection. The precision picking system for Agaricus bisporus with coordinate compensation function controlled by Industrial Personal Computer was designed, and the visual control interface was developed based on Labview. Through the performance test, the reliability of machine vision recognition and the overall operating stability of the picking platform were verified. The test results showed that in the process of machine vision recognition, the recognition accuracy rate was higher than 92.50%, the missed detection rate was lower than 4.95%, the false detection rate was lower than 2.15%, and the diameter measurement error was less than 4.50%. The image processing algorithm had high recognition rate and small diameter measurement error, which could meet the requirements of picking operation. The picking platform’s picking success rate was higher than 95.45%, the picking damage rate was lower than 3.57%, and the picking output rate was higher than 87.09%. Compared with manual picking, the recognition accuracy rate of the picking platform was increased by 6.70%, the picking output rate was increased by 1.51%. The overall performance of the picking platform was stable and practical.
Agaricus bisporus is the most extensive artificial cultivation, the highest yield, one of the largest consumption of edible fungi in the world. It is rich in protein and easy to be absorbed by the body of carbohydrates[1-3]. In recent years, with the gradual improvement of people’s living standards and scientific dietary awareness, the market development potential of Agaricus bisporus has been continuously enhanced. And Agaricus bisporus large-scale factory production has become the inevitable trend in the future[4,5]. It is an important way to reduce cost and increase income to ensure timely harvesting of Agaricus bisporus, reduce harvesting cost and improve harvesting efficiency[6,7]. Currently, the primary method employed for harvesting Agaricus bisporus mushrooms is through manual picking, which is characterized by high labor intensity, time-consuming processes, and substantial costs. Consequently, the research and development of a specialized Agaricus bisporus picking robot is of paramount importance as it aims to alleviate the workforce, enhance labor productivity, and diminish production expenses[8,9].
Machine vision is the key link in the process of picking robot. In recent years, with the development of machine vision technology, researchers have done a lot of research work in the detection of Agaricus bisporus. The algorithm developed by Tillett et al.[10] realized the location and size detection of Agaricus bisporus in the mushroom bed, but the success rate of the algorithm was low. Yu et al.[11,12] used the sequential scanning algorithm based region labeling technology of Agaricus bisporus image to realize the recognition of each Agaricus bisporus central region in the image. Yang et al.[13] proposed a background filtering algorithm with Harris corner as texture features. The watershed algorithm based on markings was used to achieve the segmentation of Agaricus bisporus, and ellipse fitting was used to achieve the positioning of the boundary and central coordinates of Agaricus bisporus. Wang et al.[14,15] obtained the diameter of bisporus mushroom by using the minimum external rectangle method after removing image shadow and interference from the stalk of Agaricus bisporus through the two-watershed algorithm. Wang et al.[16] used the background segmentation method based on depth image to segment the adhered mushrooms, and measured the position, diameter and inclination angle of the mushrooms in the world coordinate system. For the combination of machine vision technology for Agaricus bisporus picking robot development, related personnel have also made a preliminary attempt. The picking robot developed by Hu et al.[17-19] adopts the mode of mutual cooperation between visual area, picking area and auxiliary area. The vision and picking work in parallel, and the working principle of “three steps” was adopted to complete the picking task of Agaricus bisporus. The picking platform designed by Reed et al.[20] utilized machine vision and image analysis algorithm to realize automatic positioning, selection, picking, pruning, transportation and transfer of Agaricus bisporus. The existing end effector of the Agaricus bisporus picking robot mainly adopts the air suction type, which is difficult to achieve precise control and easily damages the mushrooms.
The factory production of Agaricus bisporus is in urgent need of automatic picking robot. In order to solve this problem, based on machine vision technology, design and develop picking actuators, was proposed based on the tidal “water law” of the principle of matrix background segmentation method, realized the Agaricus bisporus diameter precision of in situ test, design a constant force clamping the copying of picking actuators, and develop the truss type automatic picking robots, achieve high efficiency and low loss picking of Agaricus bisporus and collection.
The Agaricus bisporus picking platform is mainly composed of climbing device, picking robot and control system. The overall structure is shown in Figure 1. The climbing device equipped with a picker is placed at the mushroom outlet of the mushroom rack, allowing it to be raised and lowered to the mushroom beds on each level for work and easy to collect twin mushroom. U-shaped rails are arranged on the Mushroom rack. The picking robot travels on the track and carries out picking operations after arriving at the picking area. The control system’s hardware is housed within a distribution box, which is mounted onto the picking robot’s truss framework. The main structure of the picking platform is constructed with aluminum profiles to achieve convenient assembly and lightweight design.
The climbing device consists of supporting frame, climbing platform and lead screw, etc. The main function of the device is to transport the picking robot to different layers of mushroom racks to complete the picking task, and to transport the fully loaded picking robot to the designated position to replace the Agaricus bisporus collecting frame. The leading screw is connected with the climbing platform, and the stepping motor drives the leading screw to move, so that the climbing platform carries the picking robot to move up and down. After reaching the set height, the picking robot moves forward and enters the corresponding mushroom shelf layer to complete the picking task. The climbing platform is equipped with a U-shaped track, which is docked with the U-shaped track on the mushroom rack to facilitate the smooth transition of the picking robot from the climbing mechanism to the mushroom rack.
The picking robot has a rectangular structure, with a long side of 1.4 m and a wide side of 1.1 m. The picking robot is composed of truss walking mechanism, image collecting device, mushroom collecting device and picking actuator. Its structure is shown in Figure 2.
The truss type walking mechanism can realize X, Y and Z three-axis traveling in the truss, and complete the positioning operation of Agaricus bisporus in the truss area. The sliding guide rail is installed on both sides of the wide side of the picking robot to guide the beam to move along the Y-axis in the truss area. The mechanical arm is installed vertically above the beam, and is connected with the sliding guide rail to realize the X-axis movement of the mechanical arm in the truss area. X and Y axis are driven by stepper motor to drive the synchronous pulley, which drives the slider to move. The mechanical arm comprises a lead screw, a slider, and a stationary fixture. The lead screw and a slider cooperate to drive the picking actuator connected with the slider to move up and down, so as to realize the Z-axis travel of the picking actuator. During the picking operation, the picking area surrounded by the truss is detected and picked row by row along the X and Y axes.
The hardware of image acquisition device is mainly Inter Realsense SR300 depth camera (Intel Corporation, California, America) and connectors. The SR300 depth camera integrates depth information sensor and visible light sensor, which can simultaneously capture RGB image and depth image of the object. The maximum imaging resolution is 1920×1080, and the imaging distance is 0.11-10.00 m. The camera is a structured light depth camera, which is not affected by illumination and object texture. It can meet the requirements of image acquisition in weak light environment of Agaricus bisporus room.
The mushroom collection device comprises Agaricus bisporus collection box and an open-close collection box replacement mechanism. The mushroom collecting device is installed on the truss of picking robot, which moves with the picking robot to facilitate the integration of harvesting and improve harvesting efficiency. After the picking actuator picks up the Agaricus bisporus and rises, the controller sends instructions to control the picking actuator to move to the top of the Agaricus bisporus collection box, place the collected Agaricus bisporus in turn, complete the collection work of Agaricus bisporus, and the picking actuator returns to the previous image detection position to continue working. After the mushroom collection box is filled, the picking robot returns to the designated position, the open-close collection box changes the mechanism to work, the controller controls the movement of the electric cylinder, the mushroom collection box is separated from the picking robot, after the replacement of the new collection box, the picking robot returns to the last picking position to continue to work.
The stalk of the mushroom grows on the fungous materials, pulling out will take up a large number of fungous materials directly. It will not only increase the cleaning work, but also affect the normal growth of surrounding mushrooms. By driving the picking actuator to rotate at a certain angle through the steering gear, the mushroom body and the fungous materials can be separated quickly, which can effectively avoid the above situation and improve the picking efficiency.
The structure of picking actuator is shown in Figure 3. The clamping mechanism of the picking actuator is designed utilizing a parallel four-bar linkage structure, encompassed within a parallelogram configuration, featuring a maximum opening width of 70 mm. To ensure precise grasping of the mushroom cap without causing damage, a minimum clearance of 2 mm must be maintained between the actuator and the edge of the cap prior to initiating the picking action. Consequently, the maximum feasible diameter for mushroom caps that can be harvested using this mechanism is 66 mm. This structural feature ensures the consistency of the posture of the profiling jaws in space, that is, they are always vertically downward, and the two jaws keep parallel during the movement process, so that the clamping force of Agaricus bisporus is balanced. The gripper adopts cambered copying structure to increase the contact area and reduce the contact stress. The clamping surfaces are lined with TEP, a flexible material widely used in the medical devices and sports equipment industries. Known for its outstanding elasticity, wear resistance, flexibility, and skin-friendliness, this material readily conforms to the shape of Agaricus bisporus, providing a gentle yet firm grip that effectively mitigates mechanical damage caused by compression during the picking process. Adopting deceleration motor as power output to reduce speed and increase output torque. The dynamic torque sensor is used to collect the load torque of the motor, which is convenient to realize the constant force and low loss picking of the picking actuator.
First, after the picking actuator receives the picking instruction, the output power of the deceleration motor is controlled, and the power is transmitted to the clamping mechanism through the torque sensor. After the clamping mechanism closes at low speed, Torque sensor receives torque increase signal, and the output torque of the motor is increased. Until the clamping mechanism reaches the appropriate grasping force, keep the grasping force constant and rotate 45° to separate the mushroom from the base material, and realize constant force picking the mushroom.
The hardware of the control system includes the bottom controller, Industrial Personal Computer (IPC) and the upper computer.
The bottom controller uses STM32 single chip microcomputer to realize the driving control of picking robot, including traveling on the U-shaped track of the mushroom rack, traveling in the X, Y, and Z axes of the truss, and the constant force clamping of the picking actuator. The MCU adopts ARM Cortex-M3 kernel 32-bit embedded processor with USART/SPI communication protocol interface.
The IPC adopts Intel quad-core and four-thread J1900 processor, 4G operating memory, and four USB ports. It receives and processes depth image data information through the USB interface during work, and then communicates with the underlying controller of the picking robot through the USB to TTL serial port module to complete the detection process line by line. In addition to the above functions, IPC is also responsible for receiving and processing the information of the upper computer.
The upper computer communicates with the IPC through the wireless local area network to realize the remote monitoring and control of the picking robot.
The working principle is shown in Figure 4. The image acquisition device collects the image information to the IPC. The IPC analyzes and processes the image information, then sends instructions to the bottom controller of the picking robot combined with the set parameters. The bottom controller of the picking robot sends instructions to the truss walking mechanism, picking actuator and mushroom collecting device to complete the picking and collection box mushrooms. The image acquisition device is mounted on the truss type walking mechanism, and the two need to cooperate to complete the image acquisition.
According to the industry standard of Agaricus bisporus[21], the diameter of the cap of Agaricus bisporus is chosen as the characteristic parameter for determining suitability for harvest. This study primarily focuses on intermediate-grade and higher-quality specimens; hence, Agaricus bisporus with diameters exceeding 25 mm should be selectively picked. Therefore, the image processing of Agaricus bisporus was required to locate the position coordinates and detect the diameter of Agaricus bisporus.
The image processing flow is shown in Figure 5. First, the depth image was preprocessed to remove the abnormal noise points in the image, and the matrix background was segmented to extract the mushroom cap binary image. Then, the relative world coordinates were obtained through coordinate transformation, and the location coordinates of the mushroom target were obtained. Finally, the diameter of the mushroom target was obtained through circle detection.
There are some noise points with abnormal depth values in the images collected by the depth camera, which will have a direct impact on the subsequent image processing. In computer vision processing, it is common to achieve the purpose of eliminating noise points through morphological processing, but when binarizing the image, the setting of the threshold is a key problem, once the threshold is fixed, the robustness of the algorithm will be greatly reduced. The quartile method has good robustness to outliers, and it sets the threshold based on the quartile range of the data, which can effectively avoid the interference of noise points on the threshold calculation. Therefore, the main purpose of preprocessing is to eliminate these noise points. The lower quartile value Q1 and the upper quartile value Q3 with non-zero depth value in the depth image were counted, and the quartile distance IQR was calculated.
| IQR=Q3−Q1 | (1) |
The depth value within the interval [Q1–IQR, Q3+IQR] is reserved to realize the removal of abnormal noise points. After removing abnormal noise points, the depth value of the depth image can represent the distance between the target object and the camera[22].
The method in this study is inspired by the natural phenomenon of the gradual ebb tide of the seawater, in the process of the tide gradually receding, the island is gradually revealed, the area gradually increases, when the island area is no longer increased, the whole picture of the island is obtained, and the target position is determined by scanning the depth value distribution of each planing surface under the resolution layer by layer, so as to realize a gradual refinement of the segmentation process, which can more accurately determine the location of the target and improve the accuracy of segmentation. It’s more appropriate to think of it as a way of sampling or slicing. The segmentation method of sampling or slicing has strong robustness, and can effectively cope with changes and interferences in different scenes when processing complex images, with high accuracy and robustness.
The structural light from the depth camera does not reach the bottom of the mushroom cap, resulting in cylindrical shaded areas. In image processing, the background material in the depth image is regarded as the sea bottom, and the mushroom is regarded as the towering island. In the process of low tide, the sea bottom will always be submerged, and the towering islands will slowly emerge.
In the preprocessed depth image, the depth value of mushroom cap is small, while the depth value of base material is large. During background segmentation, the sea water starts from the minimum depth value of the image (seawater submerged isolated island) and gradually decreases to the maximum depth value (sea bottom). It is detected from the top of mushroom to the base material layer by layer. In the first layer T1, set the pixel A1 with the minimum depth value as 0, and the other pixel points as 1 to generate a binary image. In the second layer T2, the minimum depth value is increased by the set threshold ΔA, and the pixels less than or equal to the depth value are set to 0, and other pixels are set to 1 to generate binary image. The third layer T3, increases the minimum depth value by 2ΔA, and sets the pixel point less than or equal to the depth value as 0, and the other pixel points as 1 to generate A binary image. Continue to the next layer of detection, when the detection reaches the upper layer of the image depth value, the detection ends. The background segmentation process is detected layer by layer, as shown in Figure 6.
| {A1∈1,(A1,∞)∈0[A1,A1+ΔA]∈1,(A1+ΔA,∞)∈0[A1,A1+2ΔA]∈1,(A1+2ΔA,∞)∈0…… | (2) |
The closed area is filled for the detected binary image of each layer to get the fill figure. The island figure is obtained by subtracting the fill figure from the detected binary image. As the tide falls, a series of island maps will be generated, and the area (number of pixel points) of each island in the map will be analyzed to find the target of mushroom. The target of was the highest in height. Under the action of the cylindrical shadow formed by the cover of Agaricus bisporus, the area of the island formed by the target of Agaricus bisporus hardly changed in a certain water level interval. This section is the distance between the maximum diameter of the cap of Agaricus bisporus and the substrate. According to this characteristic, Agaricus bisporus target and fungous materials were separated.
Last but not least, the data source in this paper is the depth image, and the presence of stains may affect and interfere with the first layer of the determination of the minimum depth value in section 3.2, and even when the thickness of the stain exceeds the R value in Equation (3), it will interfere with more layers, but these layers are only process values, and the influence and interference do not affect the layer when the position of the twin mushroom is finally determined by this method, and the area in a certain water level interval is almost unchanged when the method determines the position of the twin mushroom according to the method, so the influence of the initial layers due to the thickness of the stain on the positioning effect of the algorithm is effectively avoided.
| {|Aj+k−Aj|Aj<RAj⩽ | (3) |
where, Aj is the number of pixel points in a Agaricus bisporus region (including the Agaricus bisporus adhesion region) at layer j; k represents reasonable segmentation depth between mushroom cap and base material; R represents The rate of change of the number of pixels; Aj+k is the number of pixels in the Agaricus bisporus region (including adhesion region) at layer j+k.
k value is related to the height of mushroom itself and should be less than the minimum distance between the maximum diameter of mushroom cap and the base material. According to the test statistics, when the diameter of mushroom was between 15 mm and 65 mm, the distance between the maximum diameter of the mushroom lid and the base material was 8-13 mm. Considering the influence of the surface smoothness of the base material on the detection of mushroom, k value was set as 5 mm. In the detection process, since the influence of noise is not removed from the image, R value was 0.2. Under normal conditions, the diameter of mushroom is less than 25 mm without picking. When the diameter of mushroom is 25 mm, the number of pixel points in the depth image is about 4500. If N is 4300, the number of pixel points attached to mushroom will be much greater than 4300. So it can be detected normally. In the layer-by-layer detection, the targets judged to be mushroom are saved, and the binary image containing mushroom target is obtained after the detection.
Following background segmentation, a binary image featuring the targeted mushrooms was attained. Subsequently, morphological operations, including erosion, dilation, and opening, were applied to contract the mushroom cap boundaries and mitigate noise interference. The improved Canny operator with high signal-to-noise ratio and high accuracy was used to detect the edge of mushroom lid, and the outline of mushroom lid of mushroom was obtained[23]. The diameter measurement process of mushroom is shown in Figure 7. In order to accurately measure the diameter of mushroom, the coordinates of the depth image of mushroom were changed into world coordinates through coordinate conversion. As Hough transform is only applicable to data in image format, it is necessary to translate the coordinates of Agaricus bisporus to the first quadrant of coordinates. Then according to certain discretization precision and form a discrete point of binary image. Project the three-dimensional coordinates of the edge points of the mushroom lid onto the X-Y plane and mesh them according to the accuracy in millimeter. The length of 1 unit coordinate in the image is 1 mm. At last, Hough Transform was used to detect the contours of mushroom lid and calculate the true diameter of mushroom.
In order to calculate the diameter of mushroom, the depth image coordinates[24,25] are needed to be converted. There are four common coordinates in the field of computer vision: pixel coordinate system, image coordinate system, camera coordinate system, and world coordinate system, which are respectively represented as O-uv, Oi-XiYi, Oc-XcYcZc, Ow-XwYwZw. The coordinate transformation relationship between the world coordinate system and the depth image coordinate system is as follows:
| {Z_{\text{c}}}\left[ \begin{gathered} u \\ v \\ 1 \\ \end{gathered} \right]{\text{ = }}\left[ {\begin{array}{*{20}{c}} {{f_x}}&0&{{u_0}}&0 \\ 0&{{f_y}}&{{v_0}}&0 \\ 0&0&1&0 \end{array}} \right]\left[ {\begin{array}{*{20}{c}} \begin{gathered} R \\ 0 \\ \end{gathered} &\begin{gathered} T \\ 1 \\ \end{gathered} \end{array}} \right]\left[ {\begin{array}{*{20}{c}} {{X_w}} \\ {{Y_w}} \\ {{Z_w}} \\ 1 \end{array}} \right] | (4) |
where, Zc represents depth value of depth image; u and v respectively represent the X-axis and Y-axis coordinates of the depth image at any point in the image coordinate system; u0 and v0 respectively represent the X-axis and Y-axis center coordinates of the image; fx and fy respectively represent the focal length of the camera on the X and Y axis; R, T respectively represent camera external parameters are rotation matrix and translation vector; Xw, Yw, and Zw respectively represent the three-dimensional coordinates of any point in the world coordinate system.
In the coordinate transformation process, in order to simplify calculation, the origin of the world coordinate system and the camera coordinate system were set to coincide, so the depth value of the target object in the two coordinate systems is equal, that is, Zc=Zw, then the external parameter matrix of the camera [R T]=[1 0]. In the world coordinate system and camera coordinate system, points at the same position have the following conversion formula:
| \left\{ \begin{aligned} &{X_w} = \dfrac{{u - {u_0}}}{{{f_x}}} {Z_c} \\ & {Y_w} = \dfrac{{v - {v_0}}}{{{f_y}}} {Z_c} \\ & {Z_w} = {Z_c} \\ \end{aligned} \right. | (5) |
After coordinate transformation, the depth value Zc in the depth image is known. The position of each pixel in the pixel coordinate system of the depth image in the world coordinate system can be obtained through Equation (5).
In this study, Hough transform was used to detect the mushroom cover contour. The projection of the mushroom lid contour on the X-Y plane is regarded as a circle in the plane with (a, b) as the center of the circle and r as the radius. The point (x, y) on the mushroom lid satisfies the equation:
| (x-a)^{2}+{(y-b)}^{2}={r}^{2} | (6) |
A point (xi, yi) on the plane corresponds to a cone in the parameter space formed by (A, B, R).The cone formed by all the points on the circle intersects at one point, and the coordinates of this point in the parameter space are (a, b, r), that is, the parameters of the detected circle are obtained[26,27]. Grid the (A, B, R) parameter space, and when the cone formed by each point on the detected circle has intersection points, the intersection points fall into the same grid. After defining the grid precision, the real diameter of the detected circle can be obtained.
The control system is shown in Figure 8. Power on the system to initialize, eliminate emergency stop alarm information, clear variable parameters. Since the maturation of mushroom was mainly determined by its diameter, the threshold value of diameter size was taken as the maturation index of mushroom.
When the picking platform is activated, the climbing device runs to the mushroom rack layer and the IPC sends instructions to the bottom controller of the picking robot. The picking robot moves to the picking area, and the truss type walking mechanism is reset and runs, moving in the X and Y axes. The truss type walking mechanism runs, while the depth camera also works. The IPC recognizes and processes the image information from the depth camera. After the mushroom is detected to be ready for picking, the mechanical arm compensates for the position so that the clamping claw is directly above the mature mushroom. The bottom controller of the picking robot communicates with the picking actuator to control the picking actuator to move to the specified coordinate point to complete the picking operation. Then the truss walking mechanism drives the image collecting device and the picking actuator to return to the break point position, and continues to detect and pick until the harvesting of mature mushrooms in the picking area is completed. The picking robot moves to the next picking area. After this layer of mushroom is harvested, the picking robot returns to the climbing device. And the climbing device will transport the picking robot to the next layer of mushroom bed until all the picking work is completed. During the picking period, if the collection frame is full, the picking robot will return to the climbing device. When the replacement is completed, the picking robot will return to the breakpoint and continue to work. In the process of picking, the IPC communicates wirelessly with the visual control interface to realize the setting parameter reading system and real-time monitoring of the picking state. The climbing device is set as automatic control. After starting, it will run automatically according to the set program.
After the picking actuator receives the picking instruction, it begins to perform the picking work. In the closing stage of Clamping mechanism, the motor load decreases, the torque sensor acquired the torque decrease signal, and the control system controls the low voltage work of the motor. The motor moves at low speed, the clamping mechanism closes at low speed, and the impact load is small when contacting with Agaricus bisporus, so as to avoid Agaricus bisporus injury.
In the holding stage of clamping mechanism, the motor load increases, the torque sensor acquired the torque increase signal, the motor is in a stalling state. Control system controls the motor voltage increases and the output torque increases. Until the output torque reaches the appropriate torque range (The clamping force did not damage the Agaricus bisporus and could effectively complete the picking work). The control system continues to adjust the motor voltage to keep the output torque stable. The input and output linguistic variables of the fuzzy PID controller are within the range {–6, 6}. They are divided into the following fuzzy subsets: {NB, NS, Z0, PS, PB}. Each variable utilizes a triangular membership function. The control rules for this fuzzy PID controller are depicted in Figure 9. At the last stage of the picking process, the control system controls the reverse output of the motor power, and the clamping mechanism loosens its grip on the Agaricus bisporus. The control principle of clamping stage is shown in Figure 10.
Because the clamping claw and the image acquisition device are not in the same coordinate position, and an image may collect more than one target of mushroom to be collected. When picking, it is necessary to compensate the coordinate position so that the claw is directly above the cover of the mature Agaricus bisporus. The system can calculate the compensation coordinates in the world coordinate system according to the arrangement of picking actuator:
| \left\{ \begin{gathered} {X_{co}} = {X_{l0}} + {X_m} - {X_w} \\ {Y_{co}} = {\text{ }}{Y_{l0}} + {Y_n} - {Y_w} \\ \end{gathered} \right. | (7) |
where, Xl0, Yl0 respectively represent pick the X axis and Y axis coordinates of the actuator in the world coordinate system of the initial point; Xm, Yn respectively represent pick the coordinates of the movement of the actuator in the X and Y axes; Xw, Yw respectively represent the X-axis and Y-axis coordinates of the Agaricus bisporus mushroom to be picked in the world coordinate system; Xco, Yco respectively represent the position coordinates of the center point of the gripper reaching the mushroom to be picked along the X axis and Y axis direction; Compensating coordinates can be converted to obtain the position compensation distance:
| \left\{ \begin{gathered} {L_X} = X u \\ {L_Y} = Y u \\ \end{gathered} \right. | (8) |
where, LX, LY respectively represent the movement distance of the slider in the direction of X axis and Y axis, mm; u represents the length of the unit coordinate in the world coordinate system, mm.
According to the calculated compensation distance between the clamping jaw and the center point of the Agaricus bisporus to be picked at X axis and Y axis, the pulse number of the stepper motor can be obtained[28]:
| P = \dfrac{{180LD}}{{\pi {r_p}\theta }} | (9) |
where, P represents the pulse number type; rp represents synchronous pulley radius, mm; D represents the fine fraction; θ represents step angle, (°).
In order to realize the remote monitoring and control of Agaricus bisporus picking robot, a visual control software for Agaricus bisporus picking control and monitoring was designed and developed. The visual control software communicates with the picking platform through the wireless local area network to realize remote computer monitoring and control. Visualization software was designed and developed based on LabVIEW platform. The upper computer received the image detection results and picking information sent by the IPC. After decoding, the received data was displayed in real time on the interface of the visual control software. The visual control software is shown in Figure 11. Its functions include arbitrary selection of thresholds of diameter and size suitable for picking Agaricus bisporus to meet different picking needs, real-time display of processed image morphology, recording the picking quantity of Agaricus bisporus, and monitoring the working condition of picking robot.
In order to verify the image recognition function of picking robot, and the reliability and practicability of picking. The picking experiment of Agaricus bisporus was carried out in the Agaricus bisporus production laboratory of College of Agricultural Equipment Engineering, Henan University of Science and Technology in November 2020. The single-layer experimental mushroom stand was 4. 5 m in length and 1. 4 m in width, and had three layers. Test material selection of Henan University of Science and Technology of agricultural equipment engineering college the Agaricus bisporus production independent laboratory cultures of Agaricus bisporus. In order to prevent the growth of a large number of deformed mushrooms and inferior mushrooms, and ensure the quality of the finished products of Agaricus bisporus. After the fruiting body of Agaricus bisporus grows out, the clumped mushrooms are treated with bacteria-repellent treatment, and the severely adhered mushrooms are removed, so that the high-quality Agaricus bisporus with loose growth on the mushroom bed is finally obtained.
Taking the accuracy of Agaricus bisporus recognition as an index to investigate the accuracy of image recognition of picking robot, and the recognition accuracy rde of Agaricus bisporus was defined as:
| {r_{\rm de}} = \dfrac{{{n_1}}}{n} \times 100\text% | (10) |
where, n1 represents meet the picking requirements and identify the accurate number of Agaricus bisporus; n represents the total number of Agaricus bisporus in the picking area should meet the picking requirements.
Taking the success rate of Agaricus bisporus picking as an index to investigate the picking stability of Agaricus bisporus picking robot, and the picking success rate rsu was defined as:
| {r_{\rm su}} = \dfrac{{{n_2}}}{{{n_1}}} \times 100\text% | (11) |
where, n2 represents the number of Agaricus bisporus picked successfully (for the Agaricus bisporus identified accurately), that is, the number of Agaricus bisporus put into the picking collection box after picking.
Taking the broken rate of Agaricus bisporus picking as an index to investigate the picking reliability of Agaricus bisporus picking robot, the broken rate rda of Agaricus bisporus picking was defined as:
| {r_{\rm da}} = \dfrac{{{n_3}}}{{{n_2}}} \times 100\text% | (12) |
where, n3 represents The number of collected Agaricus bisporus with visible damage to the naked eye.
The robot picking yield rou of Agaricus bisporus is,
| {r_{\rm ou}} = {r_{\rm de}} {r_{\rm su}} \left( {1 - {r_{\rm da}}} \right) | (13) |
The main factors affecting the accuracy of machine vision recognition are: omission, error and unqualified diameter measurement. The omission rate rmd, error rate rfd and the maximum error rme of diameter measurement are respectively defined as:
| \left\{ \begin{aligned} & {r_{{\rm md}}} = \dfrac{{{n_{11}}}}{n} \times 100\text% \\ & {r_{{\rm fd}}} = \dfrac{{{n_{12}}}}{n} \times 100\text% \\ & {r_{{\rm me}}} = \dfrac{{\left| {{d_a} - {d_m}} \right|}}{{{d_a}}} \times 100\text% \\ \end{aligned} \right. | (14) |
where, n11 represents the number of Agaricus bisporus that meet the picking requirements but miss detection; n12 represents the number of Agaricus bisporus that meet the picking requirements but are wrongly detected; dm represents machine vision measurement of Agaricus bisporus diameter; da represents actual measurement of Agaricus bisporus diameter.
On the bed of mushrooms, four different picking areas were selected for picking platform picking test. Each picking area was 1.5 m long and the width was the width of mushroom rack, that is, the width of picking robot operation. In order to verify the picking performance of the picking platform, the scanning speed of the picking robot was selected as 0.1 m/s. According to the preliminary test results, the clamping force of the gripper was selected as 60 N. According to the industry standard NY/T 1790-2009[21], the picking diameter of mushrooms was selected as ≥35 mm, and the allowable error was 5% (within 2 mm).
The total number of Agaricus bisporus that met the picking requirements in each picking area was recorded as n. The number of Agaricus bisporus that met the picking requirements and was successfully identified as n1. The number of Agaricus bisporus that was successfully picked as n2. The number of Agaricus bisporus that was picked with visible injuries as n3, and the time spent in picking each picking area. After the experiment, the identification accuracy rate rde, picking success rate rsu, picking damage rate rda, picking yield rate rou and picking speed of Agaricus bisporus were calculated respectively. The average value of the test results was taken. In addition, 4 picking areas of the same size were selected for manual picking. The calculation method of experimental indicators of manual picking was the same as that of the picking platform. The corresponding data were recorded and calculated during the experiment to compare and verify the practicability of the picking platform. The test process is shown in Figure 12.
The picking results are listed in Table 1. On the basis of Table 1, machine vision and performance parameters of picking platform of mushrooms were further analyzed.
| Picking way |
Serial No. |
Total quantity N |
Number of successes identified N1 |
Number of successful picking N2 |
Damage quantity N3 |
Picking time/min |
| Picking platform | 1 | 101 | 93 | 90 | 1 | 10.53 |
| 2 | 82 | 76 | 74 | 2 | 9.37 | |
| 3 | 93 | 88 | 84 | 3 | 10.82 | |
| 4 | 76 | 71 | 68 | 1 | 10.25 | |
| Avg | 88.00 | 82.00 | 79.00 | 1. 75 | 10.24 | |
| Manual picking | 1 | 112 | 92 | 92 | 1 | 5.02 |
| 2 | 85 | 70 | 70 | 0 | 4.18 | |
| 3 | 70 | 66 | 66 | 0 | 3.05 | |
| 4 | 94 | 82 | 82 | 0 | 4.50 | |
| Avg | 90.25 | 77.50 | 77.50 | 0.25 | 4.19 |
DownLoad:
CSV
There are differences in individual growth of Agaricus bisporus and poor smoothness of matrix. When the image processing algorithm is used for circle detection, the edge of Agaricus bisporus cover may be irregular due to background segmentation error and resulting in contour detection error. Therefore, it is necessary to test the algorithm. Data analysis of image processing and detection results is listed in Table 2. The picking identification accuracy of the four groups of experimental picking platforms was all higher than 92.50%, the average missed rate was 4.58%, the average wrong detection rate was 1. 67.00%, the maximum diameter measurement error rud was all lower than 4.50%, and the machine measured diameters were all within the qualified range. The image processing algorithm has high detection accuracy and small diameter measurement error, which can meet the requirements of picking operation.
| Serial No. |
Recognition accuracy/% |
Miss ratio/% |
Fallout ratio/% |
Maximum error in diameter measurement/% |
| 1 | 92. 08 | 4. 95 | 1. 98 | 3. 99 |
| 2 | 92. 68 | 4. 88 | 1. 22 | 2. 92 |
| 3 | 94. 62 | 3. 23 | 2. 15 | 4. 41 |
| 4 | 93. 42 | 5. 26 | 1. 32 | 3. 80 |
| Avg | 93. 20 | 4. 58 | 1. 67 | 3. 81 |
DownLoad:
CSV
As listed in Table 3, the average picking success rate of the picking platform was 96.34%, the picking breakage rate was 2.21%, the average picking yield was 87.79%, and the average picking rate was 8.59 pieces/min. Compared to manual picking, although picking platform is slower than manual picking, However, the identification accuracy of Agaricus bisporus was increased by 6.70%, and the yield of picking was increased by 1.51%. If the picking robot works continuously for 24 h, it can pick about 12 369 mushrooms, which is more than the daily output of a mushroom rack. Thus, it meets the requirements for automated Agaricus bisporus harvesting.
| Serial No. | Picking platform | Manual picking | ||||||
| Success rate/% | Damage rate/% | Productivity/% | Picking rate/min–1 | Recognition accuracy/% | Damage rate/% | Productivity/% | Picking rate/min–1 | |
| 1 | 96.77 | 1.11 | 88.12 | 9.59 | 82.14 | 1.09 | 81.24 | 22.31 |
| 2 | 97.37 | 2.70 | 87.81 | 8.75 | 82.35 | 0.00 | 82.35 | 20.33 |
| 3 | 95.45 | 3.57 | 87.09 | 8.60 | 94.29 | 0.00 | 94.29 | 22.95 |
| 4 | 95.77 | 1.47 | 88.15 | 7.41 | 87.23 | 0.00 | 87.23 | 20.89 |
| Avg | 96.34 | 2.21 | 87.79 | 8.59 | 86.50 | 0.27 | 86.28 | 21.62 |
DownLoad:
CSV
The reason for the higher accuracy of the identification of Agaricus bisporus on the picking platform is that there is a large error in the estimation of the diameter of Agaricus bisporus in the manual picking process, and there are more Agaricus bisporus with unqualified diameters. While the diameter measurement error of the machine recognition is small, and there are fewer missed and wrong detection cases. Manual picking would not fail, and the success rate was 100%. The picking failure of the picking platform was caused by the interference of the picking robot, and it was difficult to pick the Agaricus bisporus at the boundary of the u-shaped orbit mounted on the mushroom rack. Damage to Agaricus bisporus during manual harvesting predominantly arises from incorrect handling practices; however, the incidence of picking-related damage is notably low. When picking on the picking platform, the adhesion between the mushrooms will cause damage to part of the adhesive Agaricus bisporus. But the picking damage rate is low and acceptable. The results show that the picking platform has good accuracy, stability, reliability and practicability.
1) In this study, the Agaricus bisporus picking actuator adopts the claw type, while in some studies, the air suction type is used for picking. In the actual process of picking, the air suction type requires a large adsorption force, which is easy to cause damage to Agaricus bisporus. In the growth process of Agaricus bisporus, the base material needs to provide nutrients for the mushroom. So the hypha at the connection between the stalk and the base material is dense, and the direct suction will bring up a large amount of base material, which is not conducive to the subsequent grading and transportation operations, and also affects the continued growth of the lower tide mushroom in this area.
2) During the picking process, it is difficult for the picking robot to detect and pick the Agaricus bisporus at the edge of the mushroom bed (both sides of the U-shaped track). The reason is that there is interference between the picking actuator and the outer frame of the truss of the picking robot, and the picking area is limited. In future studies, its structure needs to be further optimized. Finally, it needs to be fully combined with planting agronomy to improve the applicability of the picking robot.
3) In practical production, instances may arise where mushroom caps overlap or become distorted due to mutual pressure, as when A partially obscures B. During the harvest of A, B’s influence can result in the misclassification of both caps as a single entity, causing a deviation in the localization center or even undetected mushrooms. Such scenarios pose a substantial challenge to the algorithm’s robustness. In subsequent research, these factors must be accounted for to refine the algorithm and augment its detection capabilities in intricate environments.
4) To enhance efficiency and speed in the actual picking operation, optimizations can be made at both software and hardware levels. On the software front, simplifying fuzzy PID control rules can be employed to reduce computational complexity, coupled with predictive and adaptive control strategies to proactively adjust picking actions, minimizing response lag. Preplanning the picking path avoids time-consuming on-site path planning, further economizing response times.
At the hardware level, upgrading to high-performance CPUs or GPUs can boost image processing speeds, algorithm execution, and instruction handling. Simultaneously, updating communication protocols or utilizing high-speed transmission media like fiber optics ensures real-time transmission of sensor data and commands, mitigating delays.
1) A machine vision based Agaricus bisporus picking system was designed, including climbing device, machine vision system, picking mechanism and upper computer control software, which realized the automatic identification, picking and collection of Agaricus bisporus.
2) Combined with machine vision technology, an online detection algorithm of Agaricus bisporus was proposed based on depth image processing. The recognition accuracy of the algorithm was higher than 92.50%, the missed detection rate was lower than 4.95%, the error detection rate was lower than 2.15%, and the diameter measurement error was less than 4.5%.
3) Based on the dynamic torque sensor signal feedback to control the motor output torque, the copying picking actuator is designed to realize the constant force clamping of the picking actuator. The picking success rate was 96.34%, the picking breakage rate was 2.21%, and the average picking yield was 87.79%.
4) Compared with manual picking, the identification accuracy of Agaricus bisporus was increased by 6.70%, and the yield of picking was increased by 1.51%. The overall performance of the designed picking platform is stable, which can meet the needs of automatic harvesting of Agaricus bisporus in factory production.
This research was supported by the National Key Research and Development Program of China (Grant No. 2023YFD2001100), the Major Science and Technology Programs of Henan Province (Grant No. 221100110800), and the Henan Provincial Major Science and Technology Special Project (Longmen Laboratory First-Class Project, Grant No. 231100220200).
| [1] |
Wang X Y, Zhang H, Li L, Liu Z. Influences of high oxygen atmosphere follow-up effect on shelf-life of Agricus bisporus. Transactions of the CSAM, 2017; 48(7): 309–316. (in Chinese) DOI: 10.6041/j.issn.1000-1298.2017.07.039
|
| [2] |
Geng Y C, Zhang T, Liu H B, Zhai L M, Yang B, Wang H Y. Effects of different briquetting modes on production of Agaricus bisporus. Transactions of the CSAE, 2016; 32(Z2): 275–278. (in Chinese)
|
| [3] |
Radzki W, Ziaja-Sołtys M, Nowak J, Topolska J, Bogucka-Kocka A, Sławińska A, et al. Impact of processing on polysaccharides obtained from button mushroom (Agaricus bisporus). International Journal of Food Science & Technology, 2019; 54(4): 1405–1412.
|
| [4] |
Dai F, Yang J, Zhao W Y, Li Z G, Xin S L, Zhang F W. Design and experiment of key assorted device based on factory production of Agaricus bisporus. Transactions of the CSAE, 2018; 34(6): 43–51. (in Chinese)
|
| [5] |
Shamshiri R R, Kalantari F, Ting K C, Thorp K R, Hameed I A, Weltzien C, et al. Advances in greenhouse automation and controlled environment agriculture: A transition to plant factories and urban agriculture. Int J Agric & Biol Eng, 2018; 11(1): 1–22. DOI: 10.25165/j.ijabe.20181101.3210
|
| [6] |
Wang Z Z, Li X F, Song S X, Liu M Y, Dong T. The progress of the research on the pretreatment and preservation of the Agaricus bisporus. Food Research and Development, 2018; 39(4): 200–206. DOI: 10.3969/j.issn.1005-6521.2018.04.037
|
| [7] |
Liu J, Wu Y C, Kan J, Wang Y, Jin C H. Changes in reactive oxygen species production and antioxidant enzyme activity of Agaricus bisporus harvested at different stages of maturity. Journal of the Science of Food and Agriculture, 2013; 93(9): 2201–2206.
|
| [8] |
Cheng D H. Design and analysis of the end effector for Agaricus bisporus harvesting manipulator. Mechatronics Information, 2020. pp.68–69.
|
| [9] |
Lee C H, Choi D, Pecchia J, He L, Heinemann P. Development of a mushroom harvesting assistance system using computer vision. 2019 ASABE Annual International Meeting, Boston: ASABE, 2019; Paper No. 1900505.
|
| [10] |
Tillett R. D, Batchelor B G. An algorithm for locating mushrooms in a growing bed. Computers and Electronics in Agriculture, 1991; 6(3): 191–200.
|
| [11] |
Yu G H, Luo J M, Zhao Y. Region marking technique based on sequential scan and segmentation method of mushroom images. Transactions of the CSAE, 2006; 22(4): 139–142.
|
| [12] |
Yu G H, Zhao Y, Li G, Shi H. Algorithm for locating individual mushroom and description of its contour using machine vision. Transactions of the CSAE, 2005; 21(6): 101–104. (in Chinese)
|
| [13] |
Yang Y Q, Ye M, Lu Y H, Ren S G. Localization algorithm based on corner density detection for overlapping mushroom image. Computer Systems and Applications, 2018; 27(5): 119–125.
|
| [14] |
Wang F Y, Feng W J, Zheng J Y, Sun J B, Niu L Y, Chen Z X, et al. Design and experiment of automatic sorting and grading system based on machine vision for white agaricus bisporus. Transactions of the CSAE, 2018; 34(7): 256–263. (in Chinese)
|
| [15] |
Wang F, Zheng J, Tian X, Wang J, Niu L, Feng W. An automatic sorting system for fresh white button mushrooms based on image processing. Computers and Electronics in Agriculture. 2018; 151: 416–425.
|
| [16] |
Wang L, Xu W, Du K W, Lu W, Zhu J H, Zhang J. Portabella mushrooms measurement in situ based on SR300 depth camera. Transactions of the CSAM, 2018; 49(12): 13–19, 108. (in Chinese)
|
| [17] |
Hu X, Pan Z R, Yang S Z, Tao Y. Study on the control system of Agaricus bisporus picking robot. Journal of Physics: Conference Series, 2019; 1187(3): 032033.
|
| [18] |
Hu X, Wang C, Tao Y. Design and application of visual system in the Agaricus bisporus picking robot. Journal of Physics: Conference Series, 2019; 1187(3): 032034.
|
| [19] |
Hu X, Pan Z R, Lyu S K. Picking path optimization of Agaricus bisporus picking robot. Mathematical Problems in Engineering, 2019; 2019: 8973153.
|
| [20] |
Reed J N, Miles S J, Butler J, Baldwin M, Noble R. AE - automation and emerging technologies: automatic mushroom harvester development. Journal of Agricultural Engineering Research, 2001; 78(1): 15–23. (in Chinese)
|
| [21] |
NY/T 1790-2009. Grades and specifications of Agaricus bisporus. Industry Standards for Agriculture, 2009. (in Chinese)
|
| [22] |
Sun J W, Zhao K X, Ji J T, Zhu X F, Ma H. Detection and diameter measurement method of Agaricus bisporus based on “submerged method”. Journal of Agricultural Mechanization Research, 2021; 43(2): 28–33. (in Chinese)
|
| [23] |
Ding L, Goshtasby A. On the Canny edge detector. Pattern Recognition, 2001; 34(3): 721–725.
|
| [24] |
Feng Q C, Zhao C J, Wang X N, Wang X, Gong L, Liu C L. Fruit bunch measurement method for cherry tomato based on visual servo. Transactions of the CSAE, 2015; 31(16): 206–212. (in Chinese)
|
| [25] |
Xu X L, Tao H, Li C J, Cheng C, Guo H, Zhou J P. Detection and location of pine wilt disease induced dead pine trees based on faster R-CNN. Transactions of the CSAM, 2020; 51(7): 228–236. (in Chinese)
|
| [26] |
Xiong J T, Lin R, Liu Z, He Z L, Yang Z G, Bu R B. Visual Technology of Picking Robot to Detect Litchi at Nighttime under Natural Environment. Transactions of the CSAM, 2017; 48(11): 28–34. (in Chinese)
|
| [27] |
Vera E, Lucio D, Fernandes L A F, Velho L. Hough Transform for real-time plane detection in depth images. Pattern Recognition Letters, 2018; 103: 8–15.
|
| [28] |
Wang G L, Liu W C, Wang A, Bai K K, Zhou H B. Design and experiment on intelligent reseeding devices for rice tray nursing seedling based on machine vision. Transactions of the CSAE, 2018; 34(13): 35–42. (in Chinese)
|
Figures(12) / Tables(3)
Supported by: Beijing Renhe Information Technology Co., Ltd. 
| Picking way |
Serial No. |
Total quantity N |
Number of successes identified N1 |
Number of successful picking N2 |
Damage quantity N3 |
Picking time/min |
| Picking platform | 1 | 101 | 93 | 90 | 1 | 10.53 |
| 2 | 82 | 76 | 74 | 2 | 9.37 | |
| 3 | 93 | 88 | 84 | 3 | 10.82 | |
| 4 | 76 | 71 | 68 | 1 | 10.25 | |
| Avg | 88.00 | 82.00 | 79.00 | 1. 75 | 10.24 | |
| Manual picking | 1 | 112 | 92 | 92 | 1 | 5.02 |
| 2 | 85 | 70 | 70 | 0 | 4.18 | |
| 3 | 70 | 66 | 66 | 0 | 3.05 | |
| 4 | 94 | 82 | 82 | 0 | 4.50 | |
| Avg | 90.25 | 77.50 | 77.50 | 0.25 | 4.19 |
DownLoad:
CSV
| Serial No. |
Recognition accuracy/% |
Miss ratio/% |
Fallout ratio/% |
Maximum error in diameter measurement/% |
| 1 | 92. 08 | 4. 95 | 1. 98 | 3. 99 |
| 2 | 92. 68 | 4. 88 | 1. 22 | 2. 92 |
| 3 | 94. 62 | 3. 23 | 2. 15 | 4. 41 |
| 4 | 93. 42 | 5. 26 | 1. 32 | 3. 80 |
| Avg | 93. 20 | 4. 58 | 1. 67 | 3. 81 |
DownLoad:
CSV
| Serial No. | Picking platform | Manual picking | ||||||
| Success rate/% | Damage rate/% | Productivity/% | Picking rate/min–1 | Recognition accuracy/% | Damage rate/% | Productivity/% | Picking rate/min–1 | |
| 1 | 96.77 | 1.11 | 88.12 | 9.59 | 82.14 | 1.09 | 81.24 | 22.31 |
| 2 | 97.37 | 2.70 | 87.81 | 8.75 | 82.35 | 0.00 | 82.35 | 20.33 |
| 3 | 95.45 | 3.57 | 87.09 | 8.60 | 94.29 | 0.00 | 94.29 | 22.95 |
| 4 | 95.77 | 1.47 | 88.15 | 7.41 | 87.23 | 0.00 | 87.23 | 20.89 |
| Avg | 96.34 | 2.21 | 87.79 | 8.59 | 86.50 | 0.27 | 86.28 | 21.62 |
DownLoad:
CSV
DownLoad: