Real-time segmentation of overlapped cherry fruits and defects using improved YOLOv8n-seg model

Cherry grading is often required for the accurate segmentation of the fruit bodies, stems and defects. However, these cherries have limited to overlap and densely distribute. Most cherries on the sorting line share the dark-colored surfaces (dark red, or black). Different cherry varieties (Hongdeng, Yellow honey, and Tieton) also exhibit the distinct colors. The low contrast and blurred boundaries have posed the great challenge on the fruit body segmentation of the acquired images. A wide variety of surface defects can also include the bruises, cracks, russets, and rots. Some surface defects are significantly imbalanced in the quantity or size. Accurate segmentation of defects can also confine to the blur and unclear boundaries of the features. In this study, a multi-branch real-time segmentation model (SemIns-YOLOv8) was proposed for the complex segmentation of cherries using an improved YOLOv8n-seg. A context-integrated semantic segmentation module was also introduced after the path aggregation network. The feature pyramid network (PAN-FPN) module of YOLOv8n-seg was used to combine the loss function of cross-entropy and Dice loss. The original instance segmentation module was replaced to identify the stems and defects. The recognition accuracy of smaller and less distinct defects was improved to reduce the recognition time. The resolution of feature map was enhanced to introduce the hausdorff distance loss (HD Loss). A boundary feature-enhanced instance segmentation module was constructed to precise separate the overlapping fruit bodies. More boundary information was focused after optimization. A series of experiments were conducted on a cherry-annotated sample set with 2,094 images in the training set, 524 images in the validation set, and 300 images in the test set. Experimental results showed that the improved model was achieved in a mean average precision at intersection over union thresholds from 0.50 to 0.95 (mAP50-95) of 98.2% for fruit bodies, an intersection over union (IoU) of 92.15% for stems, and a mean intersection over union (mIoU) of 65.97% for defects. The inference time per frame was 23ms on an NVIDIA GeForce GTX 1650 GPU, with the input size set to 1024×384 pixels. The improved model was achieved the better time and accuracy, compared with the mainstream segmentation models (such as YOLACT, Mask R-CNN, DeepLabv3, YOLOv7-seg, and YOLOv8n-seg). Specifically, the segmentation tasks were performed better. Among them, the mAP50-95 of fruit bodies, IoU of stems, and mIoU of defects were improved by 1.80, 2.21, and 4.70 percentage points, respectively, compared with the YOLOv7-seg. At the same time, the inference time per frame decreased by 47ms. Compared with the YOLOv8n-seg, the mAP50-95 of fruit bodies and IoU of stems were improved by 2.10 and 2.33 percentage points, respectively. While the mIoU of defects shared a significant improvement of 14.35 percentage points, and the inference time per frame was reduced by 1ms. The improvements were also applicable to the YOLOv11. In conclusion, the SemIns-YOLOv8 can be expected to accurately segment the cherry fruit bodies, stems, and various types of defects over the various varieties and varying brightness contrasts. The better recognition of smaller and less distinct defects was promoted to reduce the contour segmentation that caused by overlapping or densely distributed fruit bodies on the sorting line. This improved model can also provide the higher accuracy and speeds for the cherry sorting, fully meeting the practical requirements. The finding can offer a strong reference to evaluate the fruit appearance quality on the real-time sorting line.