Abstract: In view of the backward automation, low picking efficiency, and short picking cycle in the process of picking Camellia oleifera fruit in China, the machine vision technology applied to robotic harvesting technology is limited by the interference of the complex background in the real scene, which leads to the problem of low recognition accuracy. This paper takes Camellia oleifera fruit in the natural environment as the research object, and proposes an algorithm for identifying and detecting Camellia oleifera fruit in natural scenes based on Mask-RCNN. Firstly, the image of the Camellia oleifera fruit was obtained and the data set was established, and the ResNet convolutional neural network was used to extract the features of the Camellia oleifera fruit image, so as to obtain the fruit target segmentation result, and then RPN was used to operate the obtained feature map, and the full link layer was added to extract the mask pixel area of each sample is, and the target category was predicted. The test set was used to test the segmentation network model and target recognition algorithm of Camellia oleifera fruit respectively. The results showed that the segmentation accuracy of the network model was 89.85%, the average detection accuracy of the target recognition of Camellia oleifera fruit was 89.42%, and the recall rate was 92.86%. This algorithm can automatically detect the target of Camellia oleifera fruit, and effectively reduce the interference of factors such as leaf and flower bud occlusion, fruit overlap, fruit color and other factors under different lighting conditions, and provides reliable visual support for automatic fruit picking in natural scene.