Abstract: Hemerocallis citrina Baroni has a short picking cycle and relatively strict picking requirements. Aiming at the problems of the low efficiency and high subjectivity of manual harvesting of Hemerocallis citrina Baroni, a deep learning-based SSH-YOLOv5 Hemerocallis citrina Baroni maturity detection algorithm was proposed. Based on the YOLOv5 model, combined with the lightweight network ShuffleNet V2 basic residual unit to compress the size of the network model, and improve the model target detection speed. The attention mechanism module of Squeeze-and-Excitation network was integrated into the model to enhance the sensitivity of the model to useful feature information, and improve target detection precision, and ordinary convolution was replaced with depth-separable convolution module to further reduce the model computation. The experimental results showed that the number of parameters and floating point operations of the improved SSH-YOLOv5 model were reduced by 61.6% and 68.3% respectively, and the number of network layers was reduced by 18%, while the detection precision of SSH-YOLOv5 was improved from 88.8% to 91.2% of the original algorithm. The real-time detection speed reached 66.4 f/s, which was 18.1% higher than the original YOLOv5 algorithm and met the real-time detection requirements. The improved algorithm not only makes the model lightweight, but also makes Hemerocallis citrina Baroni maturity detection more accurate and faster, which can better meet the demand of Hemerocallis citrina Baroni detection.