Abstract: Seedling transplanting is the core component of modern tray seedling technology. Non-uniform substrate distribution and fluctuating environmental parameters contribute to weak seedling development and empty cell formation during seedling cultivation. Moreover, manual grading proves inefficient, costly and struggles to ensure consistent quality, while failing to meet the requirements of large-scale industrialized nursery. Consequently, the accurate identification of seedling categories along with obtaining their positional information is crucial for achieving automated grading. In this study, a lightweight instance segmentation model, YOLOv11s-RLDP, is proposed based on the YOLOv11s-Seg model. Firstly, RDSConv (reinforced depthwise separable conv) replaces standard convolutional layers in the network, enhancing feature extraction capability while reducing computational complexity. Secondly, the C3k2 module is redesigned using the Large Separable Kernel Attention (LSKA) mechanism in order to expand the model's receptive field and strengthen the perception of key seedling features. Subsequently, the DSGCF (dual stream gating cross fusion) module, incorporating Gating Convolution, is introduced to replace the original C2PSA module in the backbone, augmenting feature selection capability. Finally, the LAMP (layer-adaptive sparsity for the magnitude-based pruning) strategy is employed for model lightweighting. Ablation experiments validate the effectiveness and synergistic advantages of each module and the pruning strategy in improving segmentation performance and reducing computational resource requirements. Experimental results demonstrate that YOLOv11s-RLDP achieves the accuracy of strong weeding, recall, and mean average precision (mAP) of 91.2%, 95.1%, and 89.4%, respectively, with the improvement of 1.0, 0.7, and 1.4 percentage points over original YOLOv11s-Seg model. The mAP_50-90 increased by 1.8 percentage points, indicating significantly enhanced robustness when processing seedlings exhibiting complex growth conditions. Concurrently, the model parameter count and model size reduced by 34.0% and 32.5%, respectively against original model, facilitating future deployment on edge devices. Comparative experiments with different models reveal that the improved model holds significant advantages over the two-stage instance segmentation algorithm Mask R-CNN in both accuracy and lightweight design. Compared to one-stage instance segmentation networks like YOLOv5s-Seg, YOLOv8s-Seg, YOLOv9s-Seg, YOLOv10s-Seg, YOLOv11s-Seg, YOLOv12s-Seg, and YOLACT, the mAP of YOLOv11s-RLDP improves by 1.6, 1.3, 2.6, 1.5, 1.4, 1.5 and 8.0 percentage points, while simultaneously reducing model size by 7.0, 10.8, 5.2, 5.9, 5.8, 6.7, and 177.9MB respectively. In conclusion, the proposed YOLOv11s-RLDP model effectively enhances overall segmentation performance while reducing computational resource demands, which provides an algorithmic reference for the lightweight design and practical application of automated tomato plug seedling grading and localization. It should be emphasized that tomato seedlings exhibit complex and variable growth states with leaf outgrowth directions being highly random in real cultivation environments. Therefore, in future work, it is necessary to collect more samples of tomato plug seedlings exhibiting growth extending beyond cell boundaries to further enhance the model's capability of robustness and generalization.