Abstract: Seedling transplanting is one of the most important components in modern tray seedling. Non-uniform substrate distribution and fluctuating environmental parameters contribute to weak seedling development and empty cell formation during seedling cultivation. Moreover, manual grading cannot fully meet the requirements of a large-scale nursery, due to the low efficiency, low quality, and high cost. Furthermore, tomato seedlings exhibit complex and variable growth states, with leaf outgrowth directions highly random in real cultivation environments. Consequently, it is often required to accurately identify seedling categories with the positional information during mechanical grading. In this study, a lightweight instance segmentation model, YOLOv11s-RLDP, was proposed using the YOLOv11s-Seg model. Firstly, RDSConv (reinforced depthwise separable conv) replaced standard convolutional layers in the network, thus enhancing feature extraction to reduce computational complexity. Secondly, the C3k2 module was redesigned using the Large Separable Kernel Attention (LSKA) mechanism to expand the model's receptive field and strengthen the perception of key seedling features. Subsequently, the DSGCF (dual stream gating cross fusion) module with Gating Convolution was introduced to replace the original C2PSA module in the backbone, thus augmenting feature selection. Finally, the LAMP (layer-adaptive sparsity for the magnitude-based pruning) strategy was employed for model lightweighting. Ablation experiments were conducted to validate the effectiveness of each module for high segmentation performance, according to the computational resource. Experimental results demonstrate that YOLOv11s-RLDP achieved the accuracy of strong seedling, recall, and mean average precision (mAP) of 91.2%, 95.1%, and 89.4%, respectively, with the improvement of 1.0, 0.7, and 1.4 percentage points over the original YOLOv11s-Seg model. The mAP50-90 increased by 1.8 percentage points, indicating significantly enhanced robustness when processing seedlings under complex growth conditions. Concurrently, the parameter count and model size were reduced by 34.0% and 32.5%, respectively, compared with the original model, thus facilitating future deployment on edge devices. Comparative experiments were performed on different models. The improved model performed best over the two-stage instance segmentation Mask R-CNN, in terms of accuracy and lightness. Compared with one-stage instance segmentation networks like YOLOv5s-Seg, YOLOv8s-Seg, YOLOv9s-Seg, YOLOv10s-Seg, YOLOv11s-Seg, YOLOv12s-Seg, and YOLACT, the mAP of YOLOv11s-RLDP was improved by 1.6, 1.3, 2.6, 1.5, 1.4, 1.5 and 8.0 percentage points, respectively, while the model size was simultaneously reduced by 7.0, 10.8, 5.2, 5.9, 5.8, 6.7, and 177.9MB, respectively. In conclusion, the YOLOv11s-RLDP model effectively enhanced overall segmentation performance with low computational resources. The finding can provide a strong reference for the lightweight and practical application in tomato plug seedling grading and localization. Therefore, tomato plug seedlings can be collected with growth extending beyond cell boundaries for their high robustness and generalization.