Abstract: Objective: This study aimed to develop a lightweight model that balances accuracy, complexity, and inference speed for detecting defective litchi fruit on postharvest conveyor belts, especially on devices with limited computing resources. Methods: Cracking, decay, and pericarp browning were treated as one defective-fruit category in accordance with the rejection task in sorting. Images of four litchi varieties were acquired under laboratory conveyor conditions covering five camera heights, two light color temperatures, two conveyor-belt types, multiple viewpoints, and different degrees of occlusion. After quality screening, 780 original images were retained, comprising 226 Feizixiao, 226 Guiwei, 226 Nuomici, and 102 Xianjinfeng images and approximating a variety ratio of 11:11:11:5. Before augmentation, the images were stratified by variety and divided into training, test, and validation sets at a ratio of 8:1:1, containing 624, 78, and 78 images, respectively. Offline augmentation was applied only to the training set, expanding it to 1,953 images. No augmented sample was included in the validation or test set, thereby avoiding augmentation-derived leakage across subsets. A model named YOLOv8n-PGM was constructed from You Only Look Once version 8 nano (YOLOv8n). Generalized shuffle convolution replaced standard convolution in the neck, while partial convolution reconstructed the C2f feature-fusion module to reduce redundant computation and memory access. Mixed local channel attention was inserted before the detection head to strengthen feature representation under occlusion. The original bounding-box regression loss was replaced with Inner-WIoU, combining an auxiliary bounding-box strategy with the dynamic focusing mechanism of Wise intersection over union. Performance was evaluated using precision, recall, F1-score, mean average precision, parameter count, model size, and frames per second. The experiments included ablation, loss-function and ratio-factor comparisons, variety-specific evaluation, comparisons with nine mainstream detectors, and embedded deployment tests. Results: YOLOv8n-PGM achieved a precision of 94.3%, a recall of 93.5%, an F1-score of 93.9%, mAP@50 of 97.7%, and mAP@50–95 of 80.0%. Compared with YOLOv8n, the two mean average precision measures increased by 0.5 and 1.5 percentage points, respectively. Parameter count decreased from 3.0 million to 2.0 million, a reduction of 33.3%, and model size decreased from 6.0 to 4.1 MB. On the workstation test video, average inference speed increased from 113.6 to 118.8 frames per second. In the ratio-factor experiment, Inner-WIoU with a factor of 0.8 produced the best combined result among factors from 0.5 to 1.0. In the variety-specific evaluation, mAP@50 ranged from 96.7% to 98.1%, and mAP@50–95 ranged from 77.1% to 81.0%. Under the same dataset and workstation conditions, YOLOv8n-PGM used fewer parameters and produced higher mAP@50 and mAP@50–95 than the nine comparison detectors. The models were also deployed on a Raspberry Pi 5 with PyTorch. Under the same device and input conditions, YOLOv8n-PGM reached 3.1 frames per second, compared with 1.7 frames per second for YOLOv8n. Its mAP@50 and mAP@50–95 were 97.7% and 80.0%, respectively, representing increases of 0.5 and 1.5 percentage points over YOLOv8n. These results indicated that the proposed modules jointly improved the balance between accuracy, compactness, and inference efficiency. Conclusion: Under the dataset and hardware conditions used in this study, the proposed method reduced model complexity while maintaining high detection accuracy and improving inference speed, providing a feasible basis for low-cost litchi sorting equipment; however, independent data and continuous tests on actual sorting lines are still required to establish robustness across seasons, devices, and long-term operation.