Abstract: Aiming at the practical problems existing in the actual walnut production lines, including the high rolling speed of walnuts that makes it difficult to capture their tiny defects, and the difficulty in distinguishing narrow cracks from the natural growth textures of walnuts themselves, this study proposes a lightweight walnut external defect detection model named YOLO-EBM based on YOLO11n. In order to effectively solve the above engineering challenges and realize the dual requirements of high-precision detection and real-time operation in industrial scenarios, the model adopts three targeted structural optimization strategies, which are designed to achieve an optimal balance between detection accuracy and model lightweight performance without deviating from the actual application needs of walnut production lines. Firstly, the original backbone network of the baseline YOLO11n model is replaced by the lightweight EfficientNetV2 network. Compared with the original backbone, the EfficientNetV2 network adopts a more efficient feature extraction structure, which can not only enhance the model’s ability to extract key features of walnut defects, thereby improving the overall detection accuracy of the model, but also effectively reduce the number of redundant parameters of the model, laying a solid foundation for the lightweight design of the entire model. Secondly, a Bidirectional Feature Pyramid Network (BiFPN) is introduced into the model’s feature fusion module to optimize the efficiency of multi-scale feature fusion. This strategy appropriately sacrifices a small range of detection accuracy, and in turn achieves a substantial reduction in the number of model parameters, further promoting the lightweight level of the model while ensuring that the detection performance meets the actual production requirements. Finally, a Multi-Scale Dilated Attention (MSDA) mechanism is added to the small-object detection layer of the model. This mechanism can effectively enhance the model’s ability to capture fine-grained features of tiny defects and narrow cracks, strengthen the model’s discrimination ability between real defects and walnut natural growth textures, and further improve the overall recognition accuracy of the model, thus finally achieving a ideal balance between detection precision and model size. Comprehensive experimental tests were carried out to verify the performance of the proposed YOLO-EBM model, and the experimental results show that the improved model achieves a mean average precision (mAP) of 93.3%, the number of parameters is reduced to 0.773 M, and the size of the model weight file is compressed to 2.18 MB. Compared with the baseline YOLO11n model before improvement, the mean average precision of the YOLO-EBM model is increased by 2.1 percentage points, the number of parameters is reduced by 70.1%, and the size of the weight file is reduced by 58.2%. The model proposed in this study has both high detection accuracy and excellent lightweight characteristics, which is more conducive to the deployment and transplantation of the model on resource-constrained industrial production equipment, and can fully meet the real-time detection requirements of actual walnut production lines. In summary, this study provides a reliable technical reference and practical solution for the automatic and intelligent detection of walnut external defects in the agricultural product processing industry, and also provides a certain theoretical and technical basis for the research on lightweight detection models of other agricultural product defects.