Abstract: Macadamia nuts (Macadamia integrifolia) have been widely planted in the orchard environments of Yunnan Province, China. It is often required to accurately detect pests and diseases in the complex orchard. However, the performance of conventional object detectors is confined to the dense fruit distribution, severe occlusion between branches and fruits, strong backlighting, and high color similarity between target fruits and background foliage. In this study, a dataset was constructed with 1 952 original high-resolution images (6 000 × 4 000 pixels) under varied natural lighting conditions. Four representative categories included healthy, anthracnose-infected, stink bug-damaged and thrips-damaged fruits. Four typical scenarios—strong backlight (400 images), color coincidence (720 images), inter-fruit occlusion (432 images), and leaf occlusion (400 images)—with color coincidence accounting for the highest proportion (36.9%), due to the visual similarity between mature fruits and foliage. After that, quality screening, rotation and Gaussian blur augmentations were applied exclusively to the training set prior to data partitioning. The dataset was expanded into 5 856 images, and then divided into training (80%), validation (10%), and test (10%) sets. Stratified sampling was used to prevent data leakage at the original image and shooting sequence level. According to the characteristics of this dataset, a high-precision detection model, termed YOLO-K, was developed to introduce systematic enhancements into the lightweight YOLOv10n baseline architecture. In the data preprocessing stage, a K-means clustering algorithm with an Intersection-over-Union (IoU)-based distance metric (d = 1 − IoU) was employed to redefine the nine anchor box dimensions. The prior distribution of anchor boxes was better aligned with the actual scale distribution of small and densely packed targets. In the feature extraction stage, the C2F-BiFormer module was constructed to integrate the BiFormer attention mechanism into the C2F module. Bi-level routing attention was used to capture long-range dependencies and global contextual information with high computational efficiency. Thereby, pest and disease features were identified against complex background interferences. Additionally, the Convolutional Block Attention Module (CBAM) was introduced to replace the original PSA (Partial Self-Attention) and SPPF (Spatial Pyramid Pooling Fast) structures at the end of the backbone network. Dual refinement of channel-wise and spatial features enhanced the sensitivity to small-scale and occluded targets. In the feature fusion stage, the RepNCSPELAN4 module was incorporated to reconstruct the neck network. Structural re-parameterization and multi-level features were reused to mitigate information attenuation during deep feature propagation for the integrity of multi-scale feature representations. Extensive experiments were conducted on the self-built dataset under identical training conditions with an input resolution of 640 × 640 pixels, the SGD optimizer (initial learning rate 0.01, momentum 0.937, weight decay 0.000 5), a batch size of 4, gradient accumulation steps of 8, and a total of 300 training epochs, with mixed-precision training to accelerate convergence. The results showed that the improved YOLO-K model achieved a precision of 92.3%, a recall of 85.7%, a mAP@0.5 of 92.4%, and a mAP@0.5:0.95 of 75.6% on the test set, indicating substantial improvements of 13.1, 32.6, 26.6, and 29.4 percentage points, respectively, over the baseline YOLOv10n model (which achieved 79.2%, 53.1%, 65.8%, and 46.2%). The inference speed reached 62 frames per second (FPS), fully meeting real-time detection requirements. Compared with 11 mainstream object models—including YOLOv5, YOLOv8, YOLOv8m, YOLOv9c, YOLOv9m, YOLOv10n, YOLOv10m, YOLOv11n, YOLOv13n, Faster R-CNN, and SSD—as well as the previous YOLO-L model developed by the research group, YOLO-K achieved the highest accuracy over all evaluation metrics with the competitive computational efficiency (14.3 M parameters, 62.5 G FLOPs). Ablation studies further confirm that the complementary effects of the three modules: the C2FBIF module alone improved mAP@0.5 from 65.8% to 67.0%; directly adding CBAM on top of C2FBIF caused a performance drop to 64.8% due to feature distribution conflicts in the lightweight backbone; the introduction of RepNCSPELAN4 resolved this issue, thus elevating mAP@0.5 to 89.5% when combined with C2FBIF; Finally, the YOLO-K with three modules achieved the best performance of 92.4%, indicating the synergistic effectiveness of the enhancements. Visualization analysis under five typical scenarios—image blur, color coincidence, dense fruit clustering, strong backlighting, and cluttered backgrounds—demonstrated that YOLO-K substantially reduced false positive and missed detection rates, compared with the other models. This finding can provide a feasible and valuable technical reference for intelligent pest and disease monitoring in plateau mountainous orchards, contributing to the advancement of precision agriculture in complex agricultural environments.