Abstract: Citrus has been one of the most economically important crops in modern agriculture. Its phenotypic parameters are often required to be monitored precisely at early growth stages. Three-dimensional size parameters of immature green fruit can be expected to be acquired for yield prediction, water and fertilizer regulation, and harvest scheduling. However, conventional manual measurement and existing vision techniques suffer from low accuracy under complex backgrounds in orchards, due to weak textures, severe occlusions, and variable illumination. This study aimed to accurately measure the major and minor axes of immature green citrus under complex environments in orchards. Instance segmentation was integrated with multi-stage point cloud matching. A high-performance segmentation model, CSF-YOLO, was developed to enhance the YOLO11s-seg baseline. Three components were proposed: a Cross-Stage Spatial-Frequency Adaptive Module for dual-domain feature extraction, a Pinwheel-shaped Convolution for expanded receptive fields, and an improved dynamic upsampler for semantic preservation. In measurement, point clouds were generated from segmentation masks and depth data, according to the camera parameters after calibration, followed by LOF denoising. A multi-stage strategy was then employed: initial sphere fitting for template filtering, FPFH descriptor extraction, RANSAC correspondence establishment, and TEASER++ registration between the target and the best-fitting template. The experimental results demonstrate that the AP@50 values of CSF-YOLO were improved for both object detection and instance segmentation from 87.31% and 86.92% to 91.00% and 90.52%, respectively, compared with the baseline model; Similarly, AP@50–95 increased from 73.65% and 62.82% to 76.79% and 65.00%, respectively. Compared with current mainstream models, CSF-YOLO exhibited outstanding performance in object detection and instance segmentation with high efficiency. Ablation studies confirmed that all three modules contributed complementary improvements, indicating the best overall performance. Visualization results, including both segmentation and Grad-CAM++ heatmaps, further demonstrated that CSF-YOLO exhibited superior feature extraction and localization. Target instances were accurately focused under varying illumination, small target sizes, and severe occlusions, where other models shared significant missed detections. In the size measurement task, the best overall performance was achieved in the multi-stage matching, compared with conventional geometric fitting, such as Maximum Inter-point Distance, Sphere and Ellipsoid Fitting. Root Mean Square Errors were 3.96 and 3.10 mm, respectively, for the major and minor axes after measurement. Mean Absolute Percentage Errors were 5.52% and 5.41%, respectively. Residual analysis confirmed that the superior convergence was suitable for the local morphologies of citrus fruits, incomplete point cloud data, and asymmetric fruit morphology. An accurate solution was provided for citrus fruit sizing in the field. The segmentation model was combined with multi-stage matching. Two major challenges were given as the weak-feature target segmentation and asymmetric fruit size estimation under incomplete point cloud. High performance was also achieved to fully meet the requirements of precision agriculture. The findings can provide a practical reference to advance crop phenotyping of fruit crops in an intelligent orchard.