Abstract: Winter wheat is one of the most crucial food crops worldwide. Tiller density can serve as a key indicator to evaluate its population structure and growth status in precision agriculture, such as topdressing. Conventional tiller density monitoring can rely on manual field counting, leading to human errors. Fortunately, the existing UAV remote sensing inversion can be expected for the wide monitoring range and high timeliness. However, sufficient mechanistic interpretation and generalization are often required in the small and unevenly distributed field-measured samples. In this study, a UAV remote sensing monitoring was proposed to integrate the tiller-PROSAIL physical mechanism model and transfer learning, particularly for the high-throughput and accurate monitoring of winter wheat tiller density. Firstly, field experiments were conducted at the Quzhou Experimental Station of China Agricultural University in Handan City, Hebei Province, China. Ground data was collected, including the tiller density, leaf area index (LAI), chlorophyll content, and plant height in 2024 and 2025, with 150 samples as the modeling dataset and 100 independent samples as the external test set. Furthermore, the UAV multispectral images were acquired using a DJI Phantom 4 Multispectral drone. Preprocessing steps were performed, such as radiometric correction and image cropping. Secondly, a tiller density-guided PROSAIL model (tiller-PROSAIL) was constructed. Correlation analysis revealed that there was a significant positive correlation between tiller density and LAI (Pearson correlation coefficient = 0.69), where the LAI was selected as the bridge parameter. A linear fitting relationship between tiller density and LAI was established after segmental averaging (R² = 0.907 4). The PROSAIL model parameters (e.g., leaf angle distribution) were dynamically adjusted using tiller density. Canopy spectrum was simulated under different tiller scenarios. 4400 valid simulated data points were generated to cover a tiller density range of 100–4 500 tillers/m². An artificial bee colony algorithm was adopted to select the spectral feature for high efficiency and accuracy. Levy flight and adaptive mutation mechanisms were introduced with the 10 optimal spectral features (including RDVI, ARVI, GNDVI, KNDVI, NIR, NDRE, MCARI, EVI2, ISAVI, and WDRVI) from 19 candidate features. There was a great variation in the canopy structure, nitrogen nutrition, and chlorophyll content. Three training datasets were designed for inversion modeling using only measured data, direct mixing of measured and simulated data, and transfer fusion of measured and simulated data after Transfer Component Analysis (TCA). Random Forest Regression, Gradient Boosting Regression, and Support Vector Regression were employed for hyperparameter optimization via grid search and cross-validation. The results demonstrated that the Gradient Boosting Regression model with the TCA-fused dataset achieved the best performance, with a prediction accuracy R² of 0.763 and RMSE of 248.158 tillers/m². The transfer fusion effectively alleviated the domain shift between simulated and measured data, compared with the model with only measured data (R² = 0.720) and the direct mixing model (R² = 0.559). The large sample size and stability of simulated data were used to retain the authenticity of measured data. Validation with data from the following year (2025) showed that the good robustness (R² = 0.621, and RMSE = 213.953 tillers/m²) significantly outperformed the rest model with the measured data (R² = 0.349). The insufficient measured samples were avoided to realize the coupling between tiller density and canopy spectra using the tiller-PROSAIL model and transfer learning. The finding can provide a practical and reliable technical approach for the high-throughput, precise, and large-scale monitoring of winter wheat tiller density. Valuable support can also be offered for the decision-making on precision agriculture.