Abstract: High accuracy and efficiency can often be required to assess the freeze damage of the winter wheat. However, the traditional field phenotypic identification can be confined to strong subjectivity, serious deviations between laboratory simulations and field data, and the lengthy cycle of physiological and biochemical detection. In this study, a freeze damage classification was proposed to fuse the vegetation indices, texture, and color features. A freeze damage model was also constructed to combine with the machine learning. The unmanned aerial vehicles (UAVs) were utilized to collect the multispectral and visible light imagery. Specifically, the UAVs were equipped with multispectral sensors and high-resolution visible light cameras. The optimal weather (clear skies, stable and light) was employed to ensure the data quality. A total of 16 vegetation indices were extracted, including the Normalized Difference Vegetation Index (NDVI), Ratio Vegetation Index (RVI), Excess Blue Index (ExB), and Soil Adjusted Vegetation Index (SAVI). The spectral reflectance was also selected from the specific wavelength bands in order to effectively represent the vegetation’s photosynthetic capacity and canopy health status. Furthermore, the Gray-Level Co-Occurrence Matrix (GLCM) was employed in texture feature analysis. Some parameters were calculated to quantify the spatial distribution of the pixel gray levels in the imagery, such as the mean, variance, contrast, and homogeneity from the GLCM. These texture features were then captured for the subtle structural variations in the wheat canopy caused by freeze damage. Additionally, the color features were derived from the RGB color space of the visible light images. Statistical parameters were computed to represent the visual color of the winter wheat under freeze damage, such as the color mean, standard deviation, and color moment. Feature screening was conducted using the Pearson correlation coefficient (PCC) method and the minimum redundancy feature selection algorithm (mRMR). Firstly, the Pearson correlation analysis was used to measure the linear correlation between each feature and freeze damage levels, thus eliminating the features with a low correlation. Subsequently, the minimum redundancy feature selection was optimized the feature set, in order to balance the feature relevance and redundancy. The final features were selected for the informative and non-repetitive ones. Four classification models were then constructed: Random Forest (RF), Support Vector Machine (SVM), Backpropagation Neural Network (BPNN), and eXtreme Gradient Boosting (XGBoost). Each model was also trained with tunable parameters. For example, the XGBoost model was utilized to adjust the parameters, such as the learning rate, the number of estimators, and the maximum depth after cross-validation. The experimental results demonstrated that the multi-source feature fusion significantly improved the performance of the model. Among them, eXtreme Gradient Boosting (XGBoost) showed the optimal performance under the fusion of vegetation indices, texture, and color features. The accuracy, recall, precision, and F1-score reached 0.725, 0.683, 0.705, and 0.688, respectively. Further analysis revealed that the texture features shared the highest correlation with the manually surveyed levels of the freeze damage. The structural damages of the canopy were also sensitive to capture, as the freeze damage often disrupted the orderly arrangement of the wheat leaves and stems. The texture features were effectively detected after capture. The multi-source remote sensing data fusion was validated to assess the freeze damage. The spectral, texture, and color information were integrated to precisely monitor the freeze damage of the winter wheat for disaster prevention decision-making. Remote sensing and machine learning were also applied to assess agricultural disasters in smart agriculture. The finding can serve as a practical reference for crop stress monitoring in the disaster early warning of intelligent agriculture.