Abstract: Automatic recognition of crop disease based on deep learning has become a new research area in agricultural informatization. In order to explore the influence of the size and quality of data sets on the effectiveness of crop disease recognition based on deep learning, the model recognition effects obtained by different training data sets were studied and analyzed. Corn disease data set with the number of 338 was taken as sample, which was augmentation enhanced, background-removed and divided into smaller images containing individual lesions or localized symptom regions. Five CNN network models of AlexNet framework were designed and they were trained with different types of data sets and tested respectively. The final accuracy for each network was obtained using a 10-fold cross-validation approach. The experiments illustrated that the recognition models get different recognition accuracy of 56.80%, 78.30%、 80.50%、89.30% and 81.00%. The characteristics of the images of misrecognition by the CNNs were picked out and analyzed carefully, previous studies were analyzed as well, and nine factors of data set were identified as having the most impact on the crop disease recognition results. Analyzed comprehensively, the annotated data set size, leaf symptoms representative, covariate transfer, image background, image data acquisition condition, segmentation, feature variability, concurrent disease symptoms and symptoms of similarity are the influencing factors on the effectiveness of deep learning in crop disease recognition. This experiment can provide the basis and guidance for the application of deep learning technology in field disease recognition.