Abstract: Cultivated land non-agriculturalization has been a major challenge for the food security in China. The spatial-temporal pattern and evolution of cultivated land non-agriculturalization can be one of the most important steps for the decision-making on land use. The high-resolution satellite images have been widely used in surface remote sensing monitoring. However, it is still lacking on the classification system of non-agricultural monitoring using remote sensing, due to the complexity and diversity of the cultivated land non-agriculturalization types. In this study, a classification system was proposed for the remote sensing interpretation samples of the cultivated land non-agriculturalization, in order to construct the corresponding remote sensing interpretation sample database. At the same time, a fast sample collection was also proposed to improve the efficiency and quality of the sample collection using geographical condition monitoring. As such, high temporal, spatial precision and attribute reliability were achieved to verify the feasibility and effectiveness of the classification system and sample collection. The Hubei Province of China was selected as the study area. Nine types of samples were collected in the cultivated land non-agricultural sample system. The geographical conditions covered the flatland, hill, mountain, high-mountain and other terrains. The sample library was formed after training the deep learning model. The Efficient Net deep learning network was selected to extract the spatial distribution of cultivated land non-agricultural in study area. The result showed that: 1) The sample collection using geographical condition monitoring performed the best in the attribute accuracy. The changing pattern was quickly and accurately located in the more efficient solution for sample collection. 2) The model accuracy was significantly improved, when the number of samples exceeded 5000. The accuracy was verified by the internal visual interpretation and field verification points in the verification area. The positive detection rates were 67.0 % and 76.5%, respectively, and the recall rates were 77.9% and 76.5%, respectively. 3) The sample classification system was also used to train the optimized model. There was a significantly improved accuracy of the non-agricultural cultivated land automatic identification in the study area, compared with the full factor sample training model. The positive detection rate of the five verification areas increased by more than 20 percentage points. Therefore, the classification system can be expected to improve the efficiency and accuracy of remote sensing monitoring of cultivated land non-agriculturalization using deep learning. The improved system can be applied to the seasonal remote sensing monitoring of cultivated land at the regional scale.