Abstract: An accurate identification and measurement of the cultivated land "non-grain" are crucial to ensure the national food security, farmland resources, and agricultural land use. However, the existing research on the non-grain cannot fully meet the requirement of the spatial distribution on the specific non-grain types. It is often required for the high-frequency monitoring and survey, particularly in the mountainous areas, such as the southern China. The cultivated land plots are also fragmented with the complex planting structure. There is an urgent need for monitoring at the plot-scale. In this study, an integrated framework was proposed to extract the automatic field boundary, and then classify the fine-scale non-grain cropland. The deep learning was also combined with the temporal richness of the satellite remote sensing data. The time series were applied to identify the non-grain types in Jianyang District, Nanping City, Fujian Province, China. Firstly, the cropland imagery was clipped using data from the Third National Land Resource Survey. The Richer Convolutional Features (RCF) deep learning model was applied to extract the field boundaries, and then delineate the current cropland areas. A sequential strategy was adopted to first identify the grape greenhouses, followed by the rest non-grain types. Monthly Sentinel-2 imagery was obtained via the Google Earth Engine (GEE). The plot units were utilized as the spatial constraints to extract the texture and multi-temporal spectral features. Phenological curves were reconstructed using the Savitzky-Golay (S-G) filter. Finally, the Random Forest algorithm was used for the crop classification after feature selection. The classified data was overlaid with the Third National Land Resource Survey, in order to map the spatial distribution of the non-grain cropland types at the plot scale. The results demonstrated that: 1) The improved RCF model was achieved in the high accuracy to extract the plot boundaries. The RCF model was effectively identified the plot boundaries, with a high intersection-over-union (IoU) value. The inference of the model was closely matched the actual plot shapes. 2) The Savitzky-Golay (S-G) filter was significantly improved to represent the crop phenology. The S-G filter was effectively captured the seasonal features and variation trends in crops. The time series curves of the vegetation index were more accurately reflected the growth status of the crops. 3) Feature importance analysis revealed that the specific spectral features were dominated the classification accuracy. A great contribution to the classification was from the red-edge features from the Sentinel-2 imagery, including the Red-Edge Position Index (S2REP) and Red-Edge Inflection Point Index (REIP), along with the spectral features in the winter-spring season and peak values during the growing season. 4) The classification model was achieved in the reliable performance. The overall accuracy of the crop classification was 81.6%, with a Kappa coefficient of 0.73. The non-grain cropland types were identified, including the lotus, grape greenhouse, citrus, tea garden and abandoned farmland. Among them, the lotus covered the largest proportion of the non-grain area. The abandoned farmland was also represented the smallest. A precise method was utilized to monitor the cultivated land non-grain using deep learning, remote sensing, and cloud computing. The finding can provide some insights into the land-use dynamics in the southern mountainous regions. A practical tool can serve as the sustainable farmland and the large-scale agricultural monitoring.