Abstract: Ensuring global food security while adapting to diversified dietary demands requires monitoring internal agricultural structure adjustments. Balancing grain security with the supply of diverse food resources is a priority, as highlighted by China's 2025 No. 1 Central Document. Consequently, remote sensing serves as a method for identifying the non-grain conversion of cultivated land. However, existing municipal-scale monitoring faces two bottlenecks: spectral confusion driven by geomorphological heterogeneity, and the computational load associated with long time-series modeling using traditional architectures like Vision Transformers, which often lose spatial topological details. To address these limitations, this study develops a geomorphological-phenological framework for identifying non-grain cultivated land. Taking Tangshan City—a typical double-cropping region in northern China—as a case study, this research focuses on crop structure-based non-grain conversion, defined as the substitution of staple grains with economic crops, and distinguishes it from seasonal land abandonment. The methodology integrates multi-source remote sensing data, including Sentinel-1, Sentinel-2, and Digital Elevation Models, and is implemented in two phases. First, based on terrain elevation, the study area was divided into three geomorphological zones: the northern hilly region, the central plain, and the coastal plain. Within each zone, a Random Forest algorithm was applied using optimized combinations of optical, radar, and topographic features across spring and autumn windows to extract the cultivated land extent and identify seasonal fallow fields. Second, to execute crop classification among maize, wheat, rice, and non-grain crops, we proposed an improved Vision Mamba model named GP-Mamba. We reconstructed a 10-dimensional multi-channel phenological feature set covering growth stages. To address the topological fracture caused by the one-dimensional serialization of standard Vision Mamba, we integrated a local spatial embedding module utilizing two-dimensional convolutions to preserve field boundaries. Additionally, an adaptive attention mask was introduced to dynamically suppress background noise and zero-padding artifacts. A bidirectional state space encoder was then employed to capture temporal dynamics. The experimental results demonstrated that the geomorphological zoning strategy mitigated cross-zone spectral confusion. The overall extraction accuracies for cultivated land in spring and autumn reached 97.7% and 96.6%, respectively, outperforming the non-zoned approach which achieved 90.3%. Furthermore, the improved GP-Mamba model achieved an overall crop classification accuracy of 92.3% and a Kappa coefficient of 0.91. It maintained a computational complexity of 15.8 GFLOPs, consuming approximately one-third of the computational resources required by traditional Vision Transformer models at 48.3 GFLOPs. Spatial analysis revealed geomorphological variations in land use. The northern hilly region displayed the lowest land-use stability, ranking highest in the study area with a crop structure-based non-grain rate of 47.86% and a spring fallow rate of 88.85%. The central plain held the largest absolute scale of non-grain conversion, totaling 85,800 hectares. Constrained by environmental factors, the coastal plain experienced a spring fallow rate of 78.29%. These findings offer a methodological reference for large-scale, dynamic farmland monitoring, supporting differentiated agricultural management in double-cropping areas.