Abstract: The operation supervision system based on agricultural machinery networking technology can identify the machine type and the operation state by collecting the image of the machine tool. However, with increase in the amount of image data, manual sampling is faced with challenges such as having a heavy workload and little supervision, which does not meet the supervision requirements. In this paper, image data sets including seeder, tilting plough, erasing machine, deep looser and rotary cultivator were constructed, and the machine image data sets were annotated and preprocessed under Google’s deep learning platform Tensorflow. A convolutional neural network model was designed to meet the actual regulatory requirements and image characteristics, after which the model was optimized by reducing over-fitting and improving training efficiency. The model training experiment results showed that the recognition rate of the machine recognition network designed in this paper reached 98.5% on the verification set. Under similar experimental conditions, the recognition rate of LeNet-5 model and ResNet-50 model was 81% and 98.8%, respectively. However, in terms of recognition efficiency, ResNet-50 model needed nearly 60 hours to complete the training and 0.3 s to recognize a picture, while the machine recognition network designed in this paper needed 30 hours to complete the training, and 0.1 s to recognize a picture. In order to further verify the practicability of the model, 200 images were selected for testing, and the average accuracy of the model for all kinds of machine and tool images was 98.47%, the average recall rate was 98.37%, and the average F1-score was 98.41%, indicating that the model had good robustness and practicability.