Abstract: To solve the problem of high false positive rate of existing driver call behavior recognition, an improved Faster RCNN was proposed based on driver behavior detection method for detecting the illegal hand-held call of driver. An optimization strategy for the region proposal network（RPN） and the loss function was introduced, and the robustness of the network in detecting targets with different sizes was enhanced by applying multi-scale training, increasing the number of anchor points and introducing the residual expansion network on the original Faster RCNN. The simulation experiments of the proposed method were conducted with the images of driver behavior collected on an in-vehicle platform. The results show that compared with original Faster RCNN, RPN and Faster RCNN can realize efficient target detection by alternatively optimizing the shared feature extraction network part with 3.8% improvement in detection precision and better adaptation to the environment.