In recent years, research on distracted driving behavior recognition has made significant progress, with an increasing number of researchers focusing on deep-learning-based algorithms. Aiming at the problems of the existing distracted driving recognition algorithm, such as its oversized model and difficulty in adapting to low computing environments, a lightweight network MobileNetV2, is chosen as the backbone network and improved to design a distracted driving behavior detection method that is both accurate and practical. The Ghost module is employed to replace point-by-point convolution to reduce the computation, the Leaky ReLU function helps mitigate the problem of dead neurons, as it prevents gradients from becoming zero for negative inputs. Finally, the channel pruning algorithm is used to further reduce the model parameters. The experiment results on the State Farm dataset show that the model’s test accuracy can reach 94.66%, and the number of parameters is only 0.23 M. The improved model has significantly fewer parameters than the baseline model, which demonstrates the effectiveness and applicability of the method.