Rain streaks blur and degrade the color fidelity of images. This affects road segmentation accuracy in advanced driver assistance systems. To address this problem, a method combining the gradient property with the rain–fog model is proposed to remove rain streaks in single images. The rain detection model is used to detect rain streaks, which aids training for rain removal and determines if rain streaks exist in images. The rain removal model is based on trained patches with the highest proportion of rain streaks in the high-frequency layer for low-cost computation. In order to recover nonrain images without oversmoothing, the gradient property is used prior to handling the overlapping rain streaks in the background layer. The rain–fog model is employed to remove veiling effects and moderately enhance background scenes. Our results showed that this method outperforms existing methods in regard to visual performance and quantitative aspects.