The mode of spatial optical in wireless laser communication systems directly affects the coupling efficiency of spatial optics to single-mode fibers. The mode conversion model of a free-spatial optical path based on a liquid crystal spatial light modulator is studied to improve the content of fundamental modes. According to the spatial spectrum filtering relationship between the conversion mode and the target mode, the corresponding conversion transfer function (CTF) is obtained. Combined with the simulated annealing algorithm, the CTF is accelerated locally, and its optimal value is obtained. Therefore, the conversion efficiency from higher-order modes to fundamental mode is generally improved. In addition, an experimental platform is established to verify the rationality of the simulation results. Further, it is indicated that the coupling efficiency can be improved by mode conversion.