Nonlinear interaction induced by Kerr effects is a significant problem in the efforts made to ensure the successful development of transmission systems, such as mode-division multiplexing (MDM)-based optical multimode communication systems. A technique for reducing the interaction of the modes in MDM systems based on shaping envelopes of propagated signals is proposed. The envelopes of modes that carry m-array quadrature amplitude modulation (mQAM) are optically formed with a sinusoidal envelope to lower fiber nonlinearities by reducing the effective intensity and interference time between modes. The transmission performance of the proposed system is analytically characterized by developing an analytical model to estimate induced nonlinear phase noise and numerically investigated by examining the signal-to-noise ratio versus mode power. The effect of sinusoidal enveloped (SE)-4QAM format on the transmission distance is also explored for different mode combinations. Single-, two-, and five-mode transmissions are carried out to investigate the proposed method’s efficiency on the reduction of nonlinear interaction. In our system, the modes carry SE-4QAM format at rate 20  Gsymbol  /  s. The results show a significant enhancement in the performance of the MDM system when modes are modulated by SE-4QAM format. For example, the transmission distances of LP01 and LP11 are lengthened by 56.5% and 150% at the symbol error rate of 10^−  5, respectively.