Based on the ITO-HfO₂-graphene stacked layers, a modulator that can achieve independent modulation of TE mode and TM mode within a single silicon waveguide is proposed. Graphene and ITO are employed as electrodes of graphene-HfO₂-ITO capacitor. Under applied voltage, carriers accumulate within the 5 nm thick ITO film, leading to a modification of the optical dielectric constant. Using ITO’s epsilon-near-zero behavior and its sensitivity to optical polarization, a modulation depth (MD) of no <2.5 dB/μm and an insertion loss of less than 0.054 dB/μm are achieved at the 1.55 μm. Furthermore, when modulating TE and TM modes independently, the mode crosstalk between the two modes is <0.02 dB/μm, demonstrating the high performance of the polarization-multiplexing (PM) modulator. At near-infrared band of 1.35 to 1.6 μm, the broadband properties of ITO and graphene are used to achieve an MD of no <2.2 dB/μm while maintaining mode crosstalk and insertion loss of less than 0.06 dB/μm. This modulator has robustness to top Si waveguide sidewall angle and compatibility with complementary metal oxide semiconductor processes. It is expected to be used in highly integrated polarization de-multiplexing and polarization switching systems.