Two-dimensional nanosheets are highly effective tougheners for vinyl ester resins. The toughening effect is related to the high specific surface area and unique two-dimensional planar structure of the nanosheets. In this study, a coupling agent γ-(2,3-epoxypropoxy) propytrimethoxysilane (Kh-560) was used to modify MXene nanosheets (M-MXene) for use in toughening vinyl ester resin. The mechanical properties, including the tensile strength, flexural strength, Young’s modulus and elongation, of neat vinyl ester resin and vinyl ester resin modified with MXene and M-MXene were investigated. The results showed that modification significantly improved the mechanical properties of the vinyl ester resin. The tensile and flexural strengths of the MXene-nanosheet-modified vinyl ester resin were 27.20% and 25.32% higher, respectively, than those of the neat vinyl ester resin. The coupling agent improved the interfacial compatibility between the MXene nanosheets and vinyl ester resin, which resulted in the tensile and flexural strengths of the M-MXene-nanosheet-modified vinyl ester resin being 52.57% and 54.60% higher, respectively, than those of the neat vinyl ester resin for a loading quantity of nanosheets of only 0.04 wt %, which is economically viable. The main mechanisms by which the nanosheets toughen the resin are crack deflection and crack pinning.