In this study, the silanized coal fly ash (CFA) and silanized nano-/micro-glass fiber (nGF) were simultaneously embedded in vinyl ester resin for improvement in both mechanical properties and mode I interlaminar fracture toughness of carbon fiber-based composite material. The used CFA particles have thermal power plant origin and were grinded with the help of ball mill to obtain the uniform size. The methacryloxy groups were attached to surface of both ground CFA and pristine nGF via silanization process. The homologous mixtures of S-nGF and S-CFA in vinyl ester resin were obtained with the help of high-speed mechanical stirrer and ultrasonic technique. The sheet molding compound method and hot curing process were applied to fabricate the carbon fiber-filled vinyl ester resin-based composite. Many techniques such as FTIR, EDS, rheological testing, mode I interlaminar fracture toughness, tensile strength and SEM were used to investigate the characteristics of silanized additives, the uncured matrices as well as the carbon fiber-based composite materials. The incorporation of both 0.1 wt% S-nGF and 1 wt% S-CFA into VE resin helps to improve the tensile strength, tensile modulus and the mode I interlaminar fracture toughness (G(IC)) of carbon fiber-based composites (CFRP) by 61.39%, 39.83% and 48.61%, respectively, when compared with pristine composite material. The fracture toughness surface of modified VE was tougher and rougher as a result of higher fracture toughness, while the pristine VE exhibited the smooth surface.