The role of sizing agent in the formation of interphase and the effect on the interfacial properties of carbon fiber (CF) reinforced vinyl ester resin (VE) composite were studied by molecular dynamics (MD) simulations. A novel unsaturated sizing agent N-(4'4-diaminodiphenyl methane)-2-hydroxypropyl methacrylate (DMHM) was used in this study. Epoxy sizing agent (E44) and modified epoxy sizing agent (ME44) were used for comparison. Simulation results showed that DMHM exhibited the highest interfacial bonding strength with CF surface due to the strong polarity of the sizing agent molecule. This result was confirmed by the interfacial shear strength test. The swelling effect of sizing agent in liquid VE was evaluated by solubility in styrene and compatibility with VE component. All three sizing agents had good solubility in styrene according to the calculation of total solvation free energy, while E44 and DMHM exhibited good compatibility with VE molecule based on the mixing energy calculation. The swelling property of sizing agent was confirmed by nano-indentation test. The results of this study explain the higher interfacial shear strength (62.25 MPa) between VE and CF sized by DMHM.