Interfacial phonon transport widely exists in nanosystems, yet the physical mechanism has never been well understood. In this work, a numerical framework is developed for interfacial phonon transport between dissimilar materials with a frequency-dependent transmissivity by introducing the spectral diffuse mismatch model into an efficient kinetic-type Monte Carlo scheme. The numerical method is validated by modeling cross-plane phonon transport through several single-layer and bi-layer thin films, which shows good agreement with the discrete-ordinates solutions. Through mesoscopic modeling, the size effect of Kapitza conductance is found to be weak or vanishing when the equivalent equilibrium temperature or emitted phonon temperature is adopted for defining the interfacial temperature difference respectively. Furthermore, the effective Kapitza conductance decreases when interfacial roughness is introduced, which can be mainly ascribed to the increased interfacial area ratio by roughness. For engineering application, an empirical power law is proposed for the dependence of effective Kapitza conductance on interfacial area ratio. The present work will promote fundamental understanding and modeling capability of interfacial heat transport, as well as engineering design and optimization of interface in nano devices. (C) 2018 Elsevier Ltd. All rights reserved.