The Rayleigh convection could be induced by solute evaporation in dual solvents, such as acetone-ethyl acetate and isopropanol-water system. The onset of Rayleigh convection and its evolutionary characteristics were simulated using LBM-FDM hybrid method with liquid phase random disturbance model. And the simulation results were in reasonable agreement with experimental measurements in four aspects: Rayleigh convection onset time, concentration field acquired by Schlieren and velocity field acquired by PIV, average mass transfer coefficient, and statistical average velocity distribution along the gravitational direction. It proves that the LBM-FDM hybrid method is effective in capturing the key features in this mesoscale phenomenon. Then three mass transfer model including small eddy model, surface divergence model and the new model based on theoretical parameters concentration variance, its dissipation rate and Reynolds mass flux were proposed to predict the interfacial mass transfer coefficient, and the new developed model had advantages over other two models, in the onset time of Rayleigh convection, the peak of mass transfer coefficient and tackling the non-synchronism between velocity and concentration field in fully developed convective stage. The new developed model shows its strength in determining the mass transfer coefficient in a theoretical way. (C) 2018 Published by Elsevier Ltd.