The coefficient of thermal conductivity and viscosity of Al2O3-water nanofluid is measured, and its heat transfer is experimentally investigated in a square enclosure. In addition, a 2D two-phase Lattice Boltzmann model considering interaction forces (gravity and buoyancy force, drag force, interaction potential force and Brownian force) between nanoparticles and base fluid is developed for natural convection of nanofluid, and is applied to simulate the flow and heat transfer of Al2O3-water nanofluid in the square enclosure by coupling the density distribution (D2Q9) and the temperature distribution with 4-speeds. In this paper, the effects of different nanoparticle volume fractions (phi = 0.25%, phi = 0.5%, phi = 0.77%) and different Rayleigh numbers (Ra = 30,855,746 and Ra = 63,943,592 for phi = 0.25%, Ra = 38,801,494 and Ra = 67,175,834 for phi = 0.5% and Ra = 55,888,498 and Ra = 70,513,049 for phi = 0.77%) on heat transfer in the transition region are experimentally and numerically discussed. The numerical results have a good agreement with the experimental results. It is found that the heat transfer of nanofluid is more sensitive to the thermal conductivity than viscosity at low nanoparticle fractions and it is more sensitive to the viscosity than the thermal conductivity at high nanoparticle fractions. In addition, the forces between water and nanoparticles are analyzed, and the nanoparticle volume fraction distribution is investigated. It is found that the temperature difference driving force makes the greatest contribution to the nanopartide volume fraction distribution, and nanoparticle volume fraction distribution is opposite to that of the water phase density distribution. (C) 2014 Elsevier Ltd. All rights reserved.