The paper presents a linear stability analysis for the onset of convection in a nanofluid layer with magnetic field. The nanofluid layer incorporates the effect of Brownian motion along with thermophoresis. System of nanofluid layer in which nanoparticles concentrate near the bottom of the layer has been considered. The density gradient caused by such a bottom heavy nanoparticle distribution competes with the density variation caused by heating from the bottom as a result of which the mode of instability is oscillatory rather than stationary convection. It is established that the instability is almost purely a phenomenon due to buoyancy coupled with the conservation of nanoparticles. It is independent of the contribution of Brownian motion and thermophoresis to the thermal energy equation. Rather, the Brownian motion and thermophoresis enter to produce their effects directly into the equation expressing the conservation of nanoparticles so that the temperature and particle density are coupled in a particular way, and that results in the thermal and concentration buoyancy effects being coupled in the same way. Both stationary and oscillatory convection are investigated using normal mode technique. It is found that instability sets in as oscillatory motions rather than stationary convection. The effects of the Lewis number (Le), Concentration Rayleigh number (Rn), Modified diffusivity ratio (N-A), magnetic field (Q) on the stability of the system has been investigated. Magnetic field is found to stabilize the nanofluid layer for both the cases: the stationary convection and the oscillatory motions. The effect of various parameters on thermal Rayleigh number has been presented graphically. (C) 2013 Elsevier Ltd. All rights reserved.