The present work aims at studying the thermal instability in a rotating porous layer saturated by a nanofluid based on a new boundary condition for the nanoparticle fraction, which is physically more realistic. The model used for nanofluid combines the effect of Brownian motion along with thermophoresis, while for a porous medium Brinkman model has been used. A more realistic set of boundary conditions where the nanoparticle volume fraction adjusts itself including the contributions of the effect of thermophoresis so that the nanoparticle flux is zero at the boundaries has been considered. Using linear stability analysis, the expression for critical Rayleigh number has been obtained in terms of various non-dimensional parameters. The effect of various parameters on the onset of instability has been presented graphically and discussed in detail.