The present paper is concerned with studying the onset and nonlinear regimes of convection in a binary liquid mixture with negative separation ratio in square cavity heated from above. The problem of temporal evolution of small perturbations of the non-stationary base state is solved in the framework of the linear approach. A direct numerical simulation based on the solution of full unsteady equations is carried out. The results of calculations show that the wavelength of the most dangerous perturbations and the time of the onset of instability decrease with the growth of solutal Rayleigh number (level of gravity) according to the power laws. The data on the structure of the critical perturbations and nonlinear regimes established after the loss of stability are obtained. The results of direct numerical simulation are compared with the dependencies of the onset time of instability and wave number of the most dangerous perturbations on the solutal Rayleigh number, obtained by solving the linear problem of temporal evolution of small perturbations; good agreement is found. (C) 2016 Elsevier Ltd. All rights reserved.