We study the linear stability of a condensing thin liquid film of a binary vapor mixture by solving directly the bulk equations of the gas phase. The boundary layer of a finite thickness is introduced above the liquid film, within which the variables are disturbed. The dynamics of the liquid film is described by the long-wave equation. The neutral stability condition predicts the existence of a critical thickness below which a flat film is stable due to the mass gain effect. However, if we consider the thickening of the liquid film by condensation, the relative neutral stability can be defined such that the growth rate of a disturbance is equal to that of the basic film thickness. The critical thickness and wavenumber obtained from the relative neutral stability condition significantly change from the original ones. Employing the asymptotic analysis for large wavenumbers, the critical thickness and wavelength are numerically calculated for the water-ethanol system. Their dependence on the boundary layer thickness, temperature and ambient vapor concentration is investigated. The critical wavelength obtained from our theory has the same trend in the temperature and concentration as the initial drop distance observed in the experiment. (C) 2014 Elsevier Ltd. All rights reserved.