A study has been conducted on the analysis and fate of instabilities arising from rapid shear flow of a granular material. Specifically, this study is made through a detailed analysis of continuum rheological models, to examine whether the initiation and growth of the instabilities predicted by this analysis are consistent with the linear stability theory. It is found that rapidly sheared particles tend to form alternating bands or clusters of high and low solid concentrations. Two types of standing-wave instabilities, namely layering and stationary modes, are investigated. The fate of these density waves is tracked through transient integration of the macroscopic balance equations, and examined by a fast Fourier transform (FFT) analysis. In such a formulation, a few characteristic modes with different wave numbers have been identified. The temporal evolution of the intensity of fluctuating energy is investigated by analyzing the most dominant mode in the system. It is found that the structure and dynamics of the resulting waves can be quantitatively correlated with the power spectrum density of concentration fluctuations. The development of the layering mode is governed primarily by linear instability. On the other band, clusters of particles of the stationary mode may stretch and interact with neighboring clusters. This new finding indicates that the transient development of a two-dimensional stationary disturbance may ultimately lead tu a one-dimensional layered structure. The stationary and layering perturbations eventually evolve tu a similar structure and the stationary perturbation leads the system to the layered state more rapidly than rite layering mode.