The 3D direct numerical simulation is performed to study the isothermal and nonisothermal flow in the rotating annular cavity of aspect ratio 9. Identification and characterization of mechanisms, related to the laminar-turbulent process in rotating cavities, should improve the prediction methods and lead to new, more effective boundary layer control strategies. Numerical computations are based on a pseudo-spectral Chebyshev-Fourier method for solving the governing equations. The numerical computations allowed us to describe the steady axisymmetric basic state and 3D instability structures which appear in the transitional flow.