Modern femtosecond technology can be used to create laser pulses that induce controlled spinning of anisotropic molecules to very high angular momentum states ("optical centrifuge"). In this paper we extend our previous study [M. Spanner and M. Ivanov, J. Chem. Phys. 114, 3456 (2001)] and focus on the stability of angular trapping and forced rotational acceleration of a diatomic molecule in an optical centrifuge. The effects of laser intensity modulations and rovibrational coupling are analyzed in detail, classically and quantum mechanically. The numerical simulations show excellent qualitative agreement between the quantum and classical systems. Forced rotations of the classical system can exhibit chaotic behavior, which becomes rather unique when the accelerating rotation of the angular trapping potential combines with efficient rovibrational coupling. In this regime the Lyapunov exponent becomes time-dependent and the trajectories separate as exp(lambdaF(t)).