It is proven in a previous paper that any modal approximation of the one-dimensional quantum harmonic oscillator is controllable. We prove here that, contrary to such finite-dimensional approximations, the original infinite-dimensional system is not controllable: Its controllable part is of dimension 2 and corresponds to the dynamics of the average position. More generally, we prove that, for the quantum harmonic oscillator of any dimension, similar lacks of controllability occur whatever the number of control is: the controllable part still corresponds to the average position dynamics. We show, with the quantum particle in a moving quadratic potential, that some physically interesting motion planning questions can be however solved.