We study the controlled dynamics of the ensembles of points of a Riemannian manifold M. Parameterized ensemble of points of M is the image of a continuous map gamma : circle minus -> M, where circle minus is a compact set of parameters. The dynamics of ensembles is defined by the action gamma(theta) P-t (gamma(theta)) of the semigroup of diffeomorphisms P-t : M -> M, t is an element of R, generated by the controlled equation (x) over dot = f(x,u(t)) on M. Therefore, any control system on M defines a control system on (generally infinite-dimensional) space epsilon(circle minus)(M) of the ensembles of points. We wish to establish criteria of controllability for such control systems. As in our previous work [A. Agrachev, Y. Baryshnikov, and A. Sarychev, ESAIM Control Optim. Cale, Var., 22 (2016), pp. 921-9381, we seek to adapt the Liealgebraic approach of geometric control theory to the infinite-dimensional setting. We study the case of finite ensembles and prove the genericity of the exact controllability property for them. We also find a sufficient approximate controllability criterion for continual ensembles and prove a result on motion planning in the space of flows on M. We discuss the relation of the obtained controllability criteria to various versions of the Rashevsky-Chow theorem for finite- and infinite-dimensional manifolds.