We study the continuity properties of optimal solutions to stochastic control problems with respect to initial probability measures and applications of these to the robustness of optimal control policies applied to systems with incomplete or incorrect priors. It is shown that for single and multistage optimal cost problems, continuity and robustness cannot be established under weak convergence in general, but that the optimal expected cost is continuous in the priors under the convergence in total variation under mild conditions. By imposing further assumptions on the measurement models, robustness and continuity also hold under weak convergence of priors. We thus obtain robustness results and bounds on the mismatch error that occurs due to the application of a control policy which is designed for an incorrectly estimated prior as the incorrect prior converges to the true one. Positive and negative practical implications of these results in empirical learning for stochastic control are presented, where almost surely weak convergence of independently and identically distributed empirical measures occurs but stronger notions of convergence, such as total variation convergence, in general, do not.