This paper rigorously connects the problem of optimal control of McKean Vlasov dynamics with large systems of interacting controlled state processes. Precisely, the empirical distributions of near-optimal control-state pairs for the n-state systems, as n tends to infinity, admit limit points in distribution (if the objective functions are suitably coercive), and every such limit is supported on the set of optimal control-state pairs for the McKean Vlasov problem. Conversely, any distribution on the set of optimal control-state pairs for the McKean Vlasov problem can be realized as a limit in this manner. Arguments are based on controlled martingale problems, which lend themselves naturally to existence proofs; along the way it is shown that a large class of McKean Vlasov control problems admit optimal Markovian controls.