This paper presents a new theory, known as robust dynamic programming, for a class of continuous-time dynamical systems. Different from traditional dynamic programming (DP) methods, this new theory serves as a fundamental tool to analyze the robustness of DP algorithms, and, in particular, to develop novel adaptive optimal control and reinforcement learning methods. In order to demonstrate the potential of this new framework, two illustrative applications in the fields of stochastic and decentralized optimal control are presented. Two numerical examples arising from both finance and engineering industries are also given, along with several possible extensions of the proposed framework.