Effects of the presence of approximation errors are analyzed on the stability of adaptive optimal control using value iteration, initiated from a stabilizing control policy. This analysis includes the system operated using any single/constant resulting control policy and also using an evolving/time-varying control policy. Sufficient conditions on the 'per iteration' approximation errors are obtained for guaranteeing the stability. A feature of the presented results is providing estimations of the region of attraction, under the approximation errors, so that if the initial condition is within this region, the whole trajectory will remain inside the training region, and hence, the function approximation results remain reliable.