The Markov chain approximation method is a widely used, robust, relatively easy to use, and efficient family of methods for the bulk of stochastic control problems in continuous time for reflected-jump-diffusion-type models. It has been shown to converge under broad conditions, and there are good algorithms for solving the numerical problems if the dimension is not too high. Versions of these methods have been used in applications to various two-player differential and stochastic dynamic games for a long time, and proofs of convergence are available for some cases, mainly using PDE-type techniques. In this paper, purely probabilistic proofs of convergence are given for a broad class of such problems, where the controls for the two players are separated in the dynamics and cost function, and which cover a substantial class not dealt with in previous works. Discounted and stopping time cost functions are considered. Finite horizon problems and problems where the process is stopped on first hitting an a priori given boundary can be dealt with by adapting the methods of [H. J. Kushner and P. Dupuis, Numerical Methods for Stochastic Control Problems, in Continuous Time, 2nd ed., Springer-Verlag, Berlin, New York, 2001] as done in this paper for the treated problems. The essential conditions are the weak-sense existence and uniqueness of solutions, an almost everywhere continuity condition, and that a weak local consistency condition holds almost everywhere for the numerical approximations, just as for the control problem. There are extensions to problems with controlled variance and jumps.