The subject matter of this paper is the optimal control problem for continuous-time linear systems subject to Markovian jumps in the parameters and the usual infinite-time horizon quadratic cost. What essentially distinguishes our problem from previous ones, inter alia, is that the Markov chain takes values on a countably infinite set. To tackle our problem, we make use of powerful tools from semigroup theory in Banach space and a decomplexification technique. The solution for the problem relies, in part, on the study of a countably infinite set of coupled algebraic Riccati equations (ICARE). Conditions for existence and uniqueness of a positive semidefinite solution of the ICARE are obtained via the extended concepts of stochastic stabilizability ( SS) and stochastic detectability ( SD). These concepts are couched into the theory of operators in Banach space and, parallel to the classical linear quadratic ( LQ) case, bound up with the spectrum of a certain infinite dimensional linear operator.