We consider an N-machine flowshop with unreliable machines and bounds on work-in-process. Machine capacities and demand processes are finite-state Markov chains. The problem is to choose the rates of production on the machines over time to minimize the expected discounted costs of production and inventory/backlog. We show that the value function of the problem is locally Lipschitz and is a solution to a dynamic programming equation with a certain boundary condition. We provide a verification theorem, and derive the optimal feedback control policy in terms of the directional derivatives of the value function.