This work is concerned with models and numerical algorithms for production planning of systems under uncertainties. Using stochastic processes to describe the system dynamics, we model the random demand and capacity processes by two finite-state continuous-time Markov chains. We seek the optimal production rate by minimizing an expected cost of the system. Discretizing the Hamilton-Jacobi-Bellman (HJB) equations satisfied by the value functions and using an approximation procedure yield the optimal solution, which allows us to make production decisions sequentially throughout the process lifespan. Three case studies are presented. Using demand data collected from a large paper manufacturer, the optimal production policies of the paper machine are obtained for different machine capacity and demand processes. (C) 2003 Elsevier Ltd. All rights reserved.