This paper concerns problem formulation and solution procedure for inventory planning with Markov decision process models. Using data collected from a large paper manufacturer, we develop inventory policies for the finished products. To incorporate both variability and regularity of the system into mathematical formulation, we analyze probabilistic distribution of the demand, explore its connection with the corresponding Markov chains, and integrate these into our decision making. In particular, we formulate the Markov decision model by identifying the chain's state space and the transition probabilities, specify the cost structure and evaluate its individual component; and then use the policy-improvement algorithm to obtain the optimal policy. Application examples are provided for illustration.