Computing a numerical solution to a periodic optimal control problem can be difficult, especially when the period is unknown. A method of approximating a solution to a stochastic optimal control problem using Markov chains was developed in [Krawczyk, J. B. (2001). A Markovian approximated solution to a portfolio management problem. Information Technology for Economics and Management, 1, http://www.item.woiz.polsl.pl/issue/journal1.htm]. This paper describes the application of that method to a periodic optimal control problem formulated in [Gaitsgory, V. & Rossomakhine, S. (2006). Linear programming approach to deterministic long run average problems of optimal control. SIAM Journal on Control and Optimization, 44(6), 2006-2037]. As a result, approximately optimal feedback rules are computed that can control the system both on and off the optimal orbit. (c) 2008 Elsevier Ltd. All rights reserved.