During the milling process, self-excited vibration or chatter adversely affects tool life, surface quality and productivity rate. In this paper, nonlinear cutting forces of milling process are considered as a function of chip thickness with a complete third order polynomial (instead of the common linear dependency). An optimal control strategy is developed for chatter suppression of the system described through nonlinear delay differential equations. Counterbalance forces exerted by actuators in x and y directions are the control inputs. For optimal control problem, an appropriate performance index is defined such that the regenerative chatter is suppressed while control efforts are minimized. Optimal control law is determined based on variation of extremals algorithm. Results show that under unstable machining conditions, regenerative chatter is suppressed effectively after applying the optimal control strategy. In addition, optimal controller guarantees robust performance of the process in the presence of model parametric uncertainties. (C) 2013 Elsevier Ltd. All rights reserved.