In recent years, wind farm optimization has received much attention in the literature. The aim of wind farm design is to maximize energy production while minimizing costs. The wind farm layout optimization (WFLO) problem on uniform terrains has been tackled by a number of approaches; however, optimizing wind farm layouts on complex terrains is challenging due to the lack of accurate, computationally tractable wake models to evaluate wind farm layouts. This paper proposes an algorithm that couples computational fluid dynamics (CFD) with mixed-integer programming (MIP) to optimize layouts on complex terrains. CFD simulations are used to iteratively improve the accuracy of wake deficit predictions while MIP is used for the optimization process. The ability of MIP solvers to find optimal solutions is critical for capturing the effects of improved wake deficit predictions on the quality of wind farm layout solutions. The proposed algorithm was applied on a wind farm domain in Carleton-sur-Mer, Quebec, Canada. Results show that the proposed algorithm is capable of producing excellent layouts in complex terrains. (C) 2016 Elsevier Ltd. All rights reserved.