An accurate wake model is essential for the mathematical modeling of a wind farm (WF) and the optimal positioning of wind turbines (WTs). In order to effectively solve the WF layout optimization problem, this paper presents a newly-developed three-dimensional (3D) Gaussian wake model and applies it to the optimization of a WF layout. Firstly, the basic functions of the proposed model are deduced. Secondly, it is validated by wind tunnel measured data, and compared with the one-dimensional (1D) and the twodimensional (2D) wake models. Then, the 3D Gaussian wake model is applied in the WF layout optimization problems with identical and multiple types of WTs. The optimization objective is to maximize the total output power of the WF with a set of constraints. The Mixed-Discrete Particle Swarm Optimization (MDPSO) algorithm is used to solve the uniform and nonuniform WF layout optimization problems. Test cases of various WF sizes, wind conditions, and different wake models are simulated and analyzed. The simulation results demonstrate that the 3D Gaussian wake model can effectively address the WF layout optimization problem and further illustrate that the nonuniform design is beneficial to increase the WF's output power. (c) 2020 Elsevier Ltd. All rights reserved.