In this paper, a novel computationally-efficient optimization scheme is proposed to account for the effect of wind speed variations over non-flat terrain and irregular land plot boundary. In particular, typical analytical wake models are augmented with a wind multiplier (obtained from CFD simulations) to account for the effect of altitude variations, while irregular boundaries are addressed through novel constraint handling method in an unrestricted coordinate optimization formulation. In contrast to the optimization via CFD simulation alone, the augmented wake model provides acceptably accurate results with a reasonable computational burden. The new optimization approach is subsequently applied to two real wind farms with significant altitude variations: the Gokceada Island wind farm and the Grasmere and Albany wind farm. The optimized Gokceada wind farm layout exhibits a significant cost of energy (CoE) reduction when topography is considered (over conventional, flat-terrain optimization). However, the Grasmere and Albany wind farm shows a relatively small decrease in CoE over conventional optimization. This discrepancy can be attributed to the wind characteristics in each location: the Gokceada wind farm has a relatively consistent wind direction and the optimization can use topography to carefully avoid wake interactions. In contrast, the Grasmere and Albany wind farm with less-consistent wind directions leaves no clear method to avoid wakes using topography. These results indicate that the consideration of wind farm topography can yield significant benefits and the wake interactions can be greatly reduced through careful placement of wind turbines in the case where a narrow wind directions is dominant. (c) 2017 Elsevier Ltd. All rights reserved.