This paper investigates an iterative approach to solve the general rank-constrained optimization problems (RCOPs) defined to optimize a convex objective function subject to a set of convex constraints and rank constraints on unknown rectangular matrices. In addition, rank minimization problems (RMPs) are introduced and equivalently transformed into RCOPs by introducing a quadratic matrix equality constraint. The rank function is discontinuous and nonconvex, thus the general RCOPs are classified as NP-hard in most of the cases. An iterative rank minimization (IRM) method, with convex formulation at each iteration, is proposed to gradually approach the constrained rank. The proposed IRM method aims at solving RCOPs with rank inequalities constrained by upper or lower bounds, as well as rank equality constraints. Proof of the convergence to a local minimizer with at least a sublinear convergence rate is provided. Four representative applications of RCOPs and RMPs, including system identification, output feedback stabilization, and structured H-2 controller design problems, are presented with comparative simulation results to verify the feasibility and improved performance of the proposed IRM method. (C) 2017 Elsevier Ltd. All rights reserved.