In this paper, we prove a theorem of linearized asymptotic stability for nonlinear fractional differential equations with a time delay. By using the method of linearization of a nonlinear equation along an orbit (Lyapunov's first method), we show that an equilibrium of a nonlinear Caputo fractional delay differential equation is asymptotically stable if its linearization at the equilibrium is asymptotically stable. Our approach is based on a technique that converts the linear part of the equation into a diagonal one. Then by using the properties of generalized Mittag-Leffler functions, the construction of an associated Lyapunov-Perron operator, and the Banach contraction mapping theorem, we obtain the desired result.