Pb-free halide double perovskites (HDPs) are proposed as potential candidates for various optoelectronic applications to replace the mainstream hybrid organic-inorganic halide perovskites, e.g., CH3NH3PbI3. While it is known that ion diffusion is a critical problem to affect the structural and electronic stability of CH3NH3PbI3, the mechanism of ion diffusions in HDPs is still unclear and highly desired to be revealed. In this study, taking Cs2AgInX6 (X = Cl, Br) HDPs as prototypes, for the first time we suggest that the fast ion diffusion of the dominant defects may play an important role in the performance stability of HDPs. Importantly, we find that the Agi(+) diffusion in a multi-ion concerted fashion has a much faster diffusion rate, compared to the V-Ag(-) and V-x+ diffusion in a single-ion fashion. It is revealed that HDPs exhibit quite different diffusion properties from CH3NH3PbI3. Furthermore, we demonstrate that the diffusion rate of Agi+ in HDPs can be effectively suppressed by applying an epitaxial strain, which opens a promising way to enhance the performance stability of perovskite materials for various device applications.