The effect of impurities on the crystallite sizes and dislocation densities of ECAP-processed aluminum-magnesium alloys is studied by X-ray diffraction. It is shown that with increasing magnesium content the achievable reduction in crystallite size with ECAP eventually reaches a saturation state and a further reduction of the structural size seems unlikely. Simultaneously the dislocation density increases to a plateau level with increasing Mg content. In annealing experiments the microstructural stability of AlMg0.5 and the resulting changes are investigated by XRD profile analysis. It becomes evident that annealing leads to a moderate increase in crystallite size up to a temperature where accelerated crystallite growth begins. XRD results prior and after fatigue testing show an increase in crystallite size accompanied by a decrease in dislocation density.