The mechanical properties of Al1-xGaxN epitaxial layers grown on GaN and AlN substrates were studied by using density functional theory and anharmonic elastic calculation method. Accurate theoretical calculation of nonlinear elastic responses is a basic and important scientific question, which is the key to understand and further explore the potential synthesis and design of new solids. In this paper, detailed numerical calculations were carried out for the nonlinear mechanical properties of wurtzite GaN and AlN compounds. The variation of strain energy and stress of disordered Al1-xGaxN epitaxial layer with gallium composition was analyzed. We found that in the epitaxial structure of AlN substrate, with the increase of gallium content, the influence of nonlinear effect on internal stress reached 29.4%, which proved the great influence of nonlinear mechanical response on semiconductor alloy material and heterostructure.