An X-ray diffraction method, estimating the strain relaxation in an ultrathin layer, has been discussed by using kinematic and dynamical X-ray diffraction (XRD) theory. The characteristic parameter Delta Omega, used as the criterion of the strain relaxation in ultrathin layers, is deduced theoretically. It reveals that Delta Omega should be independent of the layer thickness in a coherently strained layer. By this method, we characterized our ultrathin GaNxAs1-x samples with N contents up to 5%. XRD measurements show that our GaNxAs1-x layers are coherently strained on GaAs even for such a large amount of N. Furthermore, a series of GaNxAs1-x samples with same N contents but different layer thicknesses were also characterized. It was found that the critical thickness (L-c) of GaNAs in the GaAs/GaNAs/GaAs structures determined by XRD measurement was 10 times smaller than the theoretical predictions based on the Matthews and Blakeslee model. This result was also confirmed by in situ observation of reflection high-energy electron diffraction (RHEED) and photoluminescence (PL) measurements. RHEED observation showed that the growth mode of GaNAs layer changed from 2D- to 3D-mode as the layer thickness exceeded L-c. PL measurements showed that the optical properties of GaNAs layers deteriorated rapidly as the layer thickness exceeded L-c.