Zn1-xMgxO epitaxial layers with x=0.5 and 0.2 were prepared by plasma-assisted molecular beam epitaxy on MgO (100) substrate. X-ray diffraction characterization revealed that both samples retain the rock-salt structure due to the confinement by the substrate lattice with low FWHM values (0.30-0.47 degrees) of the (200) rocking curves. The epilayer surfaces are flat having a root mean square roughness of similar to 1.0 nm as measured by atomic force microscopy. According to reciprocal space map and transmission electron microscopy (TEM) analyses, the epitaxial strains have been partly relaxed at film thicknesses of 110-130 nm. In fact, a further relaxation of the strain when preparing the TEM specimen from the Zn0.8Mg0.2O epitaxial layer triggers a reverse transformation from the rock-salt structure to the wurtzite one. The bandgap energy of the Zn0.8Mg0.2O epitaxial layer is found to be as low as 4.73 eV. (c) 2012 Elsevier B.V. All rights reserved.