In this paper, the Li2/3(1-x)Sn1/3(1-x)MgxO (LSMxO) ceramic systems were prepared by solid-state reaction using novel atmosphere-controlled sintering (x = 0-4/7). Pure Li2SnO3 was observed for x = 0, the Li2Mg3SnO6 and Li2SnO3 coexisted for x = 1/7, and the coexistence of three kinds of phases was detected for x = 1/5 and 1/4, including Li4MgSn2O7 impurity phase. Pure Li2Mg3SnO6-like phase with cubic rock salt structure in Fm-3m space group was obtained in the range of 1/3-4/7. All samples showed well-dense and smooth microstructures. The microwave dielectric properties highly depended on the phase composition, bond valence, FWHM of Raman spectrum, Raman shift, average grain sizes, and octahedral distortion. The LSMxO ceramics sintered at 1250 degrees C for 5 hours possessed excellent comprehensive properties of epsilon(r) = 15.43, Qxf = 80 902 GHz and tau(f) = +5.61 ppm/degrees C for x = 1/7. Typically, the LSMxO ceramics sintered at 1350 degrees C for 5 hours showed a maximum Q x f of 168 330 GHz for x = 1/2.