The formation of the fluorite phase Y0.8Ta0.2O1.7 and other related phases in the system Y2O3-Ta2O5-MO (M = Mg, Ca, Sr or Ba) has been studied. The single fluorite phase formed when up to 12 mol% MgO was added to the fluorite phase; however, MgO appeared as the second phase as well as the main fluorite phase when more than 16 mol% MgO was added. When more than 8 mol% CaO was added to Y0.8Ta0.2O1.7, Ca2YTaO6 and Y2O3 were produced as the second phases as well as the main fluorite phase. Ba2YTaO6 and Sr2YTaO6 of the perovskite-type ordered structure and Y2O3 were produced as well as the main fluorite phase when only 4 mol% BaO or SrO were added to Y0.8Ta0.2O17. The region of the fluorite single phase was found in the system Y2O3-Ta2O5-CaO. The formation of the fluorite phase is assumed to be related to the cation radius of the doped alkaline-earth oxide.