With the recently proposed formulation, an interatomic n-body potential was first constructed for the Ni-Nb-Mo metal system, and then applied to atomistic simulations to investigate the glass formation of the Ni-Nb-Mo ternary alloys. The simulations not only clarify the atomistic process of the metallic glass formation but also predict for the ternary system of a quantitative composition region within which metallic glass formation is energetically favored. In addition, the energy difference between crystalline solid solution and disordered phase i.e., the driving force for a supersaturated solid solution to amorphize could be considered as an indicator of the glass-forming ability (GFA) for a specific alloy. The GFAs of a series of Ni-Nb-Mo alloys were derived from the simulations, leading to pinpoint the Ni55Nb30Mo15 alloy with superior GFA in this ternary metal system. The Ni55Nb30Mo15 alloy can be considered as the optimized ternary metallic glass for thermal stability and manufacturability.