The catalytic dehydrogenation of ammonia borane (NH3BH3, AB) molecule is most frequently employed by metal catalysts, but a reliable dehydrogenation mechanism in molecular level has yet to be fully illuminated. Herein, adopting the density functional theory (DFT) method, the dehydrogenation mechanism and performance of NH3BH3 under the transition metal catalysts (Cu/MgO, Co/MgO, CuCo/MgO) were studied. The calculated results show that the dehydrogenation mechanism of AB refers to stepwise dehydrogenation mechanism: AB is adsorbed in the transition metal catalysts firstly, then one H(N) atom transferred to H(B) of -BH3 and to form H-2 molecule via the broken of B-H and N-H bond, finally, H-2 molecule desorption from the catalyst complexes. Among the transition metal catalysts, CuCo/MgO have the perfect catalytic activity in dehydrogenation reaction of NH3BH3, its barrier energy of the feasible pathway (path A) is 22.26 kcal/mol, which is lower than the barrier energy of AB-Cu/MgO(28.13 kcal/mol), AB-Co/MgO(27.46 kcal/mol), and the results of thermogravimetric analysis further verified the reasonability of DFT calculational results. Besides, partial density of states calculational results show the electron orbital hybridization of Cu, Co atom may account for the excellent catalytic performance of CuCo/MgO(100) compared with the Cu/MgO(100) and Co/MgO(100) in dehydrogenation process of AB.