On the basis of density functional theory computations of the well-known chiral Au-38(SR)(24) nanocluster and its Pd- and Ag-doped derivatives, we propose here a mechanism for chiral inversion that does not require the breaking of a metal-sulfur bond at the metal-ligand interface but features a collective rotation of the gold core. The calculated energy barriers for this mechanism for Au-38 and Pd-doped Au-38 are in the range of 1-1.5 eV, significantly lower than barriers involving the breakage of Au-S bonds (2.5 eV). For Ag-doped Au-38, barriers for both mechanisms are similar (1.3-1.5 eV). Inversion barriers for a larger chiral Au-144(SR)(60) are much higher (2.5-2.8 eV). Our computed barriers are in good agreement with racemization barriers estimated from existing experiments for bare and doped Au-38. These results highlight the sensitivity of chiral inversion to the size, structure, and metal composition of the metal core and sensitivity to the detailed structure of the metal-thiolate interface. Our work also predicts that enantiopure Au-144(SR)(60) clusters would be promising materials for applications requiring high resistance to chiral inversion.