The mechanical stability of a material is a fundamental issue in strength of atomic systems. Although the criterion of the mechanical stability of homogeneous structures such as perfect crystals have been successfully investigated so far, the criterion has not been able to be precisely evaluated in the cases of non-uniform deformations or bodies of inhomogeneous atomic structures. Now we present an instability criterion of an arbitrary atomic structure based on the energy balance of the whole system. This method gives the mathematically rigorous condition for the onset of an unstable deformation in any inhomogeneous atomic system. Furthermore, the method can be applied to any type of potential field, which means that ab initio evaluations of the mechanical instability of inhomogeneous structure under non-uniform deformation will be possible. The validity of the method is clarified by the application to tension of a cracked body. The onsets of unstable deformations and their deformation modes are precisely evaluated by the method.