This work proposes a topology optimization design method of metamaterials for improving the vibration reduction performance. Firstly, an optimization mathematical model of the functional element is established with the objective of maximizing the origin mechanical impedance level, and the functional element with optimal vibration reduction effect is obtained by the calculation of the mathematical model. Then, the optimized functional elements are periodically arranged to generate the metamaterials, and thus a series of metamaterials with negative Poisson's ratio ranging from - 0.5 to - 2.0 are designed. Numerical simulation shows that the amplitudes of the acceleration response are reduced by 66.5% after the vibration is passed through the metamaterials. Comparison shows that the novel designed metamaterials achieve at least 12% improvement in vibration reduction performance over the traditional honeycomb.