The viscosity of ionic liquids (ILs) is an important factor for their industrial applications, which determines mass transfer efficiency. In this work, the relationship between molecular structures and viscosity for three imidazolium-based ILs is investigated by molecular dynamics simulations and density functional theory. The shear viscosity was calculated by using periodic perturbation method, and accurate viscosities are obtained with less than 13% errors as compared to experimental values. Radial distribution functions and space distribution functions reveal that viscosities are correlated strongly to the local structures of ion pairs. Especially, the local aggregation of anions in certain sites of cations prevents movements of the ion pairs, which increase the viscosities of ILs. DFT calculations were further performed to analyze the effects of hydrogen bonds on the viscosities. The results show that asymmetrically distributed interaction sites lead to the high viscosities, which could be used as a rule of thumb for the future design of ILs with target viscosity.