In recent molecular dynamics simulations [Phys. Rev. E 2003, 68, 061508] we found that the deviation of the single-particle displacement distribution from Gaussian form is a characteristic that is common to all phases of a system confined to a quasi-two-dimensional geometry (liquid, hexatic, and solid). These deviations, which intensify with increasing density and/or decreasing temperature, are a consequence of correlated particle motion and are related to the emergence of a third dynamical relaxation mode in the intermediate time regime. It was suggested that this collective motion is generated by superpositions of instantaneous normal mode vibrations along diffusive paths. The diffusive paths are along the directions with strong bond-orientation correlation and start to grow in amplitude rapidly on entry into the hexatic phase. In this paper we report the results of a study of the relation between the distribution of the instantaneous normal mode frequencies and the observed cooperative dynamics. We find that as the temperature decreases the distribution of the instantaneous normal mode frequencies (the real and the imaginary parts) shifts to lower frequency and the deviations of the single-particle displacement distribution from Gaussian form increase. The results indicate that there is a relationship between the average time at which the cooperative dynamics of the system is maximum, , and the average value of the squared frequency for which the spectrum of the imaginary 2 2 normal modes is maximum, <-omega(max)(2)>, that has the form ln = ln <-omega(max)(2)>(-1) + c(rho), where c(rho) is a density dependent constant.