The coupled exciton-vibrational dynamics of a three-site model of the Fenna-Matthews-Olson complex is investigated using the numerically exact multilayer multiconfiguration time-dependent Hartree approach. Thereby the specific coupling of the vibrational modes to local electronic transitions is adapted from a discretized experimental spectral density. The solution of the resulting time-dependent Schrodinger equation including three electronic and 450 vibrational degrees of freedom is analyzed in terms of excitonic populations and coherences. Emphasis is put onto the role of specific ranges of vibrational frequencies. It is observed that modes between 160 and 300 cm(-1) are responsible for the sub-picosecond population and coherence decay. Further, it is found that a mean-field approach with respect to the vibrational degrees of freedom is not applicable.