The influence of solute-solvent interactions on the vibrational energy relaxation dynamics of perylene and substituted perylenes in the first singlet excited-state upon excitation with moderate (< 0.4 eV) excess energy has been investigated by monitoring the early narrowing of their fluorescence spectrum. This narrowing was found to occur on timescales ranging from a few hundreds of femtoseconds to a few picoseconds. Other processes, such as a partial decay of the fluorescence anisotropy and the damping of a low-frequency oscillation due to the propagation of a vibrational wavepacket, were found to take place on a very similar time scale. No significant relationship between the strength of nonspecific solute-solvent interactions and the vibrational energy relaxation dynamics of the solutes could be evidenced. On the other hand, in alcohols the spectral narrowing is faster with a solute having H-bonding sites, indicating that this specific interaction tends to favor vibrational energy relaxation. No relationship between the dynamics of spectral narrowing and macroscopic solvent properties, such as the thermal diffusivity, could be found. On the other hand, a correlation between this narrowing dynamics and the number of low-frequency modes of the solvent molecules was evidenced. All these observations cannot be discussed with a model where vibrational energy relaxation occurs via two consecutive and dynamically well-separated steps, namely ultrafast intramolecular vibrational redistribution followed by slower vibrational cooling. On the contrary, the results indicate that both intra- and intermolecular vibrational energy redistribution processes are closely entangled and occur, at least partially, on similar timescales.